56644 China Watershed Management Project (CWMP) Development of a Monitoring & Evaluation System Final Report ITAD Ltd 12 English Business Park English Close, Hove June 2006 1.1.1.1.1.1.1 BN3 7ET U.K. Team Leader: Tel: 01273 765 250 Johanna Pennarz Email: mail@itad.com Web: www.itad.com China Watershed Management Project Final Report (June 2006) 2 Table of Contents: Executive Summary........................................................................................................ 7 1 Background .......................................................................................................... 12 1.1 Project Background ...................................................................................... 12 1.2 Objectives and scope of this assignment (ToR) ........................................... 13 1.3 Steps in Developing the M&E system ........................................................... 14 2 Review of Loess Plateau Project M&E system...................................................... 16 2.1 M&E system administration .......................................................................... 16 2.2 Indicators monitored in Loess Project .......................................................... 18 2.3 Physical progress and outputs quality monitoring ....................................... 19 2.4 Monitoring project impact............................................................................. 20 2.5 Findings on soil and water monitoring ......................................................... 21 2.6 Findings on poverty monitoring ................................................................... 23 2.7 Conclusions on M&E system design ............................................................ 25 3 Developing an M&E System for CWMP................................................................. 27 3.1 Stakeholder Analysis.................................................................................... 27 3.2 Stakeholder agreement on focus of M&E system ......................................... 29 3.3 Objectives of M&E for CWMP ....................................................................... 30 3.4 M&E framework development....................................................................... 31 3.5 Learning and capacity building..................................................................... 32 3.6 From Loess Project to CWMP: A major shift in M&E .................................... 34 4 Towards an Approach to Watershed M&E ............................................................ 36 4.1 How to define a watershed?.......................................................................... 36 4.2 Objectives of Watershed Management ......................................................... 37 4.3 Environment-Poverty Linkages..................................................................... 38 4.4 Focus of watershed M&E.............................................................................. 40 4.5 Piloting M&E in CWMP.................................................................................. 41 5 M&E Framework for CWMP................................................................................... 43 5.1 Conceptual Framework for M&E ................................................................... 43 5.2 Indicators...................................................................................................... 45 5.3 Data collection methods ............................................................................... 49 China Watershed Management Project Final Report (June 2006) 5.4 Sampling strategies...................................................................................... 50 5.5 Reporting ...................................................................................................... 54 5.6 Data analysis................................................................................................ 57 5.7 Integrating and presenting M&E information ................................................ 60 6 Stakeholder participation in M&E ......................................................................... 64 6.1 Institutional framework................................................................................. 64 6.2 Rationale for stakeholder participation in M&E ............................................ 65 6.3 Approach to participatory monitoring in CWMP ........................................... 68 6.4 Indicators for participatory monitoring ......................................................... 70 6.5 Participatory evaluation of project activities ................................................ 73 6.6 Monitoring stakeholder participation in CWMP ............................................ 74 7 Implementing baseline surveys............................................................................ 76 7.1 What is a "baseline"? ................................................................................... 76 7.2 Basic steps in implementing a baseline........................................................ 77 7.3 Using baseline information........................................................................... 80 8 Further steps ........................................................................................................ 82 8.1 Building capacities for participation ............................................................. 82 8.2 Building capacities for M&E.......................................................................... 83 8.3 Accountability and transparency in M&E...................................................... 87 8.4 Criteria for assessing the quality of M&E...................................................... 88 9 Conclusions and Outlook ..................................................................................... 90 Annexes........................................................................................................................ 92 Loess II Log-Frame and Key Performance Indicators............................................... 92 Key performance indicators for CWMP..................................................................... 93 National System for Monitoring Soil and Water Conservation .................................. 96 Terms of Reference for Baseline Survey on Key Poverty Indicators .......................100 Terms of Reference: Baseline survey on Soil and Water Indicators .......................108 Terms of Reference: Baseline on Biodiversity Indicators .......................................116 4 China Watershed Management Project Final Report (June 2006) List of Boxes: Box 1: From the Project memorandum: ............................................................................ 13 Box 2: Indicators used in the existing M&E for Loess-I and Loess-II ................................... 18 Box 3: Progress M&E steps for Loess-Project.............................................................. 20 Box 4 Monitoring project impact in Loess ProjectI ....................................................... 21 Box 5: Stakeholders in Loess Project M&E................................................................... 27 Box 6: From implementation monitoring to results monitoring ............................................ 34 Box 7: Criteria to assess quality of indicators .................................................................... 49 Box 8: Criteria to a assess quality of data collection .......................................................... 49 Box 9: Wealth Ranking of Rural Households in Mahuang Village, Kongtong County............ 51 Box 10: Wealth Ranking of 24 Villages in Dazhai Township of Kongtong County ................ 52 Box 11: Sampling of villages within a watershed ............................................................... 52 Box 12: Quantitative & Qualitative Assessment of Poverty compared................................. 59 Box 13: Scoring key performance indicators on poverty (example) ..................................... 61 Box 14: Methods for participatory project evaluation.......................................................... 73 Box 15: Marginal Groups in Watershed Management Projects........................................... 74 Box 16: Indicators to monitor stakeholder participation ...................................................... 75 Box 17: Basic steps in implementing a baseline................................................................ 77 Box 18: Small watersheds selected for CWMP (Gansu Province)....................................... 78 Box 19: M&E skills for CWMP project staff........................................................................ 84 Box 20: Quality criteria and key features of M&E............................................................... 88 List of Figures: Figure 1: Institutional structure of Loess M&E ................................................................... 17 Figure 2: Typical Hierarchy of Watersheds at Farm and Village Levels ............................... 36 Figure 3: Levels of Monitoring.......................................................................................... 43 Figure 4: Levels for monitoring impacts in CWMP ............................................................. 44 Figure 5: Information pyramid and information flows in M&E .............................................. 55 Figure 6: Information triangle in M&E ............................................................................... 58 Figure 7: Monitoring platforms and stakeholder groups in CWMP ....................................... 64 Figure 8: National soil and water monitoring network ......................................................... 98 5 China Watershed Management Project Final Report (June 2006) List of Tables: Table 1: Steps for setting up the M&E system in CWMP .................................................... 14 Table 2: Objectives for improved M&E in CWMP, changes and institutional implications...... 31 Table 3: Overview - Building capacities in the process of M&E development ....................... 33 Table 4: Structure of indicators on environment and povert y .............................................. 46 Table 5: Key performance indicators to monitor long-term changes at the watershed level .. 47 Table 6: Indicators to monitor environment-poverty linkages within a watershed ................. 47 Table 7: Indicators to monitor direct project outcomes ....................................................... 48 Table 8: Indicators to monitor access to project services by poor and marginal groups ........ 48 Table 9: Sampling strategies for M&E .............................................................................. 54 Table 10: Reporting system on M&E ................................................................................ 56 Table 11: Responsibilities for data collection and analysis ................................................. 60 Table 12: Scoring indicators for different watersheds (example) ......................................... 61 Table 13: Integrated data analysis at different management levels ..................................... 63 Table 14: Comparing questionnaire survey and participatory data collection ....................... 68 Table 15: Indicators that can be monitored through participatory assessments (selection) ... 71 Table 16: Collecting information on indicators through participatory assessment (example) . 71 Table 17: Indicators and methods for participatory impact assessments ............................. 72 Table 18: Household poverty indictors and data (2005 baseline) ........................................ 76 Table 19: Basic data of 4 experimental small watersheds............................................ 79 Table 20: Institutional change and capacity building for M&E ............................................. 85 Acronyms used: CWMP China Watershed Management Project DFID Department for International Development EV Extension Volunteer LGU Local Government Unit Loess Loess Plateau Project (funded by World Bank) M&E Monitoring and Evaluation MWR Ministry of Water Resources MoU Memorandum of Understanding PADO Poverty Alleviation and Development Office PAB Participatory Assessment of Biodiversity PIA Participatory Impact Assessment PIO Project Implementation Office (Xi'an) PLG Project Leading Groups (provincial, city and county levels) PMO Project Management Office PTD Participatory Technology Development RSO Rural Survey Organisation WB World Bank WCB Water Conservation Bureau WCMN Water Conservation Monitoring Network YRCC Yellow River Conservancy Commission 6 China Watershed Management Project Final Report (June 2006) Executive Summary Background The objective of this assignment was to develop an M&E system for watershed management in the Loess Plateau area. The M&E system for the China Watershed Management Project (CWMP) has been developed on the base of the M&E system that has been implemented during the previous two phases of the World Bank Loess Plateau Project. The China Watershed Management Project (CWMP) is located in Gansu Province which is part of the Second Loess Plateau Watershed Rehabilitation Project. The Loess project area shows high levels of environmentally-related poverty. Without sufficient vegetation covers, soil erosion is severe in the Loess area, causing loss of fertile land in the upper reaches of the watershed, and problems of sedimentation in the lower reaches. There is a critical need to be able to reliably assess the effectiveness and impact of watershed management in the Yellow River Basin. The China Watershed Management Project (CWMP) is contributing to the improved management of the Yellow River, and other basins, by developing best practice models for oess project M&E system has been successfully watershed management. The previous L implemented during two project phases. It had its strength in monitoring progress of project interventions against planned targets. It has faced a number of problems on monitoring impacts on soil erosion, sedimentation and water quality, mostly due to capacity constraints and lack of relevant data. It had less emphasis on monitoring environmental impacts, and it had obvious shortcomings in monitoring poverty impacts. A key problem for the Loess project was attribution: how could project results possibly be linked to wider environmental and socio- economic changes within the loess area? The challenge for this assignment was to develop an approach which would allow for better monitoring of the different dimensions of change within a watershed context and at the same time address the problem of attribution. Approach A framework for M&E in CWMP has been developed through the following steps: The M&E systems review has been looking at the existing M&E in Loess Project. It has assessed existing of practices for monitoring environmental and socio-economic indicators and identified a number of practical problems and issues. The M&E systems review has also been looking at the overall structure and functioning of the Loess M&E system. It found that 7 China Watershed Management Project Final Report (June 2006) the Loess Plateau Projects' M&E system was strongly focused on checking physical implementation progress, with a less structured approach to monitoring outputs or evaluating outcomes and impact. It suffers from limitations due to design (small sample size, poor poverty targeting, etc), use and flow of data (lower levels collect and pass data upwards, where decisions are made, and passed back down), and lack of farmer participation in the system (which would improve accountability and cost effectiveness During the inception phase, a stakeholder analysis has been done and the project has agreed that stakeholders in the project area, especially poor farmers, need to participate more in future M&E. The project has also agreed on the main objectives for the revised M&E system. The new M&E system should ú Monitor livelihoods benefits for the poor ú Monitor improved ecological conditions in watersheds ú Monitor participation of the poor in the project ú Improve the quality and efficiency of data collection, handling and analysis ú Increase stakeholder participation ú Be cost effective An M&E framework has been drafted, based on those objectives, and indicators have been developed and tested subsequently. M&E framework The M&E framework developed for CWMP shows some key characteristics which make it differ from pervious Loess M&E: ú Impact levels are defined based on a watershed concept; ú The importance of linkages between environmental and poverty is reflected in the indicators framework; ú Stakeholder participation is a key element of CWMP, and platforms for stakeholder communication and participation have been created along the four M&E levels. CWMP M&E also differs from previous Loess M&E because of its emphasis on impact monitoring which will be done on fours levels: - On the farm level, farmers, technicians and project staff will monitor changes of land use practices and improvements of soil conditions. - On the community level, project staff will monitor project implementation process and changes of livelihoods and local environment together with the community, using participatory methods. - On the watershed level, sets of key performance indicators have been defined which will help to monitor changes in the physical and socio-economic conditions of watersheds. Surveys for monitoring key performance indicators will be contracted out to specialised survey organisations. 8 China Watershed Management Project Final Report (June 2006) - On the river level, the project will use existing indicators to monitor "healthy rivers" in the sectoral context. Indicators on river health will be monitored by technical institutions. Key features the new M&E system Monitoring watersheds: The M&E framework has been developed to support an approach to watershed management in CWMP has adopted a watershed-based approach to M&E. Managing and monitoring of projects for impacts in the context of a watershed is a challenging task and requires a comprehensive understanding of the natural systems and the impacts of human activities. Watershed monitoring plays an integral part by tracking progress towards watershed objectives, while collecting more environmentally relevant data than traditional compliance monitoring. Four levels of M&E: Identification of four distinct levels for monitoring was a major step forward: it allows integration of data at all levels; at the same time it helps to establish clear linkages for monitoring impacts along a results chain. Focus on key performance indicators for each level helps to reduce complexity of data at higher levels and allows comparative analysis of distinct geographic units (villages, watersheds) within a watershed context. Data quality: The revised M&E framework aims at improving reliability, validity and timeliness of data. Data collection methods have been identified and tested for each indicator with emphasis on primary information collected in the field; in addition, it has been agreed that parts of the data collection and reporting will be outsourced to specialised survey organisations which have the capacities and methodology available to ensure data quality. Participatory monitoring has been introduced as an approach to assess project impacts. Participatory monitoring is especially useful for assessing poverty impacts, but it can also be used for monitoring environmental changes. A range of indicators has been tested which can be monitored through participatory methods and concrete methods for participatory monitoring are proposed in the M&E framework. Participatory monitoring is seen as an important approach to supplement the findings from conventional surveys. Some indicators are better monitored through participatory data collection, and participatory methods are particularly strong in encouraging the weaker parts of the community to express their views and perceptions. Sampling strategies: We therefore recommend the following, "two track" approach for CWMP which would help to combine advantages from different approaches in order to gain a better understanding of impacts under this project: - For monitoring trends of development in addition to achievement of targets, the project should select one "control watershed" per county. Key performance indicators on poverty, biodiversity, soil and water should be monitored for project and non- 9 China Watershed Management Project Final Report (June 2006) project watersheds in order to understand extent of changes in the status of the watershed that is likely to happen as a result of the project. - For understanding the causality of changes and hence the impact that can realistically be attributed to the project, information would be collected from project communities and households. Participatory methods would help to understand why certain changes occur and how they affect people's livelihoods within the complex environment of a watershed. In addition, participatory studies would help to explore factors affecting changes which have not been anticipated by the project. Integrated analysis of environmental and socio-economic data is a challenging task. It is almost naturally done at lower levels, using participatory methods. Farmers and communities reflect on their livelihoods within the holistic context of their local environment; they are not inhibited by technical specialisations. Therefore, participatory assessments provide a great potential for better understanding environment-poverty linkages within a local setting. At higher levels, more sophisticated approaches will be required to analyse linkages between different indicators. Some techniques have been introduced during the workshop on data analysis. Presentation of M&E data: Environmental and socio-economic data on performance of watersheds need to be presented in a way that allows for decision-making on key issues of watershed development. A major shortcoming of the previous M&E system was that management did not have the information available they need for taking their decisions or they did not have them presented in a way that they could be conveniently used for management task. A key challenge is the integration of various types of data that are required for watershed management, especially the difficulty of integrating socio-economic and environmental data. Information pyramid: The information required by each management level is different; in addition; stakeholders have their specific priorities for the information they want to get from M&E and therefore information need to be well focused and presented. Focus on key performance indicators for each level helps to reduce the information load. A reporting and documentation system has been proposed, considering the different information needs. Future challenges Work on M&E in CWMP has created opportunities more sustainable watershed management in the Loess plateau area. It has deepened understanding of environment-poverty linkages and the importance of stakeholder participation for more localised approaches to address urgent environmental issues in this region. Through development of an M&E framework the project has sharpened its view on intervention levels and it has recognised the importance of getting lower levels, especially poor farmers, directly involved in project planning and monitoring. Still, a number of major challenges will have to be addressed by the project during the upcoming phase. 10 China Watershed Management Project Final Report (June 2006) Baseline: CWMP will have to implement a full baseline in a next step. Guidelines and ToR for baseline surveys have been developed and are included in the annex. It has been agreed that baseline surveys will be carried out by specialised survey organisations; at the same time that PMO will build its capacities for Participatory Monitoring. It will be a major challenge for the project to analysis and integrate the data from the baseline which may require some additional support. A learning exercise is recommended in which proposed methods for data analysis and integration can be tested. This workshop would also be an opportunity for the M&E to revisit and further refine the M&E framewok. Geographical vs. administrative units: It is recommended that the approach of identifying and monitoring geographical units (rather than administrative units) should be followed through for effective watershed management. The project would need to identify and target poor natural villages within a watershed in order to address addressing environment-poverty linkages effectively. Use of geographical units for monitor would allow the system to link up with a possible system for watershed GIS. Focus on poor and marginal groups: The project would need to strengthen monitoring project impacts on certain social groups: poor households, women and other marginal group. The report suggests indicators for monitoring participation and impacts on poor groups. Participation is still a new and challenging concept in CWMP. Project staff has gained initial experiences with participatory methods through practical field exercises as part of the indicators development. Participants have recognised the possibilities for use of participatory methods in M&E and the need for greater participation of primary stakeholders in general. This has created opportunities for CWMP to introduce participation more widely. Useful methods for participation would be Farmers Field Schools for technology development and participatory land use mapping for planning and monitoring purposes. These techniques would also support the shift towards more field-based methods for observation and data collection. Specialised vs. holistic approach: The present approach of monitoring environmental and poverty indicators within a modular structure has been adopted to address existing capacity constraints. External monitoring of key performance indicators helps to improve reliability and credibility of data and analysis. In addition, the project should strengthen its capacities for more holistic approaches to observe changes within a watershed context. Better understanding of environment-poverty linkages and ecological processes together with higher proficiency of field-based methodologies will enable project staff to develop such approaches. 11 China Watershed Management Project Final Report (June 2006) 1 Background 1.1 Project Background The China Watershed Management Project (CWMP) is located in Gansu Province which is part of the Second Loess Plateau Watershed Rehabilitation Project (usually referred to as "Loess II"). The natural resources of the Plateau have been so intensively utilized over thousands of years that it is now denuded and environmentally degraded due to extensive soil and water erosion causing loss of fertile land in the upper reaches of the Yellow River Watershed and sedimentation and flooding in the lower reaches in a cycle of irregular rainfall and frequent droughts and floods. Nationally, the dramatic dimensions of flooding, water scarcity and sedimentation along the Yellow River Watershed together with high levels of poverty around the upper and middle reaches of the river have made this a priority area for national and international support. The Loess project area shows high levels of environmentally-related poverty. Without sufficient vegetation covers, soil erosion is severe in the Loess area, causing loss of fertile land in the upper reaches of the watershed, and problems of sedimentation in the lower reaches. The problem becomes even more serious with the irregular rainfalls, resulting in frequent droughts and flooding along the Yellow River. Lack of irrigation, soil erosion and limited soil fertility due to low contents of organic matter further constrain agricultural production in the Loess area. As a result, livelihoods in these environmentally fragile areas are precarious; lack of food security and lack of access to drinking water are still common problems. Without sufficient opportunities for non-agricultural employment, many rural areas are facing high rates of out-migration. There is a critical need to be able to reliably assess the effectiveness and impact of watershed management in the Yellow River Basin. This is well described in a recent synthesis of information on water and water management in the basin: "Management of China's Yellow River basin is at a cross-roads. Decreasing water supplies, increasing demand and a rapidly growing economy have added new challenges to a management agenda and institutional infrastructure traditionally focused on flood control and irrigation development. As a result, basin managers must now contend with such issues as water scarcity, water allocation, and environmental degradation while still guarding against floods and contributing to China's food security goals. Because of the role of the Yellow River basin in China's overall economy, the success of its water managers in 1 addressing these new issues will have implications for the entire country." 1 Giordano, M.; Zhu, Z.; Cai, X.; Hong, S.; Zhang, X.; Xue, Y. (2004). Water management in the Yellow River Basin: Background, current critical issues and future research needs. Comprehensive Assessment 12 China Watershed Management Project Final Report (June 2006) The China Watershed Management Project (CWMP) is contributing to the improved management of the Yellow River, and other basins, by developing best practice models for watershed management. Box 1: From the Project memorandum: The objectives of the Trust Fund are to: (i) examine the contributions to poverty reduction of Bank supported projects; (ii) identify how short-comings in these projects can be improved; (iii) draw out and develop best practice approaches in watershed management programmes; (iv) strengthen the capacity of existing Bank project management offices and other Chinese government and non government organisations 1.2 Objectives and scope of this assignment (ToR) The objectives and tasks for this assignment have been specified in the consultant's ToR as follows: "The objective of the assignment is to develop a complete, comprehensive and sustainable social and environmental Monitoring and Evaluation System. This system needs to be broad enough to be adapted to other national or donor watershed programmes. It needs to look at the current Monitoring systems for key model counties (throughout the Loess Plateau Project Area) and also involve other organisations involved in monitoring. How the M&E System is going to measure the impact of the watershed rehabilitation and management programmes on the environment and how the role of GIS could play in the development and use of the M&E System also need to be incorporated into the process of review and M&E system development. In developing the system there will be an emphasis on participatory approaches. The aim is to improve project monitoring to help better evaluate project benefits for the poor and disadvantaged. Monitoring and evaluation will focus on the effect the project is having on the poor and disadvantaged in project villages including changes in incomes and also the wider impacts on people's livelihoods. Ways in which the poor may be disadvantaged by project implementation will be evaluated while those missing out on project benefits will be identified. The effect the project is having on different vulnerable groups will also be considered. Other issues that could be investigated are the distribution of improved land and the efficacy of mechanisms for repayment of loans and cost recovery in project areas. Developing improved M&E systems will focus on four counties of Jingning, Pingliang "city", Huachi and Huanxian of Gansu Province. Research Report 3. Colombo, Sri Lanka: Comprehensive Assessment Secretariat, International Water Management Institute (IWMI). 13 China Watershed Management Project Final Report (June 2006) A key feature of this assignment will be to develop a set of indicators for assessing the impact of Loess Plateau 2. The system should include qualitative and quantitative poverty impact and participation indicators. Poverty impact indicators will be used to assess the effect of the Bank supported projects on household poverty. Participation indicators will be used to assess levels of participation in the implementation of the interventions. Specific indicators will have to be devised for assessing the involvement of women, vulnerable groups, and poor households in Loess project implementation. The indicators developed for assessing the Loess Plateau 2 project impact will also be available as a base for developing indicators for other similar watershed management projects in China. The improved systems for M&E will need to be closely link with the existing M&E System and management information system (MIS) for the bank supported projects and will need to link monitoring of poverty impact with indicators for the implementation and monitoring of physical interventions. It will be important to identify gaps in monitoring the progress of physical implementation and how these might be addressed. A baseline survey will be conducted at an early stage by the PMOs. This will be related to the indicators developed for the M&E system and should use similar methodologies." 1.3 Steps in Developing the M&E system 2 Based on the ToR, the consultants have elaborated their work plan. The following main steps 3 have been implemented to develop the M&E system for CWMP: Table 1: Steps for setting up the M&E system in CWMP Finalised steps and Chapter in this outcomes report M&E systems review Task 2 report 2 Stakeholder analysis and agreed objectives of M&E Inception report 3 (Task 1 report) Select key indicators to monitor outcomes Task 3 report; reports 5 on indicators development M&E framework, including indicators, methods and Interim report 5 institutional responsibilities, as agreed 2 Consultant Team: Julian Barr, Deng Weijie, Don Brown, Liu Xiaoying, Liu Yonggong, John Taylor, Song Haokun, Wang Libin. 3 Similar steps are described in the Wolrd Bank publication: Jody Zall Kusek, Ray C. Rist: Ten steps to a results based monitoring and evaluation system. The World Bank 2004. 14 China Watershed Management Project Final Report (June 2006) Preparation for baseline survey Baseline ToR included 7 plus Annexes in final report Preparation for implementing the M&E system M&E guidelines 8 plus Annexes The final report presents the main outcomes of this process. The major findings of the M&E systems review are summarised in the following Chapter 2. The steps of developing and agreeing the M&E framework are presented in Chapter 3. Chapter 4 will describe the specific features of and challenges for watershed M&E before Chapter 5 goes into a detailed presentation of the M&E framework. The M&E framework is most significant outcome of this assignment because it brings together all the agreements and considerations around institutional and methodological issues of M&E. The key parameters for M&E are defined in the framework, but they are further specified into operational steps in the M&E guidelines which are prepared as a separate document. The ToR and guidelines for the technical surveys to be carried out as a baseline are included in the Annex. 15 China Watershed Management Project Final Report (June 2006) 2 Review of Loess Plateau Project M&E system 2.1 M&E system administration The Loess Plateau Projects I and II operate a 5 tier system of Project Management Offices (PMOs) who are responsible for project implementation. Within the PMOs are Monitoring 4 Stations, responsible for the M&E. In relation to the M&E stations, the 5 tiers are: · Central PMO the Loess Plateau Restoration Monitoring Centre; located in the Upper and Middle Reaches Bureau (YRMB) of the Yellow River Conservancy Commission (YRCC) in Xi'an. · Provincial PMO Provincial Monitoring Centres of the Loess Plateau Project; located in the Provincial Water Conservation Bureaus (WCBs), in LPP II, these were in Gansu, Shaanxi, Shanxi, and Inner Mongolia · City PMO General Monitoring Stations (GMSs) of the Loess Plateau Project; located in the City level WCBs. There were 7 of these in LPP II. · County PMO Divi sional / County Monitoring Stations of the Loess Plateau Project; located in the County level WCBs. There were 22 of these in LPP II. · Township PMO Monitoring Sub-Stations of the Loess Plateau Project, of which there are 170. Located in the Township Water Conservation Bureaus (WCBs) The M&E system involves approximately 1200 people across the Loess II area. This includes 3-5 at the Central PMO (CPMO), and up to 10 at both City and County levels. Most of this human resource is occupied in physical progress monitoring. The cost of this monitoring was reported to be about 3% of the Chinese contribution to the overall budget. Loess II, and thence its M&E system operate along the lines of a standard Chinese `bureau' system. The system has its hierarchical cascade of administrative tiers, with the control and decision making functions mainly vested in the higher tiers. Performance targets are set at the upper levels of the hierarchy (CPMO and Provincial PMO), with instructions passed down to the lowers levels, where they are enacted by comparatively poorly resourced and poorly 4 These are LPP tiers, and should not be confused with the similar 5 PMO tiers of CWMP. 16 China Watershed Management Project Final Report (June 2006) capacitated County, and mainly Township PMOs. Their work is checked from time-to-time by the higher levels. In regard to the M&E system, a blueprint monitoring plan is created by the CPMO Monitoring Centre, which is passed down to, and closely followed by, the Provincial and lower PMO monitoring teams. The County and Township thus collect the monitoring data (especially the physical progress monitoring data), collate them and pass them back up the hierarchy in a summarised, but unanalysed form. Data are entered in PCs at the City level, and submitted to the Province, where issues are identified from the data. The Provincial Monitoring Centre h prepares a report for review by the Directors of t e Provincial PMO. The Directors can recommend small changes in the implementation plan; larger ones are submitted to the Provincial Steering Committee. These must be ratified by the PIO (Xi'an) and may require further approval by the WB. Figure 1: Institutional structure of Loess M&E The Loess PMOs operate as a vertical silo within local government. This is a heavily projectised approach to development, and it means that systems are not institutionalised. Even within the YRCC/UMRB, the was a lack of evidence of how LPP M&E fits with the organisation's routine monitoring system for the whole of the Yellow River. However, it also means that there is a lack of interaction with other agencies and bureaus working in the area, not least in regard to each other's M&E The system has good vertical linkage, and is functioning reasonably well as an information and instruction communication mechanism. However this is occurring within in a vertical "water conservation" silo. 17 China Watershed Management Project Final Report (June 2006) The horizontal linkages to other bureaus and agencies, such as Forestry, Agriculture and PADO, at all levels are weak. According to the CPMO, horizontal linkages are difficult for two reasons: · It needs approval from above (Zhengzhou / Beijing) · The data are not well analysed, and thus they are not confident in sharing them. In this, they recognise that they are lacking good analytical methods. There is a missing tier, below the Township tier ­ the Village, or the Community. Farmers are providing data to the Township, but there is no feedback of information to communities and they are usually not consulted on major decisions taken.. 2.2 Indicators monitored in Loess Project The following indicators have been monitored in Loess Project Box 2: Indicators used in the existing M&E for Loess-I and Loess-II Categories of Indicators Indicators Progress and quality of the - Area and quality of terrace land ; project outputs - Forest area and survival rate and quality of new plantations ; - Area and survival rate of fruit trees - Area and density of alfalfa (fodder) - Completed no. and quality of cisterns for gathering rainfall water for human, livestock and fruit orchards - Completed no. and quality of check-dam for sedimentation - Increased no. of livestock Economic Impact - Increase of farmer's net income; - Increased production value; - Increased grain and fruit yield Social Impact - Improved drink water facilities - Increased enrolment rate of school age children; - Improved medical care conditions; - Accessibility of electricity; - Access to major road; Ecological impact - Improved soil structure and quality; - Change of the water quality; - Increased vegetation coverage; - Improved bio-diversity Impact on water and soil - Reduction of the water flow in the watershed; conservation - Reduction of soil sedimentation; 18 China Watershed Management Project Final Report (June 2006) 2.3 Physical progress and outputs quality monitoring The Loess-I and Loess-II M&E had a strong focus on monitoring the project progress and quality of the outputs. Process and quality is monitored by county PMO monitoring stations by working with township substations on the implementation of specific project activities. Physical checking of completed project activities: The majority of the M&E resource and effort is directed at this `checking' activity. The reason this is given such high priority is that only when completion of the physical works has been verified are the respective Townships reimbursed. Thus this monitoring is tied to flow of funds, and has a privileged status in the project's M&E. Once the physical works in a certain area, usually the area related to a particular village, have been completed, checking of the areal parameters is conducted with a tape measure and topographic map. Prior to project activity, the map is demarcated with the planned areas for development, and at the checking stage, the measured areas are marked back on to the map. With the exception of the terraces, the checking is conducted by a County PMO official with two assistants from the Township. A 30% sample is then verified (re-measured) by the City PMO, and further spot -checking is carried out by the City and the Province. If in more than 15% of the assessments to check completion of physical progress it is found that the reported areas are greater than the actual areas, then that area (village) does not `pass'. The consequences of this physical audit failure are that funds are not released from the Province to reimburse the Township. Underachievement of physical targets also results in reduced targets for the next annual planning cycle. Because of the mere number of watersheds and size of the project area within the counties physical progress M&E requires high inputs of labour and is very time consuming. County M&E staff and township agents spend most of their time for overall checking up the completed project outputs. PMO's at all levels have already developed an essential capacity for monitoring the physical progress and output quality. The previous M&E related training provided by provincial PMO and PIO was mainly focused on the progress M&E. Most of the county M&E staff are qualified and skilled to carry out this kind of M&E. 19 China Watershed Management Project Final Report (June 2006) Box 3: Progress M&E steps for Loess-Project Major steps Major tasks Assessment and remarks County PMO: Overall self- On-site checking of the completed High human resource demand checking up project tasks and outputs: Time consuming - Terrace land Mainly done by county M&E - Forest staff and township agents, lack of participation of the - Fruit trees communities and farmers - Alfalfa fodder Data and sheets are mainly - Cisterns processed by hand, PC is not sufficiently used - Check-dam Target and quality indicators for M&E are determined in the M&E Guideline City PMO: Rechecking and 30% sampled re-checking for validating Representation of the validating the reported targets, i.e. area, amount sampled watersheds and and quality stands Large human resource inputs PC is used in data processing Provincial PMO: Rechecking 20% sampled rechecking for the targets Representation of the and validation reported by county PMO sampled watersheds and stands Large human resource inputs PIO Rechecking and validation. 8-12% sampled rechecking by PIO M&E Representation of the staff sampled watersheds and stands Large human resource inputs Physical progress monitoring has been strong and must therefore be regarded as very successful in Loess Project. It may be further streamlined to free resources and capacities for other M&E task, but it will remain largely unchanged in CWMP. 2.4 Monitoring project impact Impact monitoring has been less successful in Loess Project, and this was the main justification for CWMP to focus on M&E. The review of Loess M&E has revealed a number of major conceptual and methodological shortcomings. Project impacts have been monitored through data, gained from different sources. The following is a brief summary of the main features of impact monitoring in the Loess Project. 20 China Watershed Management Project Final Report (June 2006) Box 4 Monitoring project impact in Loess ProjectI Level Focus/ Indicators used Methods used Institutio n in charge Sample households, Income/expenditure, Questionnaires filled in by PMO monitoring demonstration plots agricultural activity and sample households station production themselves Sample households Socio-economic indicators Socio-economic survey PMO monitoring conducted by county once station per year Sample plots Vegetation cover Field measurements County and township PMO Micro watershed Monitoring of dams in micro Local households collect PMO general watershed: data collection on data when raining and monitoring station Rainfall, Flood situation flooding occurs; PMO measures sediment in dams. County Weather Micro climate: Purchase data Weather Bureau in stations each counties City Hydrological Water quality: City PMO collects samples; Yellow River stations analysis contracted out to Management Hydrological Station Committee in Qingyang city City Hydrological Soil situation:; Purchase data Yellow River stations Management Committee located in Qingyang city Province Hydrological Hydrological formula to detect Purchase data Yellow River station impacts Management Committee and Hydrology station, Gansu Province Loess project monitoring has been heavily reliant on existing indicators or data provided by monitoring stations within the system; the PMO staff had neither the technical facilities nor the capacities to implement monitoring of environmental indicators. Social and economic indicators have been monitored by the PMO staff. 2.5 Findings on soil and water monitoring Because of the significance of Loess soil and water monitoring within the Ministry of Water Resources, the M&E has carried out a specialised which has identified a number of 5 shortcomings and issues : The effect of project interventions on sediment reduction (eg sediment load of the river) is calculated and analysed the City PMO GMS, using formulae developed by the engineers from the Hydrology Station of the Yellow River Management Committee. In the case of 5 ITAD/IRTCES. Report on Soil and Water Monitoring in Loess Project. 2005. 21 China Watershed Management Project Final Report (June 2006) Gansu, all the relevant data were purchased directly from YRMC Hydrology Station in Gansu Province. Attribution is a difficult issue when considering this aspect of project impact. Location of data collection sites in relation to where in the main and sub-watershed the project has been and is working, linked to accurate real-time weather recording in the same watersheds, is critical if attribution is to be realistic. Lack of guidance and coherence: Lack of sufficient guidance or set of requirements on the integrated project monitoring and evaluation system of monitoring on sediments and runoffs has made the project M & E system become an assembly of parts of environmental protection indicators, soil and fertilizer extension indicators etc. The system for monitoring and evaluation for sediment and runoff set by PIO are not strictly followed by various level PMO due to lack of funds and experienced technicians. Low accuracy of data and no validation of calculated results: Due to the constraints of funds and other factors, no observation on runoff and sediments in small watershed were conducted at the Gansu WB project area. Most of data on runoff and sediments were obtained from hydrological stations, soil and fertilizer experimental stations, meteorological station, environmental protection station through contract. For instance, Qingyang city, there is only one runoff plots; it can not be used because of its only one slope degree of 19.8, observed by a farmer and without sufficient representation. Neither observation was done in small watershed nor runoff plots in Pingliang MCSWC. The parameters for calculating sediment reduction induced by soil and water conservation were obtained from parent organizations, because there is no monitoring on runoff and sediment in WB typical small watersheds. Hence, the ecological benefits, especially effects of sediment and runoff reduction were calculated through "soil and water conservation method", the parameter (modulus of sediment reduction) is obtained through other watersheds; some are even very much beyond the project area. Only soil conservation method is used for calculation of impacts without validation of hydrological method, the figures obtained are hardly trusted. In some places, in order to get the reasonable figure of sediment reduction figures as the officers expected, the parameters were re-adjusted to recalculate in order to satisfy the needs of the project as planned. In addition, since there is no measured data available at micro- watershed, meso-watershed and marco-watershed, it is not possible to use "hydrological method" at small watershed and meso-watershed level and difficult to validate the "soil and water conservation". The calculated data can not truly reflect the effect of sediment and runoff reduction to the tributaries and then to marco-watershed. The figures of runoff and sediment calculated in the monitoring reports at county, city and provincial levels are hardly be recognized by many scientists and the society. People are generally not convinced by those calculated results if no further monitoring and analysis are to be done. 22 China Watershed Management Project Final Report (June 2006) Insufficient monitoring points and violation of the regulations set by PIO. The monitoring points are not sufficient enough for reliable and accurate figure and methods of vegetation coverage, deposition of sediment in warping dam, small reservoirs, ponds, cisterns, etc are not at the right layout. The sampling methods prepared by PIO Monitoring and Evaluation Center are not strictly followed. The sampling points are not sufficient for the monitoring. The accuracy of the data is relatively low. If more accurate data for the project, more monitoring points, scientific monitoring methods should be set and used and the sampling methods should be strictly followed in the future monitoring work for a better understanding of the achievements and benefits obtained through the WB project. Capacity issues: At various levels PMO staff could not use the new technology and could not follow the calculation methods, formula, the regulations and rules compiled by PIO although some training course was organized. From the monitoring and evaluation reports provided by county and city PMO, some mistakes can be easily found, some parts of the formula regulations were not calculated or included, for instance, in the "soil and water conservation method', the new soil loss caused by human activities such as mining, road construction, house building, oil well digging was not considered in the calculation and result in the doubted calculation result. Insufficient monitoring indicators: Some important monitoring indicators of the project are neglected during the first and second phase of WB project due to lack of funds and less attention paid by the PMO, and maybe other reason. For instance, wind erosion monitoring of WB project area, decrease of gully erosion and slope erosion due to the erosion control projects constructed. This also results in the low accuracy of the calculated sediment reduction. Moreover, some river like Malianhe river has got the confused figure of sediment f reduction, the calculated result o sediment in the basin is increased after soil and water conservation for many years. Because of lack of reliable and accurate data, for instance, data of long-term rain storm and new soil loss The review of soil and water monitoring concluded, that soil and water monitoring is a highly specialised tasks which requires specific technical expertise and facilities which are beyond the project PMO's capacities. The consultant therefore recommend stronger links with the 6 national soil and water monitoring network and outsourcing of survey task. 2.6 Findings on poverty monitoring 7 The M&E systems review concluded that the existing M&E system has a number of weaknesses, several of which are crucial for considering how to develop an appropriate 6 A more detailed description of the soil and water monitoring network is included in the Annex. 7 ITAD. China Watershed Management Project Development of a Monitoring and Evaluation System. Task : Review of Existing M&E Systems. May 2005. 23 China Watershed Management Project Final Report (June 2006) system to assess environmental improvements and poverty reduction. For Task 3, the more important of these are the following: · Quantitative vs. qualitative: There is a heavy reliance on quantitative data, and a concomitant lack of qualitative evaluation. This is particularly the case for the evaluation of livelihood and poverty outcomes. · Beneficiaries: Project beneficiaries (poor farming households) are not involved in the M&E system, except through their supplying data. This results in asymmetric relationships between farming household members and PMOs. As outlined in the DFID social development assessment of the impact of Loess Project, this limited the 8 extent to which Loess 2 could meet the needs of relatively poorer households . · Participation: As a consequence, there is little village and household involvement in planning and decision-making on the basis of M&E data. · Gender: There is no gender disaggregated data collection. · Sample size: Household monitoring is limited by the extremely small sample size. In Loess I, the sample size was reported to be 480 households (~2000 ­ 2500 people) out of a total beneficiary population of 1.29 million (less than a 0.2% sample of the population). · o Sampling strategy: The selection of sample households is also biased t wards relatively less poor households (focusing ­for example- on households located on or near roads, or on literate households). · Questionnaires: Household monitoring forms are too complex, often making it difficult for respondents to complete them. Households are meant to self-complete these quite detailed tables on a regular basis. For the economic data, some data, such as income and expenditure are meant to be entered on to a form on a daily basis, and others, eg agricultural inputs and production on a monthly or seasonal/annual basis. In practice a Township level PMO officer usually helps with the task, or a year's worth of data may be collected with a single household visit by one of the PMO M&E staff. · Indicators: The use of existing indicators and data collection instruments results in a lack of ability to distinguish between changes attributable to project interventions, and changes attributable to other factors. The set of indicators for measuring the economic impacts was fairly simple ­ income/expenditure, agricultural activity and production. · Participatory monitoring: There is an urgent need for capacity building for the development of more participatory based monitoring and evaluation, and for the undertaking of more integrated analyses of environmental, economic and social data. This is crucial, if the project is to assess its ability concomitantly to improve livelihoods, reduce poverty and generate a basis for environmental improvements. 8 See Zhong Bingfang, Wang Haimin and John G. Taylor (2001), Social Development Research Report: An Assessment of the Poverty Reduction Impact of the Loess Plateau Watershed Rehabilitation Projects in Counties of South Eastern Gansu. Report for DFID China. 24 China Watershed Management Project Final Report (June 2006) 2.7 Conclusions on M&E system design The M&E systems review has been looking at the structure and functioning of the Loess M&E system. It found that the Loess Plateau Projects' M&E system was strongly focused on checking physical implementation progress, with a less structured approach to monitoring outputs or evaluating outcomes and impact. It suffers from limitations due to design (small sample size, poor poverty targeting, etc), use and flow of data (lower levels collect and pass data upwards, where decisions are made, and passed back down), and lack of farmer 9 participation in the system (which would improve accountability and cost effectiveness). · Independence. As identified by the CPMO, there is a lack on independence in the M&E system. Implementation and M&E are both carried out by the PMOs. · Guidance and Coherence. The CPMO also reports a lack of guidance, or set of requirements on the project M&E system (from the WB). Hence the present system is an assembly of parts of other projects' systems, other bureaus' systems (e.g on technical environmental indicators) and the requirements of the Yellow River Commission. It thus does not hold together as a very coherent system. · Imbalance. There is a very heavy human resource investment in 100% physical checking / audit. Efficiency gains should be possible, eg (i) through use of slightly more sophisticated equipment than tape measures, such as cheap surveyors' ranging equipment. This could also be done by remote sensing, but needs careful consideration of the cost-benefit. (ii) through deciding that spot-checking a sample of less than 100% of all physical works gives enough confidence in the data. · Risks. The PIO is well aware that assumptions in the project design are not holding true (such as incidence of natural events, and the influence of market factors ­ eg. over-supply of apples), thus that not all implementation progress targets are being met. However these assumptions or risks are not being monitored. · Planning. Planning and M&E go together. Planning sets up a framework against which to monitor. Planning is done at high level in Loess II. Hence M&E is controlled by the upper management tiers (CPMO and Provincial PMO). Since there is little lower level involvement in planning, there is little lower level involvement in decision making on the basis of M&E data. This extends below the Township to the village - there is a lack of community / farmer involvement in both planning and M&E.. · Silo. The LPP PMOs operate as a vertical silo within local government. This is a heavily projectised approach to development, and it means that systems are not institutionalised. Even within the YRCC/UMRB, the was a lack of evidence of how Loess M&E fits with the organisation's routine monitoring system for the whole of the Yellow River. However, it also means that there is a lack of interaction with other agencies and bureaus working in the area, not least in regard to each other's M&E. 9 ITAD. China Watershed Management Project Development of a Monitoring and Evaluation System. Task : Review of Existing M&E Systems. May 2005. 25 China Watershed Management Project Final Report (June 2006) · Frequency: The review of monitoring data occurs mainly on an annual cycle. This is too infrequent for review, if action is meant to be taken on the basis of monitoring findings. Problems with progress and implementation will have arisen, and either gone away or become entrenched by the time the monitoring data are analysed and something done about them. A quarterly cycle is much more common and useful. The frequency can be reduced if the data do not have to pass all the way up and down the hierarchy. If lower levels are empowered to act on their own monitoring data, then responsiveness and speed of reaction will be greatly increased. · Information load: The PMOs, at the higher levels, suffer from too much data and not enough analysis. Not all the data are useful for project management. A more parsimonious set of indicators is required for CWMP. · Data use : Information is not used at the level at which it is collected. At Township and County levels the capacity, empowerment, and enabling environment to reflect on their own monitoring data and take decisions and act accordingly are lacking. · Data analysis: Lower level PMOs only collect and tabulate data, and pass it up to the next level. They do not analyse or reflect on their own data. City and Province PMOs do simple data analysis annually, and pass instructions back down the command chain on the basis of this analysis. Thus at each level, they lack a good system for learning from their own successes and their own mistakes. · Capacity issues: There is a lack of expertise, knowledge and capacity for technical monitoring and analysis of environmental variables. There is also a lack of capacity and experience in integrated analysis of indicators arising from different components of the monitoring system. Specifically, there is a need for PMOs to know how to undertake integrated analysis of environmental, economic and social data, all of which may be in tension with each other. As a conclusion, the consultant recommended capacity building on general M&E administration and management within the PMO system; in addition, he recommended use of specialist expertise available in the poverty network and soil and water monitoring network through outsourcing of impact surveys.. 26 China Watershed Management Project Final Report (June 2006) 3 Developing an M&E System for CWMP 3.1 Stakeholder Analysis A watershed-based M&E should involve all stakeholders that have a primary interest ("stake") in maintaining good conditions in the watershed. This would include farmers and other users of natural resources in the watershed as well as institutions that are (directly) managing the resources or (indirectly) regulating the use of resources in the watershed. A watershed-based M&E system will aim at bringing together the perspectives and priorities of different stakeholders on watershed management and monitoring changes within the watershed. During the end-of-inception participants have conducted an analysis to identify stakeholders and their possible interest and contributions to (Loess) project M&E. The table included below shows both the possible (adverse) impacts of uncoordinated interventions by different stakeholders in the Loess Project area and the potential areas for cooperation and exchange on M&E. They have also agreed to increase the role of stakeholders in CWMP. Box 5: Stakeholders in Loess Project M&E Stakeholders Activities of Areas of Interest Roles in LP I and LP II Projects Should Stakeholders in p roject or provision of have a are which impact on support role in Loess Project CWMP County and City Rural Households Charge Services Provide Social-economic Statistic ü Statistic Bureau Data Inconsistent Data Statistic Data Household Survey County Land Land Development Provide land survey data to be ü Bureau used for forestation planning Terrace Construction Farmers are willing to participate County Forestry Land Conversion to Technical Member of Project Leading ü Bureau Forestry Consultation for Group Seedlings Labor Division Compensation Methods Forestation Poverty Whole village promotion Coordination Member of Project Leading ü Alleviation and policy (social, Group Development infrastructure, Cooperation communication, Resource education, etc.) Integration Has positive impacts Finance Bureau Financial Member of Project Leading ü Monitoring Group Financial Monitoring 27 China Watershed Management Project Final Report (June 2006) Stakeholders Activities of Areas of Interest Roles in LP I and LP II Projects Should Stakeholders in p roject or provision of have a are which impact on support role in Loess Project CWMP County Potentials of Terrace New Species of Member of Project Leading ü Agricultural Crops Group Bureau Resource Development Terrace Construction of Rural Households County Cow and Sheep Pens Provided the Member of Project Leading Livestock seeds of grass, Group Bureau Planting tress and and provide the raising animals breeding animals Protected the forestry Office for land and grasses, relieved protection the pressures of protection Communities Herding, Collecting Interested in Key implementation role: ü and Families Firewood, Digging constructing Organize project implementation Medicine terrace, cisterns, etc. Beneficiaries Township Wit hout the township Key implementation role: ü Government government, the Coordinate labors, organize project can not be implementation, management implemented and protection polices, communication, participated in the annual planning County Rural Loans Willing to provide Disbursement of project funding Credit loans to project Cooperative Help in constructing households terrace, cisterns, and animal husbandry City Water Data analysis, ü Conservancy benefit analysis Research after the project Institute is completed County Weather Observation Station of Provide weather Bureau Rainfall data and charge fees City Water Water ü Conservation Conservation Monitoring Monitoring Station Station Environment Environment Protection Evaluation Bureau Women's Women Federation Development Stakeholder participation means a huge challenge for institutions that are established and working with clearly defined technical and departmental boundaries. Even disclosure and joint use of basic information on environmental and socio-economic conditions within the watershed seems to be a challenging task. 28 China Watershed Management Project Final Report (June 2006) The long-term objective of stakeholder participation will be to work towards a more sustainable management of watersheds, which reduces conflicts over resource use and integrates environmental and socio-economic perspectives. Sustainable management requires a good understanding of the (environmental and socio-economic) conditions in the watershed and a careful monitoring of changes that are happening as a result from human interventions. M&E provides opportunities for bringing stakeholder together and sharing views and information about those conditions and changes, and those will lay a foundation for considering actions and interventions in relation to (intended and unintended) impacts. The work on M&E provided a first opportunity for identifying key stakeholders around focal areas of M&E and involving them in the monitoring process. Stakeholders have joined the process through field exercises for indicators development and institutional linkages have been addressed in the M&E framework. 3.2 Stakeholder agreement on focus of M&E system During the inception phase, a shared understanding has been achieved that the M&E system developed in CWMP will help to better assess the impacts of watershed-based interventions in the context of Loess projects (I and II). For an in-depth understanding of environmental- poverty linkages in a local setting, CWMP will focus its activities on selected `small watersheds' The study of local impacts from project interventions will inform future interventions in the CWMP pilots and possibly a future project phase of Loess (III). Other major agreements that have been achieved during the inception phase include: ú Project stakeholders have agreed that impact monitoring should include the wider dimensions of environment and poverty. Practical frameworks for analysing impacts resulting from project interventions and for exploring dimensions of environmental- poverty linkages have been suggested in the M&E system review report. ú Agreement has also been reached that indicators should be developed in consultation with other stakeholders, including those who have not been part of the project before. Indicators development will include a review of existing environment and poverty indicators and key indicators for M&E will be decided in a consultative process with project stakeholders. ú Information flows and sharing within the existing system have been assessed during the end-of-inception workshop and participants have reached agreement that those should be improved to include stakeholders within and outside of the project M&E system. Lower management levels within CMP should share information and participate in the use of information. 29 China Watershed Management Project Final Report (June 2006) ú During the end-of-inception workshop, the World Bank representative has agreed that work relating to M&E could be financed from loans in future projects. Secure funding is an important prerequisite (and incentive) for developing an improved M&E system. During the inception workshop, it became obvious that expectations of the future M&E system seem to vary among different stakeholders: "The central and regional PMO's hope that the primary purpose of the proposed M & E system is to serve the water resource sector. If other sectors want to use the system, they could adopt it, but the first priority is to ensure that this M&E system is accepted and workable within the water sector. The provincial and county PMO's in particular are concerned with the practical utility of the M & E system. They are concerned with the practical issues when they conduct the M & E, such as the validity and reliability of indicators, how to deal with the qualitative indicators, sampling techniques, data collection, processing, analysis, and reporting. They want to make sure that the most relevant and important indicators are used to avoid their workload being increased. They also want to know how to differentiate their project impacts from the impacts made by other factors so that their project impacts can be well justified. In addition, the local PMO's are concerned about the support (both financially and in terms of staffing levels) from their up level leaders so that they have the resources they need to 10 develop and use the proposed M & E system." 3.3 Objectives of M&E for CWMP During the end-of-inception workshop, CWMP defined and agreed the objectives of an "improved" M&E system for use in CWMP and possibly later Loess project phases. The general understanding was that CWMP should build on the experiences and achievements of the Loess M&E, but at the same time identify ways to address the obvious shortcomings of the previous M&E system. At the same time, CWMP would develop an approach for integrated watershed management in the Loess area which would be more effective in addressing environment-poverty linkages. While usually the M&E system is defined on the base of a project plan, in this case the objective was to address the shortcomings and fill in the gaps in the previous M&E system. This approach has defined the focus of this assignment and also had obvious implications for the sets of indicators that have been developed. During the inception phase the following objective for an "improved M&E" have been agreed: 10 From the "Report on Inception Workshop". 30 China Watershed Management Project Final Report (June 2006) Table 2: Objectives for improved M&E in CWMP, changes and institutional implications Objective Issues to be addressed Changes in CWMP M&E Monitor livelihoods ú Equity issues ú Include livelihoods monitoring benefits for the poor and evaluation ú M&E focussed on soil and water monitoring ú Include indicators for monitoring access of the poor ú Targeting issues to project services and ú Livelihoods impacts not visible resources Monitor improved ú M&E focussed on soil and water ú Include biodiversity indicators ecological conditions in monitoring M&E focussed on soil and monitoring approach watersheds and water monitoring Monitor participation of ú The poor have not been ú Indicators and approach to the poor in the project sufficiently consulted and monitor participation during involved in project planning, planning, implementation and implementation and monitoring monitoring Improve the quality and ú Existing data have not been ú Identify and use existing efficiency of data shared between departments sources of information collection, handling and ú Limited information flow bac k to ú Define key indicators and analysis lower tiers focus of analysis for different levels ú Analysis weak and confined to higher levels ú Improve feedback of information Increase stakeholder ú Stakeholders have not been ú Develop approaches to enable participation sufficiently consulted and stakeholder participation in involved in M&E M&E ú Issues of farmers participation ú Develop approach to participatory monitoring at the community level M&E system should be ú Size of project area ú Budgeted costs for M&E cost effective ú Complexity of watershed ú Effective sampling strategy 3.4 M&E framework development The CWMP M&E system has been developed on the basis of the previous Loess M&E system. The M&E system developed for CWMP shows some key characteristics which make it differ from pervious Loess M&E: ú Impact levels were defined based on a watershed concept; ú The importance of linkages between environmental and poverty is reflected in the indicators framework; 31 China Watershed Management Project Final Report (June 2006) ú Stakeholder participation is a key element of CWMP, and platforms for stakeholder communication and participation have been created on different M&E levels. CWMP M&E also differs from previous Loess M&E because of its emphasis on impact monitoring which will be done on fours levels: - On the farm level, farmers, technicians and project staff will monitor changes of land use practices and improvements of soil conditions. Methods for participatory and field based monitoring of land improvement are well established and should be explored under CWMP. - On the community level, project staff will monitor project implementation process and changes of livelihoods and local environment together with the community, using participatory methods. Participatory methods for poverty monitoring and monitoring of biodiversity have been introduced during field exercises, and the project should build on these experiences. - On the watershed level, sets of key performance indicators have been defined which will help to monitor changes in the physical and socio-economic conditions of watersheds. The key challenge for the project will be to develop a practical definition of "watershed" that combines hydrological and administrative criteria and to identify indicators and data sets that will apply within that context. - On the river level, the project will use existing indicators to monitor "healthy rivers" in the sectoral context. Following on from the review of monitoring and evaluation (M&E) systems undertaken in Task 2 of the China Watershed Management Project (CWMP), the project team began a process of specifying and designing indicators for the development of a system to assess the impact of Loess 2, with reference to environmental improvements and poverty reduction. The team carried out participatory fieldwork in Huanxian, Kontong and Jingning Counties, aiming to identify community-relevant indicators defining poverty locally, and assessing the impact of Loess 2 component activities in the areas of livelihoods, participation and gender. Working with these groups, the team devised a set of criteria to define poor households and villages, and then developed these within a sustainable livelihoods framework. Based on these criteria, they then developed a set of general indicators, as a basis for assessing the livelihood impacts of Loess 2. The M&E framework is described in detail in Chapter 5. 3.5 Learning and capacity building The M&E system was developed through a process of stakeholder consultation. A series of workshops has been organised with project staff and line agencies at all levels which provided opportunities for discussion, review and learning. The process has helped CWMP to raise awareness and build capacities on M&E-related issues; it has enabled project staff - To understand key M&E concepts and to distinguish between different levels of M&E. 32 China Watershed Management Project Final Report (June 2006) - to explore the institutional and technical dimensions of an M&E system; - to adopt a critical attitude on their own work as basis for learning and improving; - to learn and apply new methods for M&E; - to understand the importance of stakeholder participation in M&E; and - to embrace new approaches to monitor outcomes and impacts in CWMP. Table 3: Overview - Building capacities in the process of M&E development Activities Contents Outcomes February 2005 · Introducing overall framework of M&E · Looked into the entire framework system and main content of the M&E Project kick-off system meeting · Reviewing M&E work of Loess 1 and 2 · Reached agreements on · Discussing the contents to be included shortcomings of M&E in the in the new M&E system previous Loess 1 and 2 · The project staffs accepted M&E of sustainable livelihood as a component of M&E in CWMP May 2005 · Introducing project objectives, · Better understand about the activities, impacts, and the logic purposes of M&E indicators Indicator relationship between project M&E and Development · Know and understand how the M&E indicators Workshop indicators developed under · Introducing sustainable livelihoods different frameworks reflect the framework (SLF) and pressure ­ status project outcomes and impacts ­ reaction (PSR) framework of environmental analysis · Preliminarily understand the importance of poor households' · Discussing about M&E indicators and participation in M&E who will participate in M&E Developing · Joint field work for developing · Understand farmers' perceptions indicators in the indicators on poverty and on poverty and environment field environment in the field August 2005 · Reflecting in combination with · Deepen understanding about the previous project poverty impacts, and importance of the participation of Training on introduce participation idea communities, especially poor participatory households, for ensuring success of impact · Introducing PM&E tools water conservancy projects. assessment · Practicing, reflecting and summarizing · Understand about rationale of PM&E tools participatory approach; · Basic understanding and skills for handling the approach, common tools and methods for participatory monitoring September 2005 · Introducing methods for qualitative · Understand and preliminarily M&E of information command the method for Training on qualitative analysis of M&E integrating and · Introducing methods for qualitative information, and the methods for analyzing and quantitative M&E of information integrating and analyzing information information from different sources Field-testing · Arrange and supervise surveys on · Understand approach of technical 33 China Watershed Management Project Final Report (June 2006) Activities Contents Outcomes Indicators poverty and biodiversity survey organisations · Critical assessment of indicators and methods September 2005 · Displaying, introducing and discussing · Have a complete understanding overall framework of M&E system about the overall framework of Project M&E system summary · Displaying, introducing and discussing meeting the itemized and finalized M&E · Clarify the roles and specific jobs indicators, and M&E levels of staffs from different levels of PMOs in the M&E system 3.6 From Loess Project to CWMP: A major shift in M&E Beginning with the M&E systems review, CWMP has taken steps to transform the previous Loess "implementation monitoring" into a "results-based monitoring" system. The previous Loess M&E was heavily focussed on monitoring inputs, activities and outputs; for CWMP, a M&E system has been developed which will be appropriate for monitoring outcomes of watershed management. The following table illustrates the key elements associated with those different types of M&E systems. Box 6: From implementation monitoring to results monitoring Implementation monitoring Results monitoring 11 ú Description of the problem or situation ú Baseline data to describe the problem or before the intervention situation before the intervention ú Benchmarks for activities and immediate ú Indicators for outcomes outputs ú Data collection on inputs, activities, and ú Data collection on outputs and how and immediate outputs whether they contribute toward achievement of outcomes ú Systematic reporting on provision of inputs ú More focus on perceptions of change among stakeholders ú Systematic reporting on production of ú Systemic reporting with more qualitative outputs and quantitative information o the progress toward outcomes ú Directly linked to a discrete intervention ú Done in conjunction with strategic partners (or series of interventions) ú Designed to provide information on ú Captures information on success or failure administrative, implementation, and of partnership strategy in achieving desired management issues as opposed to broader outcomes. development effectiveness issues 11 Jody Zall Kusek, Ray C. Rist: Ten steps to a results based monitoring and evaluation system. The World Bank 2004. 34 China Watershed Management Project Final Report (June 2006) The major shift for CWMP, however, was to adopt a watershed concept for understanding and monitoring environmental poverty changes. The idea of watershed M&E is described in the following Chapter 4. 35 China Watershed Management Project Final Report (June 2006) 4 Towards an Approach to Watershed M&E 4.1 How to define a watershed? A watershed, or catchment, is an area that drains to a common point. More specifically, it supplies water by surface or sub-surface flow to a given drainage system or body of water, be it a stream, river, wetland, lake or ocean. The watershed is the product of the interactions between land and water, particularly its underlying geology, rainfall patterns, slope, soils, vegetative cover and land use. Water - its availability and its flow - is a critical determinant in the various production functions in a landscape, especially since it is open to interference by human agency. The water moves through a network of drainage pathways, underground and on the surface, which converge into streams and rivers. Every stream, tributary or river has an associated watershed, and small watersheds join to become larger watersheds. Watershed boundaries follow major ridgelines around channels and meet at the bottom, where water flows out of the watershed (stream or river). The hierarchy of watersheds is formed as water drains down a watershed or catchment from a number of zero-order watersheds and combines to form a small sub-catchment, a number 12 of which combine to form a branch, a tributary, etc. (see Figure below). Figure 2: Typical Hierarchy of Watersheds at Farm and Village Levels Project watershed, often covering several villages Tributary River Village community, situated in small watershed Farming Plots located in micro-watershed 12 Adapted from: John Dalton. CWMP Best Practices Review. October 2005. 36 China Watershed Management Project Final Report (June 2006) While the nature of watersheds seems obvious and can be easily observed in the field, definition and delineation is difficult in practice. Delineation of watersheds can be carried out using a topographical map that shows stream channels. However, the resources involved in such mapping are considerable, and generally only major watersheds have been mapped. A further difficulty is that watershed boundaries may not follow administrative boundaries (which are more likely to be lines along the bank of 13 the river than along the top of the watershed) . The term "Project watershed" combines the idea of hydrological watershed with an "administrative" watershed. For practical application, the natural watershed is combined with and area for easy administrative coordination and 2 2 management which normally covers 5-30 km and a maximum less than 50 km . Another difficulty lies in the identification of different levels of watersheds. Although, MWR uses a terminology of "micro, meso, and macro" watersheds, there is no exact definition to define these different watershed levels. The boundaries of a watershed are the mountains, hills, and other high points where land slopes toward the body of water. A small watershed is 2 2 a small natural watershed of normally 5 to 30 km and at maximum less than 50 km . It flows into a small stream forms part of a big watershed, which in turn forms part of a larger 2 watershed. A "micro watershed" comprises an area of less than 1 km . 4.2 Objectives of Watershed Management A watershed approach can be a coordinating framework for management that attempts to focus public and private, community and individual efforts toward addressing high priority land and water-related issues within the hydrologically-defined geographic area. Watersheds are considered the unit of management for many natural resource-related issues including land degradation, water conservation, non-point source pollution, etc. Watershed management is simultaneously a technical, social and economic undertaking. From a technical perspective, it involves reducing soil erosion, promoting vegetative cover, and harnessing rainwater resources. From a socio-economic perspective, it involves coordinating the actions of numerous land users in the watershed who may have multiple and possibly conflicting objectives. In its broadest sense the objective of watershed management is to rehabilitate and sustain the ecosystem services, the productivity of the natural resource base and the well-being of all inhabitants living within the watershed boundaries, including its natural biodiversity and the social and economic well being of its rural and urban communities. In this context the success 14 of watershed development projects can be rated against some general criteria: 13 Corn L. 1993. Ecosystems, Biomes and Watersheds: Definition and Use. CRS Report 93-655 14 John Dalton. CWMP Best Practices Review. October 2005. 37 China Watershed Management Project Final Report (June 2006) · Hydrological capacity: The capacity of the watershed to manage the decent of rainwater without destruction or damage to the productive capacity of its natural resource base. · Water yield capacity: The effective utilization of incident rainfall to sustain ecosystem services, e.g. through controlling run-off and soil erosion (by dams and correct land uses) and by maintaining water quality through reduced sedimentation and point source pollution. · Productive capacity. The effective utilization of sunlight for primary (photosynthetic) production, e.g. through maximizing vegetation cover. · oil Capacity of soils to sustain the biosphere, e.g. through s conservation and control of soil erosion. · Managing the biotic capacity is largely achieved through controlling soil erosion and soil fertility decline by matching land use to land capability, re-cycling nutrients and maintaining soil health and organic carbon (e.g. by composting and minimum tillage and controlled grazing). These general criteria indicate the inherent complexity of watershed management and its integrated and holistic nature. Watershed management activities must cover a wide range of disciplines (e.g. hydrology, ecology, engineering, forestry, agriculture, etc) as well those associated with managing highly industrialized human settlements and burgeoning urbanization (e.g. sociology, public administration, etc). The complexity of watershed management is reflected in the indicators that are used to monitor watershed projects. They usually cover a wide range of disciplines and approaches in a comprehensive approach to cover major dimensions of ecological change and human development. 15 Typical indicators to monitor watershed projects include : ú Soil erosion ú Measures taken to arrest erosion ú Groundwater recharge ú Soil moisture retention ú Agricultural production ú Productivity of non-arable lands ú Household welfare 4.3 Environment-Poverty Linkages Watershed development is rural development in the context of watershed management. However, to achieve both rural development and natural resource management outcomes 15 JKerr and K Chung: Evaluating watershed management projects. CAPRI working paper no 17. August 2001, p. 23 38 China Watershed Management Project Final Report (June 2006) watershed development can only be implemented in collaboration with the local resource managers. Watershed development necessitates close collaboration with farmers and their 16 rural communities. Upper portions of river basins, catchments or "watersheds" are by their nature normally the more rugged, isolated, inaccessible and least developed. The rural communities in these areas most often comprise the rural poor including various cultural minorities who have retreated to the more isolated areas. Strong linkages between poverty and environment are obvious in the loess area; however, few projects have developed an explicit strategy to address those linkages. Watershed development projects assume that environment-poverty linkages will be addressed because certain environmental conditions can be important causal factors for poverty. Previous work on environment-poverty linkages within the loess area has in fact found that even those xplicit objective of reducing poverty have had some impact projects that did not have an e 17 through environmental interventions . However, while linkages between poverty and environment usually work both ways, project are usually designed to address them either from an environmental or a poverty angle only. Environmental projects could be much more effective if they were addressing those linkages rather than focussing on measures for direct ecological improvements only. For example, planting trees or protecting forests without providing fuel wood or fodder resources for the poor will not help to increase vegetation cover within the watershed. Similarly, it would be insufficient to construct terraces for control of soil erosion, if the poor don't have the resources to recover soil fertility; at the same time, if farmers could be motivated to adopt more sustainable land use practices and, for instance, increase ground coverage throughout the year, loss of surface soil could be reduced significantly. CWMP provides an opportunity to understand how these environment-poverty linkages work in the loess plateau context and to develop strategies to enhance positive impacts (and reduce negative ones) for more sustainable poverty reduction in within an extremely fragile environment. Using its analysis of environment-poverty linkages, the project may want focus on interventions that have a strong poverty impact (without turning our project into a pure poverty reduction project) and feed back positively on the environment. Environment-poverty linkages need to be monitored to better understand the dynamics of poverty and processes of environmental change. Key indicators that help to monitor environment poverty linkages could be the following: 16 John Dalton. CWMP Best Practices Review. October 2005. 17 J Taylor report on environment-poverty linkages in the Loess area 39 China Watershed Management Project Final Report (June 2006) egradation and equity issues Indicators to monitor reversal of poverty-linked resource d 18 include : ú Reduction in run off and soil loss ú Reduction in biotic pressure on forest watersheds. ú Rehabilitation of degraded watersheds ú Improvement in base flow of treated watersheds ú Protection of land and rural infrastructure ú Availability of fuel, fodder and commercial grass ú Employment generation ú Increase in crop and milk production. 4.4 Focus of watershed M&E The connectivity of the stream system (physical connection between tributaries and the river, surface and groundwater, and wetlands and water) is the primary reason for doing M&E at the watershed level. Because water moves downstream, any activity that affects the water quality, quantity or rate of movement at one location can affect locations downstream. For this reason, everyone living or working within a watershed has an interest in ensuring good 19 watershed conditions . Therefore, watershed projects have numerous distinguishing features which have important 20 implications for both project implementation and impact assessment : ú Spatial inter-linkages and externalities which are related to the flow of water. These will require coordination or collective action, hence the need for stakeholder participation and inclusive and informed decision-making. ú Multiple objectives, dimensions and determinants of watershed projects which are related to the wide variety of watershed development contexts. Effective monitoring of participatory watershed development is made difficult by the multiple objectives of programmes and trade-offs between environmental protection and short-term 21 productivity gains . ú Long gestation and difficulty in perceiving project benefits. Some watershed projects may have short term effects, but all watershed projects have long term impacts, some of which may be difficult to evaluate or even perceive. As a result, it is difficult 18 Grewel et al 2001 19 Water stewardship education programme training guide , Oregon State University 20 John Kerr and Kimberly Chung: Evaluating watershed management projects. CAPRI working paper no 17. August 2001 21 DFID Key Sheets on resource management 5: Participatory Watershed 40 China Watershed Management Project Final Report (June 2006) to know what conditions would have prevailed in the absence of project interventions. As a result, it can be difficult to pinpoint the specific contribution of a watershed project in 22 improving land management, and it can be difficult to compare across projects . Managing and monitoring of projects for impacts in the context of a watershed is a challenging task and requires a comprehensive understanding of the natural systems and the impacts of human activities. Watershed monitoring plays an integral part by tracking progress towards watershed objectives, while collecting more environmentally relevant data than 23 traditional compliance monitoring . Watershed M&E system differs from other project M&E systems because it needs to provide information on ú Socio-economic and environmental impacts of watershed-based interventions ú Interactions between people and environment in the context of watershed ú Long-term changes within the watershed A well functioning and performing watershed M&E system should be informative, intelligent 24 and interactive . ú Informative about changes in the condition of key natural resources as well as about socio-economic changes. Indicators for monitoring CWMP need to reflect the need for information required by different stakeholders on important changes of environment and poverty status within the watershed. ú Intelligent in that it identifies the causes of change and suggests appropriate responses by key stakeholders for fixing the problem. CWMP would need to understand causes and linkages of environment- poverty change at different management levels in order to develop corrective actions. ú Interactive that it brings the key stakeholders together to review achievements and shortcomings which will require corrective actions. Stakeholder participation will be an important characteristic of M&E in CWMP. 4.5 Piloting M&E in CWMP CWMP is expected to contribute lessons on integrated watershed management to possible later Loess Project phases. The M&E system that has been developed for CWMP would be 22 John Kerr and Kimberly Chung: Evaluating watershed management projects. CAPRI working paper no 17. August 2001 23 Louis C. Regenmorter et al. Development of a monitoring program to support the Rouge River Watershed Management Plan. 24 See: Ujjayant Charovorty, et al. policy relevant monitoring systems for natural resource management. 41 China Watershed Management Project Final Report (June 2006) piloted within the context of that project, but with the perspective of being rolled out throughout the Loess Project area at a later stage. The parameters of this M&E system have therefore been defined to cater for large-scale implementation in loess watershed rehabilitation projects. 42 China Watershed Management Project Final Report (June 2006) 5 M&E Framework for CWMP 5.1 Conceptual Framework for M&E As described in Chapter 3.6 there are two main types of monitoring: implementation monitoring and results monitoring. Both are important in tracking results. Implementation monitoring tracks the means and strategies (inputs, activities and outputs included in annual plans) used to achieve a given outcome. Results monitoring focus on the outcomes and impacts a project wants to achieve. The following figure illustrates the different levels of M&E: Figure 3: Levels of Monitoring ú Long ­term, E.g. environmental or Goal widespread poverty status; (impacts ) changes in environment-poverty watersheds linkages results ú Immediate effects E.g. improved Outcomes on target productivity of land; beneficiaries land covered by grass ú Products and E.g. terraced land, Outputs services provided farmers trained, by the projects trees planted i mplementation ú Specific tasks to E.g. training provided Activities transform inputs by technicians; to outputs terracing ú Financial, human, E.g. funding, tools, Inputs and material training material and resources staff Loess II had a strong focus on implementation monitored; only few indicators have been developed to monitor project outcomes, and it was difficult to follow the effects of project interventions through different levels of outcomes and impacts to the project. This assignment has therefore focussed on developing indicators to monitor project outcomes and impacts in a systematic way. 43 China Watershed Management Project Final Report (June 2006) The nature of watersheds implies that impacts follow a chain of cause and effects along the flow of water; the status of the resources that are managed on different levels changes as a result of those effects and with it the status of livelihoods, depending on those resources. The indicators framework developed for monitoring the environment -poverty status of watersheds in CWMP distinguishes between four main impact levels: - Farmers: land use systems and soil conservation methods that will affect the condition of resources; - Communities: livelihoods that will impact on the habitat of a small watershed; - Small watersheds: natural resources that are used by a number of communities and managed by different stakeholders; - Tributary rivers: waters flowing into the Yellow River which are used by a number of stakeholders, settled in several administrative units Figure 4: Levels for monitoring impacts in CWMP Level Farmers Community Watershed River Impact chain Resources managed Farm plots Agricultural land, Land and water Land and water forest land, grass resources within a resources feeding land, drinking water small watershed into the r iver and irrigation facilities Management unit Household Natural villages Administrative Counties villages Motivation Sustainable land use Sustainable Resilient watersheds "Normal" river through technology livelihoods and through joint ecology development and environment through management of investment into equitable access to watershed resources resource base resources Key indicators ú Reduce soil ú Increase ú Reduce poverty ú Reduce erosion livelihoods assets sedimentation ú Improve ú Improve soil ú Improve biodiversity ú Reduce flooding fertility livelihoods outcomes ú Reduce soil ú Improve land erosion productivity ú Reduce environment - ú Improve multi- linked poverty department participation ú Reduce vulnerability Management of resources at each level follows its own rationale, depending on how use of resources is controlled and decided. Farming households and communities are the natural unit for managing resources and livelihoods. Watersheds and tributary rivers, however, are usually managed by different stakeholders and, unless an explicit watershed management approach is adopted, their behaviour and interventions are not coordinated for a shared 44 China Watershed Management Project Final Report (June 2006) management objective. However, stakeholder behaviour and resource use at each level will ill create impacts further down; impacts w aggregate at higher levels. The M&E framework links impact levels with the corresponding levels of monitoring. The overall status of the watershed could only be improved through behaviour change at all levels, starting from farmers and involving the key stakeholders who are involved in the use of small watersheds. The framework also helps to identify and observe the close linkages between environment and poverty at different levels and to refine the project strategy accordingly. ú Sustainable land use is at the core of integrated watershed management, and improving the sustainability of land use practices is thus an integral target. Land use practices should therefore be monitored and this needs to be done in close cooperation with farmers. ú Communities settling in small valleys are the basic unit within the project area where the close interactions between livelihoods and environment become manifest and can be observed. Typically, this would be one or several natural villages. This is the level at which environment-poverty linkages and vulnerability indicators can be closely observed and monitored. ú The small watershed is the unit where interactions between land and water- geology, rainfall, slope, soils, vegetation, past and present land use as a result from human interventions, become evident. Processes of environmental degradation and poverty can be analysed in the context of watersheds. ú The "normal river" ecology reveals the final outcomes of interactions between people and the environment at all levels. It is the main concern of users at the lower reaches and central level government. 5.2 Indicators The structure of indicators has been developed along the four levels. Methods for monitoring those indicators have been selected that will help to maximise both stakeholder involvement and quality of data for each of these for these levels: ú Participatory module at the lowest levels which will help to ensure that primary stakeholders are sufficiently consulted at all stages of this project: including participatory monitoring of sustainable land use; ú Integrated monitoring of livelihoods and environment at the community level, using participatory methods. ú Specialised surveys at the watershed level, using the technical expertise available in 45 China Watershed Management Project Final Report (June 2006) different institutions to collect data to monitor changes in the environmental and poverty status of the watershed. ú Technical monitoring of indicators related to river ecology Table 4: Structure of indicators on environment and poverty Impact level Indicators Healthy river ecology Indicators on river ecology Soil and water Resilient watersheds Poverty indicators Biodiversity indicators indicators Livelihoods indicators Biodiversity indicators Soil and water Sustainable developed with developed with indicators developed communities communities communities with communities Sustainable land use Indicators to monitor (process and impact of) technology change Over the last months, indicators have been developed, discussed and agreed along this structure. Through the discussions on indicators, the project has achieved an understanding of the different types of indicators and their functions within the core group of project staff working on M&E in the PIO and the four Gansu PMOs. The Sustainable Livelihoods framework has been used as a checklist to help ensure that the selected indicators cover a balance of the factors affects the livelihoods of poor rural people. The SL framework takes an holistic view of poverty reduction. In essence, it says that people have a bundle of assets ­ their natural assets (land, trees, livestock, etc), their human assets (education, health [ability to work], skills, etc), their social capital (social networks, relationships with more powerful people, etc), their financial capital (savings, access to credit, etc), and physical capital (farm machinery, infrastructure, etc) ­ they need some of ALL these things to make a living. There needs to be a certain balance of these assets. However, how they can use these assets productively is affected by factors outside their direct control ­ the weather, price fluctuations, etc. This is their vulnerability context. Their ability to convert assets into a better livelihood is also affected by legislation and tradition, the way government and markets work, etc. Within these contexts, the poor must make a strategy for their livelihood ­ often their options are limited. The outcome is measured on a number of factors which indicate whether they are becoming less poor and have more sustainable livelihoods. The theory of the SL approach is that projects can work on a number of fronts ­ reducing vulnerability, improving one or more of the assets, or working to make policies, markets, etc more favourable to the poor. Indicators have been piloted and the information requirements tested before the project will move to full implementation during the next stage. Piloting of indicators has supported 46 China Watershed Management Project Final Report (June 2006) learning on what works and what does not. CWMP has now produced a list of indicators for 25 the different aspects and levels that will be monitored. The following key performance indicators have been selected to monitor long-term changes at the watershed level. They can be used to compare performance of different watersheds over a longer period. Table 5: Key performance indicators to monitor long-term changes at the watershed level Soil and Water Poverty Biodiversity ú Soil fertility ú Reduction of poverty rate ú Vegetation cover ú Soil erosion control ú Per-capita income ú Diversity of flora ú Soil quality ú Reduction of food shortage ú Richness of species in planted forests ú Availability of fuel from fuel- ú Harnessing benefit/ wood plots ú Diversity of avifauna sedimentation control ú Availability of grassland for ú Richness of insect species ú Water quality livestock In addition, the project will need to identify (and monitor) environment -poverty linkages within a watershed. The following indicators are expected to vary significantly across villages within one watershed. Table 6: Indicators to monitor environment-poverty linkages within a watershed Soil and Water Poverty Biodiversity ú Soil conservation measures ú (Per capita) availability of ú Variety of tree species on terraced land household plots ú Soil quality ú (Per capita) availability of ú Variety of crop species ú Soil fertility grass land for livestock production ú Availability of medicinal plants ú Soil erosion from wild harvest ú (Per capita) grain production ú Access to safe drinking water ú Migrant labour The two types of indicators introduced above will help the project to assess the status of watersheds and villages and to monitor long-term changes of livelihoods. For identification of project impacts, an additional set of indicators will be needed which show clear linkages between project interventions and outcomes. Project outcomes are expected to vary between villages and households. Indicators can be monitored on the community and household level. 25 An overview of all indicators to be monitored in CWMP is included in the Annex. 47 China Watershed Management Project Final Report (June 2006) Table 7: Indicators to monitor direct project outcomes Soil and Water Poverty Biodiversity ú Soil conservation measures ú (Per capita) availability of ú Variety of tree species on newly newly terraced land planted fuel wood plots and ú Soil fertility on newly terraced orchards land ú (Per capita) availability of newly planted fuel wood plots ú Variety of crop species on newly ú Soil erosion terraced land ú Crop yields on newly terraced ú Quality of drinking water land ú Variety of livestock species introduced by project ú (Per capita) water consumption Although projects will be implemented for entire villages, access to project services is expected to vary across households. We therefore propose the following indicators to monitor access to project services by poor and marginal groups within the village. Table 8: Indicators to monitor access to project services by poor and marginal groups Soil and Water Poverty Biodiversity ú Soil conservation measures ú Number of very poor ú Variety of tree species on newly implemented by very poor households that have obtained planted fuel wood plots and households newly terraced land orchards ú Number of very poor ú Variety of crop species on newly ú Soil fertility on newly terraced households that have been terraced land land owned by very poor provided with water cisterns households ú Very poor households maintain ú Number of women that have stable income from medicinal ú Soil erosion increased livestock production herbs harvested from the wild under the project ú Percentage of fuel from fuel wood plots increased for very poor households These indicators provide a good start, and they should be tested for establishing baseline information in CWMP. It is, however, likely that the need for more or different indicators will arise, once CWMP has defined its intervention strategy. In that case, we suggest that CWMP 26 assesses the quality of proposed indicators, using the following checklist: 26 It might also be that CWMP decides to maintain indicators that have been found useful in Loess II; in that case, the same quality criteria should be applied. 48 China Watershed Management Project Final Report (June 2006) Box 7: Criteria to assess quality of indicators 27 Is the indicator þ As direct as possible a reflection of the outcome itself? þ Sufficiently precise to ensure objective measurement? þ Calling for the most practical, cost-effective collection of data, but relatively unaffected by other changes? þ Disaggregated as needed when reporting on diverging trends across watersheds, villages, households and gender? þ Relevant for monitoring the key aspects the project wants to achieve? 5.3 Data collection methods A major shortcoming of the previous Loess M&E system was the insufficient quality of data. Lack of data, low frequency of data collection or use of unreliable data have been mostly attributed to the lack of technical qualification, but also to inappropriate data collection methods. The new M&E system will therefore put significant efforts in improving data quality. Box 8: Criteria to a assess quality of data collection 28 Key criteria for collecting quality performance data are: þ Reliability: the extent to which the data collection approach is stable and consistent across time and space þ Validity: The extent to which indicators clearly and directly measure the performance intended to be measured þ Timeliness: - Frequency: how often are data collected? - Currency: how recently have data been collected? - Relevance: are data available frequently enough to support management decisions? The revised M&E framework aims at improving reliability, validity and timeliness of data. Data collection methods have been identified and tested for each indicator with emphasis on primary information collected in the field; in addition, it has been agreed that parts of the data collection and reporting will be outsourced to specialised survey organisations which have the capacities and methodology available to ensure data quality. 27 Modified after Kusek and Rist, 2004. 28 Modified after Kusek and Rist, 2004 49 China Watershed Management Project Final Report (June 2006) Data source Data Data collection Who collects Who collection frequency data reports method data Tributary river Monitoring Measures taken Regular Monitoring Monitoring stations at established stations stations stations with Provincial PMO Small Watershed Sample Soil and water Beginning, middle Survey Survey catchments survey and end of organisation organisation project Sample villages Poverty survey Beginning, middle Survey Survey and households (questionnaire) and end of organisation organisation project Sample Biodiversity Beginning, middle Survey Survey watersheds and survey and end of organisation organisation communities project Community Communities and Participatory Annual, as part of PMO with County PMO livelihoods and households methods annual project communities habitat planning Land use and Farmers Participatory Ongoing, as part Technicians with Technicians land methods of implementation farmers improvement process Detailed description of data collection methods included in M&E guidelines and ToR for survey organisations. (see Annex) 5.4 Sampling strategies In addition, it is important that data sources are properly sampled for all levels. Sampling would need to consider both aspects of coverage required to achieve sufficient representation of socio-economic and environmental diversity and financial implications. The following principles should be applied: - For participatory monitoring, CWMP should aim at covering a maximum of farmers and communities. It is understood that participation during all stages of the project will be a key feature of CWMP, and participatory monitoring would thus be one key aspect of participatory project implementation. - For watershed monitoring, small watersheds should be sampled to represent different types in terms of poverty and environmental conditions. The project should include at least one very poor and remote watershed per county. - Purposed sampling of households and villages (rather than random sampling) is recommended for specialist surveys: Households should be sampled in a way that poor and very poor households are sufficiently represented, meaning that the number of households sampled in the two categories "very poor" and "poor" should represent 50 China Watershed Management Project Final Report (June 2006) (at least) the actually proportion of very poor and poor households in the population of the administrative village. Villages should be sampled along the major zones of the watershed. Field work carried out during the phase of indicators development illustrates the importance of identifying poverty types as a basis for sampling. The following wealth ranking of poor households highlights the differences between poverty groups that need to be sufficiently reflected in the survey. Very poor households need to be covered to ensure that they will benefit from the project as well. Box 9: Wealth Ranking of Rural Households in Mahuang Village, Kongtong County Type Number of Key Characteristics Households Ordinary Households 50 o Per capita grain production 300kg. o Per capita net income over 700 Yuan. o Wage incomes received mostly from migration Poverty Households 70 o Per capita grain production 320kg. o Per capita net income over 300 Yuan. o Do not receive wage income from migration. o Limited cash income Extremely Poor 7 o Insufficient labour for cultivation Households o Have large number of dependent family members ­ young children, daughters. o Lack cash income. o Dependent on food aid Households Living on 4 o Do not have sufficient food. Social Welfare o Have disabled family members o Single parent households o No cash income o Dependent on food aid Poverty types are only one aspect that should guide the sampling strategy for M&E. In addition, where geographical sampling is applied it will be important to cover the different zones within a watershed. Typically, a watershed includes the three zones, each of them having distinct bio-physical properties and related environment-poverty linkages: - Upper watershed: Usually the poorest communities living on a degraded resource base. - Middle watershed: Communities whose resource use will critically affect the status of the watershed. - Lower watershed: Usually the communities who are most affected by unsustainable resource use in the middle and upper watershed. 51 China Watershed Management Project Final Report (June 2006) During the field study in Kongtong, villages have been grouped according to environment- poverty types in the context of a watershed. Box 10: Wealth Ranking of 24 Villages in Dazhai Township of Kongtong County Group Names of Villages Characteristics I : Located in the Zhaoyuan, Baitu, Dazhai, Per capita grain production is approximately 750kg; Per plateau areas, over Kangqiu, Zhangzhuang, capita net income is over 1,150 yuan; the materials of the 30% of the arable Gaoyuan, Laozhuangwa, houses are brick and wood, or brick and concrete; the per land is plateau or Guihua, Tongzui, Xiaoma, capita area of accommodation is over 15 square meters. terraced land Liyuan Over 80% of the population has reliable access to drinking water.. II: Small areas of Liugou, Qingshuiling, Per capita grain production is about 500kg; per capita net plateau and Suojia, Mujia, Huangjia, income is about 700-1,150 yuan, and the housing situation terraced land within Youfang, Liujia is poor, with few houses of brick construction. Over 50% the village of the population has reliable access to the drinking water. . III: Located on the Panling, Tugezhuang, Per capita grain production is about 300kg, and per capita fringes of shrub and Leishen, Mayingzi, Yulin, net income is less than 600 yuan. Most of the population mountain areas and Caozi has difficulties in accessing drinking water. Over 90% of the rural households live in cave dwellings. The above example illustrates the importance of sampling villages within a small watershed to cover different environmental poverty types. In bigger watersheds, administrative villages would be an appropriate unit for sampling; in smaller watershed where administrative villages may stretch into more than one watershed, natural villages would be the appropriate unit for sampling and analysis. Geographical sampling within a watershed context will be the base for collection and analysis of data to be processed within the project GIS. Box 11: Sampling of villages within a watershed Recommended steps for sampling villages within a watershed: 1) Divide watershed into (at least 3) zones, according to predominant environmental and geographical features: e.g. upper, middle and lower watershed. 2) Identify number of villages within each zone: e.g. 11 villages in Zone I; 7 villages in Zone II; 6 villages in Zone III. 3) Select 20% villages within each zone: e.g. 2-3 village in Zone I; 1-2 villages in Zone II; 1-2 villages in Zone III. 4) Collect data according to defined methodology within each village: e.g. select households for household survey; convene community meetings, sample field plots. 5) Maintain separate data records for each village to allow for analysis of different environment- poverty types. 6) Data can be integrated into project GIS at later stages. Aside from considerations around the representation of different socio-economic and 52 China Watershed Management Project Final Report (June 2006) environmental types, the project needs to collect information from selected groups in a way that it helps to asses possible project impacts. Project impacts are usually assessed through comparisons: - Comparison across time: Comparing a post project situation with the baseline information that has been collected before the project start. The disadvantage of this type of comparison is that it does not help to clearly distinguish changes that have resulted from project intervention from changes that would have happened anyway and can therefore not be attributed to this project, for example increases of cash incomes as a general trend of poverty reduction or a result of market integration. - Comparison between project groups and non-project groups: The approach of comparing groups of villages or households that have benefited from the project with a control group that has been outside of the project would help to identify differing trends of development. The main difficulty is, however, the selection of such a control group. Ideally, all factors affecting the environment-poverty situation would have to be similar for both groups and then kept constant for the control group while the project group only will be influenced by the project. In reality, it will be difficult to simulate such a situation. We therefore recommend the following, "two track" approach for CWMP which would help to combine advantages from different approaches in order to gain a better understanding of impacts under this project. - For monitoring trends of development in addition to achievement of targets, the project should select one "control watershed" per county. Key performance indicators on poverty, biodiversity, soil and water should be monitored for project and non- project watersheds in order to understand extent of changes in the status of the watershed that is likely to happen as a result of the project. - For understanding the causality of changes and hence the impact that can realistically be attributed to the project, information would be collected from project communities and households. Participatory methods would help to understand why certain changes occur and how they affect people's livelihoods within the complex environment of a watershed. In addition, participatory studies would help to explore factors affecting changes which have not been anticipated by the project. The following table presents and overview of sampling strategies applied at different levels. 53 China Watershed Management Project Final Report (June 2006) Table 9: Sampling strategies for M&E Unit Sample size Sampling strategy Tributary river Project province/city/ According to measuring According to measuring points county points Small Watershed Project watershed 2-3 watersheds per county; Cover villages in different parts 20% (natural) villages within of the watershed; cover (include non-project one watershed different poverty groups; cover. watershed as control group) Community Project communities 100% natural villages Cover poor villages and villages livelihoods participating in project in different parts of the (only) watershed Land use and land Project farmers (only) 100% farmers participating in Cover poor farmers and women improvement project farmers 5.5 Reporting People have a tendency to over-estimate the amount of information they need to make decisions. Specification of information needs involves a trade-off between the amount of information required to make decisions, and the amount of information that a decision-maker can practicably read and analyse. All too often, a manager over-specifies his or her information needs, only to find that it is simply impossible to read the reports and absorb the information contained in them. Information needs must be related to levels of management, and selection of indicators should reflect this through the specification of a minimum set of information. More detail is required at the day-to-day operational level, while aggregated and summarised data is used at higher levels. 29 The pyramid framework has been introduced earlier. Its main principle is that information should be increasingly distilled and summarised the further up the management pyramid one 30 progresses . This framework provides a useful guide to the design and use of M&E in hierarchical organisations. It is useful where, as is often the case, too much information is collected in M&E systems, and most people in the M&E system collect or process data, but are not empowered or capacitated to act upon the information. This is in essence what happened in Loess II, with the majority of raw data arriving at the Province level for analysis and interpretation. This places a heavy data burden on the higher levels, whose costs are higher, and leaves them less time for the tasks they are trained for ­ taking management decisions based on good data. The information required by each management level is different; in addition, stakeholder have 29 ITAD. China Watershed Management Project Development of a Monitoring and Evaluation System. Task : Review of Existing M&E Sys tems. May 2005. 30 The monitoring of the MDGs is a good example of this in practice. At the top of the pyramid is small set of indices that signal the key progress towards achieving the MDGs, globally. 54 China Watershed Management Project Final Report (June 2006) their specific priorities for the information they want to get from M&E. The M&E reporting and documentation system would need to consider the different information needs. - Farmers and communities together with technicians need hands-on information to improve implementation process. - Project management staff requires information to support their specific tasks. - Within the administrative system, the higher up the chain of command, the less need there is for extensive detail and explanation. - Stakeholders outside of the project management system would require well- presented information on key aspects of watershed development only. We recommend the following structure of the reporting and documentation system which would combine the need for M&E information both along horizontal and vertical lines. Figure 5: Information pyramid and information flows in M&E PIO/MWR monitor river KPI ecology PIO/PMOs/stakeholders monitor performance of project watersheds KPI Communities monitor changes of livelihoods and environment with PMO as part of the project implementation process KPI Farmers monitor land use and land improvements together KPI with technicians as part of the technology development Documentation and analysis of information on each of the four levels would accommodate the need for information at all management levels and support horizontal integration and sharing within the M&E system. In addition, systematic reporting on key performance indicators will ensure that higher management levels receive the information they require for their purposes. The idea of having key performance indicators is to reduce the amount of information that needs to be reviewed at management levels and to enable informed decision-making on key areas of concern. The concept of performance indicators has been introduced earlier. Typically, in a well-structured M&E system, more and more specific indicators would be monitored at lower levels in support of various management and implementation tasks while 55 China Watershed Management Project Final Report (June 2006) progress on key performance indicators is systematically documented and reported in the upward chain of command. Key performance indicators can and should be used to monitor outcomes and provide continuous feedback and streams of data throughout the project. In addition to using indicators to monitor inputs, activities, outputs, and outcomes, indicators can field a wealth of performance information about the process of and progress toward achieving these outcomes. Information from indicators can help to alert managers to performance discrepancies, shortfalls in reaching targets, and other variabilities or deviations from the 31 desired outcome. In addition, performance indicators are critical in the context of watershed management because they help to document progress on implementation of selected controls 32 and management activities . The following table summarises the reporting system for M&E. Table 10: Reporting system on M&E Level and responsibility Documentation Land use and land improvements: Farmers together with technicians document · Detailed maps, showing land use, land technology change: degradation, improvement measures and soil productivities · Calendars and diaries showing changes of land use and implementation of improvement measures · Recording on samples taken · Records and information available in farmers households for reference and later comparison Farmers and technicians feedback · Reporting on KPI according to defined information for KPI on land use and land formats improvement Community livelihoods and environment Community monitors changes of livelihoods · Detailed maps, showing distribution of and environment resources and project interventions, social maps showing distribution of poverty; biodiversity maps showing habitat of different species. 31 Jody Zall Kusek, Ray C. Rist: Ten steps to a results based monitoring and evaluation system. The World Bank 2004. 32 Louis C. Regenmorter et al. Development of a monitoring program to support the Rouge River Watershed Management Plan. 56 China Watershed Management Project Final Report (June 2006) Level and responsibility Documentation · Calendars showing productivity and food shortages over time; also: access to water, incidence of diseases, access to non- agricultural income etc. · Tables to assess achievements of targets. · Detailed maps, showing distribution of resources and project interventions, social maps showing distribution of poverty; biodiversity maps showing habitat of different species. · Calendars showing productivity and food shortages over time; also: access to water, incidence of diseases, access to non- agricultural income etc. · Records of discussions and meetings KPI on livelihoods, biodiversity and soil · Reporting on KPI according to defined improvement that have been assessed by the formats community Small watersheds performance Specialist surveys provide detailed reporting · Survey reports by specialist organizations, on poverty, biodiversity, soil and water providing a detailed analysis and indicators background information on survey findings. Survey reports provide information on · Reporting on KPI according to defined poverty, biodiversity, soil and water KPI formats River ecology Technical reports provided by monitoring · Technical report by specialist institutions organization · Reporting on KPI according to defined formats 5.6 Data analysis Setting good indicators and producing quality data will only be part of a well performing M&E system in CWMP. Much will depend on how this information is handled and used within the system and how it is shared with other stakeholders. The M&E systems review has looked at processes of data handling and the quality of analysis. As a result, the project has decided to revise the way it is handling M&E information within the management system and to build capacities for data analysis at all levels. The 57 China Watershed Management Project Final Report (June 2006) institutional framework that has emerged from the discussions over the last months shows the following important characteristics ú Platforms for data collection, analysis and sharing at different levels; ú Building capacities for data collection and analysis through practical application; ú Stakeholder involvement in data collection and analysis at all levels; ú Improved quality of data collection and analysis through involvement of independent technical agencies. ú Increased frequency feedback from primary stakeholders through participatory assessments as part of the project implementation process. The "information triangle" aims at increasing the overall quality and of information and analysis. Better defined roles allow project staff to focus on what they are best at while bringing in additional stakeholders for M&E. Farmers and communities are primary stakeholders and they should therefore asses the quality of project services and the benefits. They are no longer considered to be providers of data only, but they will assume an active role in analysis changes. Independent survey organisations will be selected based on their capacities for data collection and analysis. Figure 6: Information triangle in M&E PIO and PMOs monitor whether progress has been made according to planning and targets have been met. They monitor · Inputs and outputs · Farmers and communities assess Specialist survey organisations · how technologies can be used, provide an independent assessment · who benefits from project of changes over time: services · Poverty · Improvements that have been · Biodiversity The system not only combines the strengths of different stakeholders, but also introduces an aspect of cross-referencing findings from different data sources. For example, aspects of changes that have been overlooked by the (standardised) surveys can be further explored through participatory assessments. . 58 China Watershed Management Project Final Report (June 2006) Box 12: Quantitative & Qualitative Assessment of Poverty compared Participatory Impact Assessment Quantitative Survey Level Communities Small Watersheds Main Assess the effectiveness of project Monitor key performance indicators on Purpose interventions for improvement of environmental and livelihood status at watershed environment and livelihood levels Approach Participatory assessment with project Quantitative survey of sampled villages and communities households indicators Indicators can vary between communities Using pre-defined indicators Done by County PMOs, with some NGO involvement RSO/Specialized Survey Organizations Methods Participatory mapping, group discussion etc. Questionnaire form used Main Case studies of valleys/communities, Quantitative analysis, statistics and report on Outputs qualitative description and analysis the general situation of the environment and livelihood in the watershed Users To be used by farmers and County PMOs to To be used by PIO watershed management realign the project activities project for the purposes of macro decision- making and strategy realignment A major shortcoming of the previous M&E system was that aspects of data provision, data collection, data analysis and data use have been divided among different institutions. Monitoring of environmental indicators has been to a large extent relying on secondary data, where sampling, frequency and quality of data could not be controlled. Primary stakeholders as well as PMOs have been providing data, but not been involved in the analysis of data at all. The revised M&E framework recommends the following structure for data collection and analysis. A major feature of responsibilities within the new framework is that those who are good at analysing the information should also collect it in the field. 59 China Watershed Management Project Final Report (June 2006) Table 11: Responsibilities for data collection and analysis Main contents Who collects Who analyses Who uses information? information? information? Tributary river River ecology Specialized Specialized PIO with MWR institution institution with PIO Small Watershed Poverty, Specialized Specialized PIO with PMOs biodiversity, soil institutions institutions with and stakeholders erosion and PIO sedimentation Community Livelihoods and Villagers with Villagers with Villagers with livelihoods environmental PMO PMO PMO change Land use and Technology Farmers with Farmers with Farmers with land change and technicians technicians technicians, PMO improvement impact on soil fertility, erosion 5.7 Integrating and presenting M&E information For sustainable watershed management, management staff will have to gain a clear picture on how different watersheds are performing in terms of environmental and socio-economic change. This would require a clear way of analysing and presenting data collected from monitoring indicators which is easily understandable to project members and communities is required. A major shortcoming of the previous M&E system was that management did not have the information available they need for taking their decisions or they did not have them presented in a way that they could be conveniently used for management task. A key challenge is the integration of various types of data that are required for watershed management, especially the difficulty of integrating socio-economic and environmental data. Integrated analysis is almost naturally done at lower levels, using participatory methods. Farmers and communities reflect on their livelihoods within the holistic context of their local environment; they are not inhibited by technical specialisations. Therefore, participatory assessments provide a great potential for better understanding environment-poverty linkages within a local setting. At higher levels, more sophisticated approaches will be required to analyse linkages between different indicators. Some techniques have been introduced during 33 the workshop on data analysis. The simplest approach is to collate the different data into a format whereby the changes in key indicators (indices) can be easily compared. 33 See Report on "Workshop on Integrated Data Analysis", ITAD 2005. 60 China Watershed Management Project Final Report (June 2006) 34 The balanced scorecard (BSC) is a technique, developed in 1992 to give managers in business a fast, comprehensive view of the performance of their business. The approach of identifying a small number of key indices, and presenting them together has possibilities, not least in achieving the sought-after balance between social and environmental information. We therefore propose a "traffic-light system" for scoring key performance indicators at the watershed level.For this, indicators would have to be scored in a first step. The following table provides an example on how key performance indicators could be scored Box 13: Scoring key performance indicators on poverty (example) Reduction of Per-capita Households Use of fuel Availability of poverty rate income with food from fuel- grassland for security wood plots livestock (pc) High > 10% >1000Y 100% >50% >5 mu Middle 8-10% per year 600-1000Y 80-100% 20-50% 2-5 mu Low < 8% < 600Y <80% 0-20% 0-2 mu In a next step, those scores would have to be taken forward into a balanced score card or matrix. The table below shows how key performance indicators could be presented within such a system. Table 12: Scoring indicators for different watersheds (example) Watershed 1 Watershed 2 Watershed 3 5 KPI on Poverty status KPI KPI KPI 5 KPI on water and soil KPI KPI KPI 5 KPI on biodiversity KPI KPI KPI 3 KPI on participation KPI KPI KPI Key: low ­ middle ­ high values, as scored for each indicator. "Red" light indicates need for immediate management action. A more sophisticated approach than the score balanced card is that of the `dashboard' which requires use of software for analysis and presentation. Dashboards have an analogy to car dashboards, where the driver is presented with simultaneous information on speed, engine revs, fuel, engine temperature, etc. Dashboards have been developed in many areas of performance assessment, including those based on 34 http://www.balancedscorecard.org/basics/bsc1.html 61 China Watershed Management Project Final Report (June 2006) BSC information. The most relevant to CWMP might be the Dashboard of Sustainability produced by the Consultative Group on Sustainable Development Indicators, based out of the 35. International Institute for Sustainable Development The Dashboard of Sustainability aims to present an integrated view of environmental, social and economic indicators. Two factors affect the utility of the dashboard. The first is the frequency of monitoring ­ dashboards work well when there is change in the underlying data. The Loess M&E system monitors many indicators that are slow to change, and it only measures them once a year, or in some cases less frequently. If dashboards are used, KPI selection will need to account for the fact that the tool is less useful if the `dials' are static for long periods. Secondly, it is understood that the M&E data are stored by the PIO in an MS Access database. Xcelsius is dedicated to MS Excel. Nonetheless, there are many brands of dashboard software that do interact with Access. If the M&E database was converted to a SQL database, then this would increase the options for data manipulation and presentation. Another approach for integrating data could be GIS. CWMP has now decided to develop a GIS for data analysis and presentation. It is recommended that the project should assess 36 existing experiences with use of GIS for watershed monitoring. The integrated analysis of social and environmental information through GIS is not as easy as it may initially be thought. These two data types are different in a number of ways. Social phenomena tend to be more spatially discrete (e.g., people, farms, and political units), though population density, a continuous social variable. The subjects of environmental data often exhibit continuous spatial variation (e.g., weather and soils), though land use is a discrete environmental variable. There are different GIS approaches to dealing with discrete and continuous variable. GISs present data as `spatial units' (i.e. areas) ­ in these units, internal homogeneity is assumed to some degree ­ this presents problems with social data. The level of integration possible in GIS can be addressed at different levels of complexity: ú simple ­ essentially side-by -side visual comparison of maps; ú partial ­ with some manipulation of spatial units to facilitate the analysis of social and environmental attributes of common units; ú full ­ in which models of interactions between people and the environment are produced. For this to occur, social and environmental data must be in a common format, suitable for geo-statistical modelling. As a result, we suggest the following structure and approach for integrating data along the four levels of M&E. 35 http://www.iisd.org/cgsdi/dashboard.asp 36 A good starting point might be to contact the Kunming office of the World Agro forestry Institute who has developed an approach for GIS-based watershed monitoring in a small watershed in Yunnan. 62 China Watershed Management Project Final Report (June 2006) Table 13: Integrated data analysis at different management levels Management Who needs the analysis? Key indicators Methods for integrated objective used data analysis Sustainable land Farmers together with ú Soil Participatory analysis in use through technicians need to know conservation the field, using maps, technology what technologies will help to notes and field ú Soil fertility development and reduce soil erosion and observations. investment into improve soil fertility and ú Yields resource base productivity to further select and develop land use systems ú Crop diversity ú etc Sustainable Villagers together with PMOs ú livelihoods Participatory analysis in livelihoods and need to know what changes/ assets ands the community, using environment improvements in livelihoods outcomes group discussion, through equitable are being brought about by mapping tools, access to project interventions to plan ú biodiversity presentations/ records resources and improve future project ú vulnerability prepared by the implementation: community and ú soil and water household visits. resources ú etc. Resilient PMO, PIO together with other ú poverty Integrating data on KPI watersheds stakeholder want to know through specialised ú Improve through joint whether targets on key methods and software biodiversity management of performance indicators have (dashboard, scorecard, watershed been met in project ú soil erosion GIS) resources watersheds ú water quality ú stakeholder participation ú physical targets ú etc. "Normal" river Yellow river bureau/MWR ú Sedimentation Specialised data analysis ecology needs to know whether the as part of the national ú Flooding projects implemented in the monitoring system upper watersheds actually ú Water quality contribute to improved river ecology. 63 China Watershed Management Project Final Report (June 2006) 6 Stakeholder participation in M&E 6.1 Institutional framework The institutional approach that has been worked out during this assignment aims at: ú Using existing experiences and capacities within the PMO system while at the same time gradually building additional skills and knowledge on new areas from which the overall project management will benefit, e.g. participation. ú Getting stakeholders involved in the monitoring system, with a priority given to those stakeholders who have been identified as important during the stakeholder analysis: farmers, communities, Soil and Water Monitoring Centres (SWMC), Rural Survey Organisations (RSO), non-government organisations (NGOs), Poverty Alleviation Office (PADO), Forestry Bureaus. The quality of monitoring as well as overall project implementation is expected to improve with greater stakeholder involvement. The process of developing an M&E system for CWMP presents a move away from the vertical `silo structure' where PMOs were responsible for monitoring at certain levels with a strong upward flow of information to a more open and networked structure which includes an expanded set of stakeholders around monitoring certain focal areas, e.g. poverty. The new structure involves creating `platforms' for stakeholder participation in M&E at different levels. The advantage of the new system is that it is easy for PMOs to manage, coordinate, and communicate with stakeholders on issues which are `close' to them. Figure 7: Monitoring platforms and stakeholder groups in CWMP 64 China Watershed Management Project Final Report (June 2006) Healthy river SWMC PIO Forestry RSOs Resilient PMO NGOs PADO PMO Sustainable Livelihoods Communities Biodiversity PMO Farmers Sustainable land Soil and water Soil and water Soil and water Intervention Intervention Intervention Intervention s s s s Watershed At the lower tiers (farm plots and natural villages) the PMOs as project implementing agency will closely interact with farmers and communities to monitor the implementation process and resulting changes. Because of their understanding of poverty and participation issues, the involvement of local NGOs and PADOs will add value to the monitoring process. At the watershed level, monitoring of more specialised indicators will bring together stakeholders who have data available already or can provide the methodology to collect and analyse those data in a professional way. Naturally, this would include stakeholders like the RSOs, SWMC and Forestry Bureaus. Monitoring of more specific indicators on river ecology will be done by the SWMC which has the equipment and methodology to monitor those indicators, mostly as part of the routine and long-term monitoring system. 6.2 Rationale for stakeholder participation in M&E The basis for promoting participatory approaches to watershed development and poverty alleviation lies in the principles of subsidiarity (if an activity can be performed effectively at a lower level then it is best done at that level) as well as freedom and responsibility to mobilize individual and community self interest and capability in the task of human development (and 65 China Watershed Management Project Final Report (June 2006) 37 local-level nation-building). Stakeholder participation often peaks during project preparation and planning only; it tends to flatten out during implementation where community participation is limited to provision of labour contributions only. Participation in project monitoring is not that common yet, although it has a number of important benefits, such as improving project quality of projects and strengthening community capacities and ownership. Many projects involve people in monitoring progress and evaluating impacts and are increasingly using M&E for internal learning to improve their work. They see that for maximum 38 benefits, learning needs to happen with diverse groups and people . Watershed systems are complex with multiple uses and users; resources and the institutions that manage them span multiple scales. There are a number of widely accepted arguments for involving farmers in the monitoring and evaluation of watershed and other natural resource management programmes. These may be seen through a spectrum of roles of farmers in M&E from: farmers monitoring their own environment and sharing observations with the project, to community empowerment. Together, these provide a strong rationale for participatory monitoring. · Farmers monitor the state of their own environment and share their observations and knowledge with the project. Farmers have expertise and detailed knowledge of environmental and other processes at work in their own farms and communities. Outcomes of changes in natural resource management practices are incremental and 39 often not immediately observable . Farmers are able to understand linkages between inputs and outputs and provide qualitative measures of indicators which are difficult to measure (e.g. soil loss, erosion, moisture retention). They can also identify and select indicators which will be used for monitoring variables. These may complement those selected by project staff. · Local people assess how project objectives are being met. Farmers may be involved in assessing progress of the project and their own activities against qualitative and quantitative indicators. · For learning and organisational change. Monitoring becomes a dynamic process and is directly linked with planning and taking decisions and corrective action. · Participation can empower the community to take action using the information. Participation is not simply another way to deliver prescribed technological solutions. Commitment to participatory approaches may demand significant changes in the way we think about both the theory and practice of sustainable watershed management. Participation 37 John Dalton. CWMP Best Practices Review. October 2005. 38 Guijt et al. 1998 PLA Notes 28-36, IIED 39 Nanci Johnson et al. CAPrI Working Paper No 10, September 2001. 66 China Watershed Management Project Final Report (June 2006) implies that stakeholders will work together to set criteria for sustainable management, identify priority constraints, evaluate possible solutions, recommend technologies and policies 40 and monitor and evaluate impacts. Using participatory methods for monitoring will help the project to develop its understanding and skills on participation, which need to go into the planning and implementation process. It will also help improve linkages between the different layers of M&E and enable effective contributions from the various stakeholders. A number of factors can help encourage people to participate willingly and effectively in 41 M&E . Some of those which might be considered by the project in establishing a participatory M&E system are: · The perceived benefits (and short-term costs) of M&E · Relevance to the priorities of participating groups · Quick and relevant feedback of findings · Flexibility of the M&E process to deal with diverse (and evolving) information needs · Meeting expectations that arise from M&E, such as providing feedback · Degree of maturity, capabilities, leadership and identity of the groups involved · Local political history, which influences society's openness to stakeholders' initiatives · Incentives to make M&E possible (e.g. pens, books, etc.) Monitoring within CWMP communities should be done by the communities themselves in order to ensure effective monitoring of both environmental and livelihoods indicators. Communities should participate in all stages of the process from identifying what should be monitored, to evaluating findings. Therefore, rather than the project prescribing a set of indicators to be used, indicators on livelihoods, biodiversity and soil and water should be developed with communities. Because the community has a holistic way of perceiving its livelihoods and environment, environmental and socio-economic indicators will be closely intertwined and monitored. Ensuring participation of farmers in monitoring is vital to its success. Land users frequently have detailed knowledge and make careful observations of changes in soils, moisture, vegetation etc. and the interaction of human activities and the environment. Many farmers also carry out their own `experimentation' on their fields, trying out ways of blocking gullies with vegetation or stones, adapting varieties or plant management according to drainage and soils in different parts of their fields, etc. Ensuring community level monitoring is successful will require development of project capacity for facilitating the process and communities' capacity for analysis. Some issues to be 40 Nanci Johnson et al. CAPrI Working Paper No 10, September 2001. 41 Guijt et al. PLA notes (1998) 28-36 67 China Watershed Management Project Final Report (June 2006) 42 addressed include: · how to make monitoring and evaluation more participatory ­ how to ensure diverse groups participate · and how to maintain high levels of involvement; · what methods are feasible in which contexts? · How do we use participatory monitoring in hierarchical situations and in conflict situations? · What are the gender needs and implications of participatory monitoring, and how can they be built into the process? 6.3 Approach to participatory monitoring in CWMP Quantitative surveys have their strength in comparing levels of poverty and changes of poverty according to predetermined and more generic indicators. They have their limitations when it comes to understanding the local dimensions of poverty and the underlying factors affecting poverty changes. Participatory impact monitoring will produce insights into how and why changes have occurred. It will also provide opportunities for individual interviews with the poorest and most vulnerable to discuss participation levels and issues of inclusion and exclusion. These interviews are particularly important in accessing those who might not have participated, as well as those who did. CWMP needs to gain experience in monitoring livelihoods indicators. The statistical survey to be carried out by the Rural Survey Organisation will help to monitor conventional indicators through use of questionnaires. In addition, the project will explore ways of monitoring changes of livelihoods and environment at community levels. Both methods of data collection have their advantages. The following table contrasts participatory data collection with formal surveys to highlight their relative strengths and weaknesses. Table 14: Comparing questionnaire survey and participatory data collection Questionnaire survey Participatory data collection Is best... Is best... · When you need to measure known parameters or · When you need to explore what type of change is behavioural characteristics from many units of taking place and the reasons why study (individuals; households; villages etc.) · When you need an estimate of a population · When you need to understand differences among characteristic with known precision, by using a groups in society random sample · If the information to be collected is sensitive or the people being interviewed may be reluctant 42 Guijt et al. 1998 PLA notes 31 28-36 68 China Watershed Management Project Final Report (June 2006) respond openly to a questionnaire · When you need to understand the motivations of people or unexpected causes or changes which have not been anticipated by the project Is not... Is not... Necessarily linked to randomisation, nor do Only descriptive and can be linked to randomised questionnaire surveys have to be of any particular selection of people or villages so that results can be scale generalised to a wider population Participatory monitoring is therefore seen as an important approach to supplement the findings from conventional poverty surveys. Some indicators are better monitored through participatory data collection, and participatory methods are particularly strong in encouraging the weaker parts of the community to express their views and perceptions. Qualitative and participatory monitoring also involves more holistic approaches to assessing watershed conditions and rely on field-based evidence (instead of measuring few technical indicators). Such an approach would also offer opportunities for strengthening stakeholder involvement in watershed M&E. One approach has been used successfully by Malcolm Douglas with local government units 43 to document and then monitor the status of their watershed natural resources . This approach has a strong focus on qualitative indicators, which are based on a more holistic observation of field conditions and indicate (positive or negative) changes. The monitoring process is carried out by local government units in participatory consultation with the resources users (upland farmers, forest users/managers, livestock keepers, etc.). Areas covered include: · Basic meteorological data: daily rainfall and temperature. This can be obtained by local government units by establishing simple meteorological stations. · Conservation effectiveness of upland farming, forest management and other NRM practices. Data collected will include annual and perennial crop production and woodland practices. This should be derived from a range of sources (to ensure coverage at all levels, and to triangulate findings): farmer records, extension reports, case study and household monitoring surveys, direct observation, participatory mapping and transects. · Erosion status. As above. Erosion status and soil conservation effectiveness would be collected by participatory appraisal techniques to document land users own criteria. · Water resources. Information on the status of the areas of water resources should come from key informant interviews and group interviews. 43 Malcolm Douglas. LGU Watershed Natural Res ources Status/Monitoring Report 69 China Watershed Management Project Final Report (June 2006) 6.4 Indicators for participatory monitoring The focus of participatory monitoring in CWMP has been defined in order to ensure that important dimensions of livelihoods and the dynamic relations between environment and poverty will be sufficiently monitored. Participatory monitoring should focus on the following issues for poverty assessment at both poor household and community level: · Changes of social capital; Changes of physical capital, such as the change on land using system; · Access to physical assets and resources, such as terraced land, grassland, forest land; · Food security and strategies for ensuring food security (e.g. borrowing, seasonal labour) · Access to agricultural technology and information; · Migration and non-agricultural work as part of livelihood strategies; · Gender roles and gender impact, e.g. change of women's work load after adoption of agricultural innovations; · Community capacities, e.g. for implementing projects or supporting vulnerable groups; · Seasonal shocks or disasters; · Long-term trends of poverty situations. Land user level indicators are expected to cover the areas of soil and water, biodiversity and livelihoods at household level. Vulnerability and gender indicators should also be included. As at watershed level, indicators to monitor the process and impact of technology change under CWMP need to be developed with the strong participation of land users. At the same time, farmer capacity to analyse observations should be strengthened and feed into a process 44 of participatory technology development (PTD) . Indicators that have been developed together with primary stakeholder are more likely to reflect the key aspects of poverty within a local setting and also the priority areas where people would aspire some improvements. Some of those indicators, like food security, can be used by communities to assess changes and improvements under the project. 44 Also refer to the FAO Visual soil assessment methodology developed for China 70 China Watershed Management Project Final Report (June 2006) Table 15: Indicators that can be monitored through participatory assessments (selection) Soil and Water Poverty Biodiversity - Types of soil erosion - Access to improved (terraced) - Changes of vegetation cover land - Changes of soil fertility - Occurrence of common species - Access to fuel wood - Improvements in land use - Occurrence of indicators species - Seasonal food shortage - water availability - Variety of tree species - Women's work load - Variety of food crops Indicators that would be assessed through participatory monitoring would need to be translated into specific questions that can be discussed with farmers during the field assessment. In addition to (oral) discussions, visualisation tools or joint explorations in the field are useful ways for collecting information on those indicators. Table 16: Collecting information on indicators through participatory assessment (example) Indicators Issues Questions Tools Equal access to terraced The project assumes that - Who are the poorest - List or map of land by poor households the access to terraced households in the households, indicating land will vary across village? the poorest different households. households Possible factors - How many poor influencing the access to households have - Land use map, land could be selection of owned land in the indicating location terracing site, modes of terracing site or have and ownership of land allocation and been given terraced terraced land resources that households land? - Joint field assessment have available to put - How have they used of terraced land terraced land into use. the terraced land? - Group discussion on The project management - Are there cases that the process and would need to assure that poor households had consequences of land poor households are not to give up newly - allocation. excluded from access to terraced land? terraced land. - How many non-poor households have benefited? The methods used for the participatory impact studies cover common tools for participatory assessments, focus groups and individual interview, resulting in a wealth of broader information born from group discussion, as well as more specific information on more sensitive issues. Individual interviews focus on particular target groups within the communities, and will allow for detailed discussions on focal topics with the most vulnerable groups. Information to be obtained here might be on issues of inclusion and exclusion and their links to poverty, power relationships within the community, and particular changes in livelihoods. The focus groups, with representatives of different groups or socio-economic types, provide a basis for more of a communal discussion, giving light to changes at community level. Issues to be discussed here might be on the links between community activities and changes in poverty, cohesion within the community, relationships between the community and external project and non-project stakeholders, and background information on 71 China Watershed Management Project Final Report (June 2006) changes to the community Table 17: Indicators and methods for participatory impact assessments Level Indicators Methods Community Changes of household assets: Group discussions; - Distribution of terraced land and use of Visual tools: Resource maps terraced land by poor and very poor households. - Access to fuel wood - Distribution and use of grassland for livestock Changes of poverty situation: Group discussions: - Food security and strategies to achieve Visual tools: Calendars and diagrams food security - Income sources - Migration - Seasonal changes of poverty - Relative cha nges of poverty over time Changes of environment Group discussions: - Occurrence of common animal species Maps and diagrams and indicator species - Varieties of trees and herbs used for livelihoods - Changes of vegetation cover over time - Occurrence of natural disasters and hazards Individual farmers Indicators to monitor changes of land use Farm visits: at the farm level, e.g. Land use mapping, notes and field - crop varieties observations - soil quality and fertility - soil conservation measures Very poor Indicators to monitor vulnerabilities and Interviews with selected households, households and inclusion identified through participatory poverty women assessment; - participation of poor households in project activities use of visual tools, like seasonal calendars and resource maps - access to resources - seasonality and extent of grain shortages and grain borrowed Mapping of land use systems and of resource status will be an important method for participatory and field based observation. For a more comprehensive approach to use of 72 China Watershed Management Project Final Report (June 2006) mapping in watershed management, CWMP should look into the tools for documenting 45 watershed management practices developed by WOCAT. 6.5 Participatory evaluation of project activities In addition to monitoring changes of livelihoods, participatory methods can be used to assess project performance and success. Participatory monitoring of project implementation will help to build the community's capacity to track the progress of its own development. Data are collected about the progress of activities so that the community can make its own decisions about o What is working well; o What is not working well; and o How to proceed next. Participatory project evaluations can be carried out at the end of an activity or project. Common tools to evaluate project success include report cards, which will produce a quantitative score on progress, various PRA tools used without external facilitation (such as pie charts and proportional piling), and community photography (using cheap/disposable cameras from villagers to record good and/or poor progress). Box 14: Methods for participatory project evaluation Mapping § How many households have participated in the project activities § How many people have used certain new services § Reductions or increases in disease patterns § The percentage of girls enrolled in school Diagramming § Venn diagram to illustrate which projects have been most useful § Venn diagram to illustrate new collaborations between organizations as a result of the project § Flow diagrams to show the progress of the project implementation § Flow diagrams to show how the project has affected the community in other ways Ranking/Scoring § Using matrix scoring to compare successful projects with unsuccessful projects according to various criteria (e.g. number of people who contributed labour, amount of external resources obtained, 45 WOCAT has published a comprehensive and very practical approach to document and monitor land use practices in integrated watershed management projects. Chinese language material is available on the internet. 73 China Watershed Management Project Final Report (June 2006) number of times the community met to work on the project) Semi Structured Interview § Discussions with community members about their perceptions of the project's success § Discussions about problems with project implementation § Discussions about the direct and indirect effects of the project on the lives of community members Pie Charts § Illustration of estimated incomes or expenses of households § Illustration of the increases or decreases in the percentage of people engaging in certain kinds of behaviours 6.6 Monitoring stakeholder participation in CWMP It is important that project beneficiaries are able to participate in the monitoring of project progress and in the evaluation in order to understand whether the project has actually met their needs and had positive impacts on the livelihood and poverty indicators that they consider important. In reality, it may be difficult for the farmers to participate in the project throughout: On one hand, the farmers' structure of knowledge is different from that of science, and their generally low level of education and financial poverty often tend to make the farmers being unconfident, timid and acting in conformity with the majority, as well as other hurdles to participation in the project. If during the course of project implementation, the means, methods and environments of participation cannot be adapted to the habitual means of expression for the farmers, especially the disadvantageous groups, the voices of the farmers may not be truly heard, not to say that they may have effective participation in making choices or decisions. Some marginal and very poor groups may find it difficult to participate and voice their choices or decision and the project will need to find ways to encourage their participation. Box 15: Marginal Groups in Watershed Management Projects l Households from the poorest villages in the watershed: Without good access to transportation, shortfall of infrastructure, harsh ecological environment, and particularly vulnerable to natural disasters. l The poorest households in a community: With extremely little domestic assets, low education level and often being physically handicapped or chronically sick. l Women in the community: The tradition not only restricts them from participating in any affairs beyond family, but also demands them to be burdened by both domestic economy and housework. Their needs are often covered by domestic needs, but they could not automatically benefit from development projects. l Small ethnic minorities in the watershed: Due to historical reasons, they often live in remote places, and because of linguistic barriers, their communication with the external world is limited; their special difficulties and needs could hardly be reflected in a complete manner. 74 China Watershed Management Project Final Report (June 2006) The following indicators are suggested to monitor stakeholder participation in CWMP. The indicators are based on the assumption that CWMP will develop and adopt an approach to stakeholder participation throughout all phases of the projects. The indicators should be further specified, once the approach to participation has been defined. Box 16: Indicators to monitor stakeholder participation Indicators How to monitor those indicators Stakeholders, including communities within the Documentation of stakeholder analysis for all different parts of the watershed, have been project watersheds, to be submitted as part of identified during the preparation/planning the planning document. phase of the project. Opportunities for stakeholder participation have Roles for project stakeholders in decision- been created during planning, implementation making and opportunities for consultation and monitoring of the project. defined as part of the overall management approach. Project effectively identifies and addresses the Participatory poverty assessment for each priorities of the poorest groups within the watershed implemented during project watershed. preparation and priorities reflected in project planning. Technologies are identified and developed Approach to farmers-centred technology through farmers participations. development defined in operational manual. Effective stakeholder participation and Methodology for stakeholder participation communication during all phases of the project, defined in operational manual. including M&E. 75 China Watershed Management Project Final Report (June 2006) 7 Implementing baseline surveys 7.1 What is a "baseline"? Baselines are derived from outcomes and indicators. A performance baseline is information ­ qualitative or quantitative ­ that provides data at the beginning of, or just prior to, the monitoring period. The baseline is used as a starting point, or guide, by which to monitor future performance. Baselines are the first critical measurements of the indicators. Baseline data will be used to describe the situation before the project has started. h For example, t e poverty survey which has already been implemented by the RSOs has 46 established a first baseline on poverty indicators in selected areas. The following table presents some of the indicators that have been tested and the baseline data that have been established for two pilot watersheds in Huachi and Kongtong. Key performance indicators that have been tested and found useful would be monitored over the project's lifetime to detect significant changes. Table 18: Household poverty indictors and data (2005 baseline) Indicators Unit Huachi Kongtong Daily water consumption Liter/HH 155.91 332 Amount of firewood collected weekly Kg/week 25.23 23.32 Expenditure on health Yuan/H 159 565 Loss of working days due to the illness Day 18 24 Expenditure on education Yuan /H 251.37 129.53 Household debt as a percentage of household income % 9.06 54.97 It is recommended that key performance indicators should be monitored in the beginning, middle and end of a typical 5 years project. In the case of CWMP, project duration would be shorter; it would therefore be sufficient to monitor the same indicators again, once the project has been concluded. 46 See consultant's report on poverty survey, implemented by RSOs 76 China Watershed Management Project Final Report (June 2006) 7.2 Basic steps in implementing a baseline The project will have to conduct a more comprehensive baseline survey for all indicators that have been agreed, once the planning for CWMP has been finalised. The following overview presents the basic steps in conduction a baseline survey: Box 17: Basic steps in implementing a baseline þ Selecting key indicators to monitor outcomes þ Establishing baseline methodology þ Selecting sample watershed for baseline survey þ Contracting survey organisations and implement surveys according to ToR þ Reporting data from technical surveys þ Training staff for Participatory Monitoring þ Select villages for Participatory Monitoring and implement participatory monitoring according to guidelines þ Integrate and present data These basic steps are described in further detail in the following. ¨ Selecting key indicators to monitor outcomes Key indicators have been selected during the process of indicators development and testing. Annex 1 presents an overview of the indicators that are recommended for use in CWMP; it describes indicators that would be needed to describe the different aspects of changes that are expected to happen under the project (environment ­poverty) and the different levels of outcomes: indicators that help to monitor access to and use of project services, indicators to monitor direct project outcomes and long-term changes that can be related to the project. CWMP will need to ensure that all types of indicators are represented and that they fulfil the basic quality criteria, mentioned earlier. The project might need to customise those indicators re during the process, for instance to select those indicators that a most relevant for the planned interventions. to modify single indicators, for improved measurability or cost- effectiveness; or to add additional indicators which are found relevant in the local context. Indicators that are recommended for use during the baseline are included in the Terms of References for the survey organisations. They include Key Performance Indicators that will help to monitor project outcomes at different level together with some more general indicators that have been tested previously and might be used for collecting additional information to compare the status of project watersheds, villages and households. 77 China Watershed Management Project Final Report (June 2006) ¨ Establishing baseline methodology For each indicator, the project would need to define the methodology and other aspects of data collection. The overview included in Annex 1 describes the basic methodology; the detailed methods are further explained in the survey guidelines included in the Annex. f The following template is recommended as a checklist to ensure that all aspects o the baseline have been sufficiently clarified and as guidance for planning the baseline survey. KPI Methods used Frequency of Location and Responsibility Costs for Responsibility to monitor data collection sample size for data data for reporting KPI collection collection ¨ Selecting sample watershed for baseline survey CWMP has now selected four watersheds for pilot activities The project will carry out a full 47 baseline in these watersheds. The following watersheds have been selected: . Box 18: Small watersheds selected for CWMP (Gansu Province) Name County Area No of Population Farmers Poverty Erosion Integrated (km2) villages net rate modulus treatment income (t/km2) Rate (%) Baicha Jingning 25.1 5 6321 1020 24% 9360 30.1 Jiajiyu Kontong 28.2 6 7385 1200 30% 8280 44.9 Gaogou Huanxian 38.6 4 1338 910 35% 7569 37.3 Fanzhuang Huachi 32.0 2 1100 1100 30% 7303 29.2 It is understood that a baseline will be carried out for all four watersheds. Within each watershed, the survey needs to sample villages, based on the recommendations in this report. 47 Because this assignment was scheduled as the first activity within CWMP, watersheds have not been selected at the time the M&E team carried out its field work. Therefore, we have used different watersheds for case studies in this and earlier reports. 78 China Watershed Management Project Final Report (June 2006) ¨ Contracting survey organisations It has been agreed that the specialised surveys on poverty, biodiversity and soil & water e would be carried out be t chnical survey organisations, contracted for this purpose. The detailed Terms of References, specifying the tasks of the contractor together with the key indicators and methods for data collection and analysis; are included in the Annex. ¨ Reporting data from technical surveys Reports prepared by the survey organisations need to present the data clearly and according to the required levels of analysis and disaggregation. The reporting requirements are specified in the Terms of References. The flowing example from soil and water monitoring shows how information on indicators could be presented to illustrate major changes. The Soil and Water Monitoring Centres has been monitoring watersheds under the Loess Project; it has selected two project watersheds and two non-project watersheds with similar geo-physical conditions. The table shows that the rate of erosion control together with the vegetation coverage is significantly higher in the project watersheds; both indicators have been selected to monitor project outcomes in CWMP. Through use of additional indicators to measure livelihoods changes and long-term changes within the watershed, the project would be able to reach some conclusion on how useful its interventions have been to address key environment-poverty linkages. Table 19: Basic data of 4 experimental small watersheds Watershed Location Control Topography Watershed Rate of Vegetation Area under erosion erosion coverage (%) ( km2) control control (%) Miaogou Huachi 3.5 II Hilly yes 60 50.3 county region of LP Baizuigou Huachi 3.2 II Hilly no 17.5 11.6 county region of LP Qiaozigou Huanxia 2.84 II Gully no 33.5 25.7 county region of LP Yataigou Huanxian 2.95 II Gully yes 62 46.2 county region of LP ¨ Training staff for Participatory Monitoring At the same time, the project would have to start training the PMO staff that would be carrying out Participatory Monitoring as part of the implementation process. Details of the capacity building plan are included in the Manual. 79 China Watershed Management Project Final Report (June 2006) ¨ Select villages and implement Participatory Monitoring Participatory Monitoring will use a different approach for establishing a baseline: it will start together with the planning process in individual villages that go into implementation and monitor gradual changes throughout the process. Indicators that will be monitored should be developed as part of the planning process and baseline data sets should be collected accordingly. Changes would be monitored as part of the annual planning and review process. For projects on technology development, data would be collected and monitored even more frequently. The detailed steps for Participatory Monitoring are described in the Manual. ¨ Integrate and present data Data collected from the survey will have to be presented in a clearly defined format. The baseline is the first measurement of critical indicators; it presents the starting point by which the project's progress and performance will be monitored. Once baseline information has been collected for key performance indicators, the project would need to agree realistic target. Data collected at later stages will be compared with the baseline data to indicate progress towards agreed targets. The following table is recommended to present baseline data: KPI Baseline Current Target Difference Corrective action Chapter 5.6 presents some methods and tools for integrating and presenting data from different surveys. Scorecards are recommended for use during the baseline. They are a simple but effective tool for visualising the comparative status on different indicators. 7.3 Using baseline information CWMP will start into implementation shortly; at this stage of the Baseline information would be used for a number of purposes: · Apply methods for data analysis in practices The M&E system so far has not produced "real" data sets yet; the baseline will provide the project with a wealth of information that needs to be processed and analysis. As such, this will 80 China Watershed Management Project Final Report (June 2006) be the first opportunity to put methods for data reporting and analysis into practices. At this stage, the project should review the data collected on different indicators and find the most useful ways for analysing and presenting the information for use in project planning, management and stakeholder communication. The project would also have to assess whether the information collected is useful and appropriate for describing the status of watersheds, villages and households and whether it captures the key dimensions of the problems CWMP wants to address. It is therefore recommended that the process of data reporting and analysis should culminate in a workshop for review of findings and reflection on methodology. · Compare status of project watersheds and further define project strategy Baseline information will provide the project with a complete set of data on key characteristics f of the project area. Comparative analysis o the status of project watersheds will help the project to achieve further clarity on the major aspects of environmental degradation and the most important environment-poverty linkages it seeks to address. These are expected to differ across different watersheds, and thus the project will be able to further refine and focus its strategy for individual watersheds. · Achieve further clarity about objectives and set targets Analysis of baseline data will enable the project to achieve further clarity about the objectives it wants to achieve and set realistic targets that will help to monitor progress at regular intervals. Monitoring intervals will depend on the nature of the indicators. Targets for inputs, activities and outputs (e.g. provision of infrastructure and services; use of infrastructure and services by target beneficiaries; participation of target beneficiaries in project activities) are usually monitored on a quarterly base during the implementation process. Targets in relation to project outcomes would be monitored in longer intervals, as recommended in Annex. · Reflect and improve indicators and methodology It is recommended that the baseline should be used as an opportunity to reflect on the methodology and to further develop and improve the indicators systems as required. The baseline will provide the project with a better understanding on the usefulness and relevance of certain indicators; it may also reveal certain gaps of information which may require definition of additional indicators. It is important that CWMP understands that the development of M&E is a dynamic process which requires ongoing learning and improvement in order to generate the kind of information that stakeholders find is most needed and useful to support adaptive project management and implementation. · Communicate with stakeholders Generation of a complete set of information from baseline surveys will also provide an 81 China Watershed Management Project Final Report (June 2006) opportunity to present the key concerns and strategy of CWMP in a concise way and with practical data to illustrate issues and provide evidence from the field. The project will need to think about how to present and communicate the main findings for the audience it wants to target. This is the point of time where M&E will link into the communication strategy adopted by CWMP. Baseline findings could be communicated through stakeholder workshops, technical survey reports, visual presentations or newsletters, depending on the audience. 8 Further steps 8.1 Building capacities for participation This assignment has helped to identify and further emphasise opportunities for participation in CWMP. The project will need to build on these opportunities and take them forward into a more comprehensive approach to build capacities for participation through all stages of the project. During implementation of Loess Project the PMO staffs, who were mostly technical people, without much experience in participation, realised the importance of farmers participation in the project. The following insights have been voiced by PMO staff: § "The farmers' opinions should be heard in the choice of projects, and they just would not do it well if they were unwilling to do it." § "The water conservation forests planted by the project cannot possibly survive without the sustainable tendering of the farmers. We should figure out ways to make the farmers participate in the tendering." § "The project should also take into account the farmers' requirements and needs, or they would not have incentives." § "If you want to convince the farmers, you got to consult them slowly. Executive orders do not work anymore." § "Projects of other departments are also influential to our project. If we reach agreement with them, and we implement the same subsidy policy to the farmers about forestation, the project could be more easily implemented." § "We water conservancy department could not possibly solve the issue of watershed management, and cooperation of multiple departments is needed." § "The leaders should read the outcomes of monitoring, or they would be meaningless." § " We realized that a mechanism is needed for participation and consultation of different stakeholders." As part of the indicators development, the M&E team has organised an exercise for participatory impact monitoring where attendants realized that applying suitable participatory approach could effectively promote the farmers to express their opinions and wishes, and deepened their acknowledgement about farmers' participation. They also learnt to apply some participatory tools, such as the poverty grouping, resources mapping, transect mapping and 82 China Watershed Management Project Final Report (June 2006) interviewing techniques, in the field; the practical exercise has obviously enhanced their interests in using participatory tools. As an outcome the Gansu PMO decided to integrate participatory monitoring into the M&E system. More specific agreements include: § Farmers should participation in project choice, project implementation and project monitoring and evaluation. § The opinions of the farmers should be one of the bases for making decisions and adjustments about the project. § The farmers not only can provide information for project monitoring, but also should participate in evaluating the project. § The opinions of the poor households should be heard in project M&E. These agreements are reflected in the choice of indicators and methods for community participation, but they need to be followed up by CWMP with a more comprehensive approach to participation, which would include participatory M&E as an integral (but not isolated) part. It has been agreed that the PMO will be responsible for implementing participatory M&E. Some local NGOs with practical experiences in implementing participatory approaches in the project are could support the PMO in establishing and implementing a system of participatory M&E. e More training would b needed to support the PMO in developing a practical approach to participation. Training should include § Concepts and operation methods of integrated watershed management and principles of subsidiarity and participation. § Participatory technology development and farmers field schools in support of sustainable land use system. § Participatory mapping tools and methods to document field observations. § Participatory poverty analysis and methods for pro-poor sampling. § Facilitation skills for staff implementing annual planning and monitoring exercises. § Gender awareness training § Methods for verifying and analyzing qualitative information. 8.2 Building capacities for M&E Designing and building a reporting system that can produce trust-worth, timely, and relevant information on the performance of a government program requires experience, skill, and real institutional capacity. The capacity for a results-based reporting system has to include, at a minimum, the ability to successfully construct indictors; the means to collect, aggregate, analyse, and report on the performance data in relation to the indicators and there baselines; and managers with the skill and understanding to know what to do with the information once it 83 China Watershed Management Project Final Report (June 2006) 48 49 arrives. Previous assessments have described the gaps in terms of institutional capacities. CWMP will have to develop its strategy for building capacities on M&E. This will include ú Training on clearly defined technical and methodological issues; ú Exposure trips and project visits to study existing best practices; ú Ongoing learning from practice in M&E; ú Use of existing capacities among stakeholder groups, especially for clearly defined technical services. Statistical capacity is an essential component of building results-based M&E system. Information and data should be valid, verifiable, transparent, and widely available to the government and interested stakeholders. Through the cooperation with the RSOs, the project aims to build up the statistical capacity required for reliable monitoring of poverty. In addition, the use of Soil and Water Monitoring Centres will help the project to produce high quality data on the physical changes within the watershed. Although part of the M&E will be contracted to specialised organisations, the revised M&E system will put high demands on PMO staff. They will have to acquire new skills for managing their tasks within the M&E system Box 19: M&E skills for CWMP project staff Basic skills (for all PMO staff): Project planning and monitoring methods (logical frame work approach) Integrated watershed management Skills for managing M&E systems Managing surveys and project evaluations Results-based monitoring Sampling strategies Skills for field staff and facilitators: Understanding poverty (sustainable livelihoods, poverty analysis) Stakeholder analysis Field-based observation and documentation of environmental data Methods for participatory assessment and facilitation of group discussions 48 Jody Zall Kusek, Ray C. Rist: Ten steps to a results based monitoring and evaluation system. The World Bank 2004. 49 See M&E sys tems review and capacity assessment during the inception phase 84 China Watershed Management Project Final Report (June 2006) Specific M&E skill for management staff: Indicators development and review of indicators systems Reporting and data handling Integration and analysis of data from qualitative assessments Data analysis and presentation (including tools like BSC, Dashboard) Software and systems for data processing (GPS, GIS) CWMP plans to apply the proposed M&E framework in four pilot watersheds. For this, the project will face some immediate tasks which will provide opportunities for learning and capacity building through practice. In addition, the baseline exercise will provide a good opportunity to apply skills for data analysis and integration. It also offers a chance for reviewing the practicalities of the framework. In addition, CWMP aims at building capacities on M&E as part of a more comprehensive training and capacity building program. The following table provides an overview on the strategic areas that should be addressed through immediate tasks and longer term training and capacity building. Table 20: Institutional change and capacity building for M&E Objective of M&E Institutional Immediate tasks Training and capacity implications building Monitor ú Additional ú Baseline survey to ú Training on poverty livelihoods capacities for collect data on poverty analysis benefits for the monitoring ú Data should be ú Building capacities for poor livelihoods impacts analysed through community learning exercise participation through training and practical application ú Develop methodology for participatory poverty monitoring in cooperation with NGOs Monitor improved ú Additional ú Baseline survey to ú Approach to ecological capacities for collect data integrated watershed conditions in monitoring environmental data management in CWMP watersheds environmental ú Data should be ú Methods for field- impacts analysed through based observation and learning exercise monitoring (WOCAT) ú Approach to process and integrate data on a watershed level Improve the ú Build capacities for ú Review and revise ú Training on quality and participatory institutional roles in participatory data efficiency of data collection and M&E according to collection and collection, analysis of data at framework analysis. handling and lower levels. ú Implement ú Training on computer- 85 China Watershed Management Project Final Report (June 2006) Objective of M&E Institutional Immediate tasks Training and capacity implications building analysis ú Engage specialised participatory based tools for organisations for assessments for integrating data (BSC, technical surveys baselines communities Dashboard). ú Build capacities for ú Contract survey ú Training on use of GIS use of computer- organisations for software based systems for baseline surveys reporting and data ú Information and ú Establish M&E feedback across integration at reporting system administrative levels higher levels as key management ú Improve inter- task departmental cooperation and communication Monitor ú Build capacities for ú Develop participatory ú Training on methods participation of community approach for CWMP for participatory the poor in the participation within and integrated planning, monitoring project project principles of and evaluation management community methods participation into ú Establish ú Training on gender project management participatory awareness guidelines approach to project ú Training for PMO planning, ú Apply and review facilitators implementation proposed methodology and monitoring for participatory ú Methods for farmers monitoring and participation (Farmers evaluation Field Schools) ú Test proposed ú Sharing with other indicators for projects monitoring stakeholder participation and integrate indicators on community participation into project framework Stakeholder ú Establish platforms ú Stakeholder analysis ú Monitor and improve participation in for stakeholder for CWMP watersheds stakeholder M&E participation and participation ú Participatory planning communication for CWMP ú Sharing with other institutions and ú Customise indicators projects for CWMP together with stakeholders ú Establish opportunities for stakeholder ú Define stakeholder participation in M&E roles in M&E, based on through defined roles M&E framework and regular communication (e.g. for sharing of baseline results) M&E system ú M &E work needs for ú Costs of `outsourcing' ú Budget for M&E should be cost be budgeted parts of the survey effective work need to be assessed after baseline survey 86 China Watershed Management Project Final Report (June 2006) 8.3 Accountability and transparency in M&E An inclusive and transparent M&E system will be key for increasing accountability in large- scale projects. The Loess Project area covers a huge region and project management has been established on four administrative levels, a setting that provides a huge challenge to flows of information among stakeholders. In the previous Loess M&E system, stakeholders at lower levels have not been sufficiently involved in M&E; they have been providing data only and feedback from higher levels has been limited. Stakeholders outside of the project management system have hardly been involved. It will be a huge challenge for CWMP to work on the internal communication structure of the system and information flows among stakeholders. Communication of M&E information is not an issue of M&E alone; it is part of the wider management and communication structure and the problems and bottlenecks need to be addressed through far reaching institutional change. Accountability, transparency and stakeholder participation are key quality criteria for good M&E, but they are also key criteria of good governance which need to be reflected throughout project management. A well performing and sustainable M&E system will depend on broad support from project stakeholders, but those are only likely to engage in and contribute to an M&E system which they find credible, trustworthy and accountable. Stakeholder engagement will help to the 50 monitoring system self-sustaining in the long run: The big challenge for CWMP and later Loess phases will therefore be to sustain a good quality M&E system. Quality will be as important as funding for sustaining M&E in CWMP and later Loess Project phases. Zall Kusek and Rist (2004) cite six critical components for sustaining an M&E system which are all linked 51 to institutional aspects: ú Demand ú Clear roles and responsibilities, ú Trustworthy and credible information ú Accountability ú Capacity ú incentives The M&E systems review and M&E framework development have placed high priority on institutional and stakeholder issues in M&E. At the moment it seems that the project is still focussed on the technical aspects of M&E and it will therefore be a key challenge for CWMP to build awareness on the importance of institutional processes and linkages as a basis for M&E. 50 Ujjayant Charovorty, et al. policy relevant monitoring systems for natural resource management. 51 Jody Zall Kusek, Ray C. Rist: Ten steps to a results based monitoring and evaluation system. The World Bank 2004. 87 China Watershed Management Project Final Report (June 2006) 8.4 Criteria for assessing the quality of M&E The M&E framework has been developed to address previous shortcomings of M&E in Loess Project. The M&E framework proposed with this report aims at improving the quality of M&E through indicators development, methods for data collection and analysis and use of information for management purposes and stakeholder communication. CWMP will have to continue the approach to strengthening M&E along these lines. In this final Chapter 8.4 we present the criteria for quality of M&E that have been guiding the revision of the M&E system. They can be used both for further development of this system and for assessing comparative analysis of other systems that may want to share some of the key features of M&E in CWMP. Box 20: Quality criteria and key features of M&E Criteria Key Features of M&E in Risks/Recommendations CWMP Quality of indicators Indicators system ú covers aspects of CWMP may not be able to overcome monitoring livelihoods and ecology, routines; needs for aggregate and quantitative information at higher levels may overwhelm attempts ú includes project to develop and monitor indicators at lower levels. outcomes and long term impacts; ú includes disaggregated indicators for different levels Completeness and Data collection methods : quality of data ú Data collection modules Completeness and quality of baseline data has not to combine different been tested yet; they should be reviewed after the methods of data CWMP baseline collection and optimise PMO system is not yet acquainted with use of quality of data qualitative data which must complement quantitative data sets. ú Sampling strategies for effective and reliable Project management needs to ensure that data are data collection reliable, valid and timely Data handling and ú Four levels of data quality of analysis analysis and integration PMO staff still needs to build capacities for data handling and analysis. ú Institutional roles assigned according to capacities for data collection and analysis; Use of information ú Information pyramid to for management improve availability and decision-making The proposed M&E framework needs to be accepted usefulness of M&E and put into practice by project management at all information at higher levels. management levels ú KPI to focus data collection 88 China Watershed Management Project Final Report (June 2006) ú Integrated analysis for decision-making at lower management levels Feedback of ú Issue of feedback has information to lower been addressed; levels Mechanisms for feedback of information are not yet in place; they need to be developed and tested in CWMP. ú Stakeholder platforms at all levels have been proposed to improve information flows Accountability, ú Participatory M&E transparency and The present management system is focussed on stakeholder ú External institutions vertical linkages and upward accountability. participation involved in key stages ú Horizontal linkages and downward accountability through stakeholder platforms 89 China Watershed Management Project Final Report (June 2006) 9 Conclusions and Outlook Work on M&E in CWMP has created opportunities more sustainable watershed management in the Loess plateau area. It has deepened understanding of environment-poverty linkages and the importance of stakeholder participation for more localised approaches to address urgent environmental issues in this region. Through development of an M&E framework the project has sharpened its view on intervention levels and it has recognised the importance of getting lower levels, especially poor farmers, directly involved in project planning and monitoring. Identification of four distinct levels for monitoring was a major step forward: it allows integration of data at all levels; at the same time it helps to establish clear linkages for monitoring impacts along a results chain. Focus on key performance indicators for each level helps to reduce complexity of data at higher levels and allows comparative analysis of distinct geographic units (villages, watersheds) within a watershed context. It is recommended that the approach of identifying and monitoring geographical units (rather than administrative units) should be followed through for effective watershed management. The project would need to identify and target poor natural villages within a watershed in order to address addressing environment-poverty linkages effectively. Use of geographical units for monitor would allow the system to link up with a possible system for watershed GIS. In addition, the project would need to strengthen monitoring project impacts on certain social groups: poor households, women and other marginal group. The report suggests indicators for monitoring participation and impacts on poor groups. Participatory monitoring has been introduced as an approach to assess project impacts. Participatory monitoring is especially useful for assessing poverty impacts, but it can also be used for monitoring environmental changes. A range of indicators has been tested which can be monitored through participatory methods and concrete methods for participatory monitoring are proposed in the M&E framework.. Participation is still a new and challenging concept in CWMP. Project staff has gained initial experiences with participatory methods through practical field exercises as part of the indicators development. Participants have recognised the possibilities for use of participatory methods in M&E and the need for greater participation of primary stakeholders in general. This has created opportunities for CWMP to introduce participation more widely. Useful methods for participation would be Farmers Field Schools for technology development and participatory land use mapping for planning and monitoring purposes. These techniques would also support the shift towards more field-based methods for observation and data collection. 90 China Watershed Management Project Final Report (June 2006) The present approach of monitoring environmental and poverty indicators within a modular structure has been adopted to address existing capacity constraints. External monitoring of key performance indicators helps to improve reliability and credibility of data and analysis. In addition, the project should strengthen its capacities for more holistic approaches to observe changes within a watershed context. Better understanding of environment -poverty linkages and ecological processes together with higher proficiency of field-based methodologies will enable project staff to develop such approaches. The institutional challenges for decentralised watershed management are huge, but the M&E system has created opportunities for horizontal integration and stakeholder participation at lower levels. It is important that those will be followed-through in CWMP, through piloting of corresponding planning and management approaches and a consistent communication strategy. The M&E system will only become effective within a management system that aims at improved accountability and transparency through clear horizontal and downward linkages. 91 China Watershed Management Project Final Report (June 2006) Annexes Loess II Log-Frame and Key Performance Indicators Narrative Summary Key Performance Monitoring and Critical Assumptions Indicators Supervision Sector Related CAS No large external shocks Objectives affecting terms of trade, current account Promote better Expansion of areas under Government and balance or foreign utilization of high-production provincial data. exchange risks. marginal lands agriculture. through introducing Government maintains sustainable Increased grain yields budgetary allocations techniques. for food security and poverty alleviation programs. Project Development Objective: Per capita income Monitoring of Help achieve household incomes sustainable Per capita grain and production with development in the consumption and without project. Loess Plateau by increasing Crop yields Sediment monitoring agricultural Levels of sediment flow. in one watershed per production and county. incomes and improving ecological conditions in tributary watersheds of theYellow River. Project Outputs: Area of slopeland (i) Natural disasters are converted to terraces and not unduly severe. Cropping on Household (ii) planted with slopeland declines surveys/plot orchards/forests/grassland production data Adequate fruit storage, Grain yields per hectare processing available. Household grain Share of household income Household production increases Markets for orchard, from grain crops, cash surveys/plot crops, orchard products, production data forest, livestock Household production product markets do not diversifies forest products, livestock collapse. Pen-fed sheep and goats Techniques/resources Grazing management per household Household surveys for pen-feeding practices improve Open range grazing sheep adequate. and goats per household In high population Household surveys Percentage of land density land is /County agricultural developed under the distributed equitably. More Secure Land-use statistics project contracted to Rights project households areas, Project monitoring with long-term land use data rights. 92 China Watershed Management Project Final Report (June 2006) Key performance indicators for CWMP Key performance Level to monitor these Methods to monitor Frequency Responsibilities indicators indicators these indicators Long term changes that Soil and water indicators: Project watersheds Soil and water survey Before, mid and after the Contracted to technical will be monitored for project agency (recommended: different (project) ú Area under erosion Provincial Soil and Water watersheds control Monitoring Centres); ú Reduced soil erosion Supervised by PIO and improved soil fertility ú Reduced sedimentation Poverty indicators: Project watersheds Poverty survey Before, mid and after the Contracted to survey project organisation ú Increased farmers' (recommended: Rural incomes Survey Organisation); ú Reduced number of Supervised by PIO people below official poverty line ú Improved food security ú Reduced vulnerability Biodiversity: Project watersheds Biodiversity survey Before, mid and after the Contracted to NGO project (recommended: Lanzhou ú Improved diversity of University); crop production Supervised by PIO ú Improved diversity in (planted) forests 93 China Watershed Management Project Final Report (June 2006) Key performance Level to monitor these Methods to monitor Frequency Responsibilities indicators indicators these indicators Direct project outcomes ú Benefits from assets Project villages Poverty survey Before, mid and after the Contracted to survey that will be monitored created by the project project organisation for different project (e.g. drinking water, (recommended: Rural villages and different fuel wood), monitored Survey Organisation); types of households for poor households Supervised by PIO ú Improved productivity Project villages Soil and water survey Before, mid and after the Contracted to technical of land resources, project agency (recommended: monitored for poor Provincial Soil and Water farmers Monitoring Centres); Supervised by PIO ú Soil conservation measures, as implemented by poor farmers ú Diversity of crops on Project villages Biodiversity survey Before, mid and after the Contracted to NGO improved land, project (recommended: Lanzhou monitored for poor University); households Supervised by PIO ú Levels of Project villages Participatory monitoring Before, mid and after the Monitored by County participation, project PMOs in cooperation with monitored for poor local NGOs (e.g. Oxfam) households Supervised by Provincial PMO ú Access to project services by poor people ú Social capital built by the project 94 China Watershed Management Project Final Report (June 2006) Key performance Level to monitor these Methods to monitor Frequency Responsibilities indicators indicators these indicators Project process that will ú Transparent budget Communities Participatory monitoring ongoing Monitored by be monitored as part of and project Communities, supported the implementation management at by County PMOs process community level Supervised by Provincial PMO ú Participation of women, poor and marginal groups ú Development of Farmers Participatory monitoring Ongoing Monitored by farmers sustainable land use together with technicians techniques and soil Supervised by County conservation measures PMOs by poor farmers 95 National System for Monitoring Soil and Water Conservation National Monitoring Centre for Soil and Water Conservation (MCSWC) The Monitoring Centre for Soil and Water Conservation, Ministry of Water Resources (MCSWC, established in 1998) is the state centre in charge of the monitoring, program evaluation, technology training, standard making on soil erosion and soil & water conservation in China. The Center has a mandate to be responsible, inter alia, for: · "the construction and management of the nation-wide monitoring network of soil and water conservation and environments" · "the general survey and dynamical monitoring of soil and water loss and ecological conditions" · "the investigation and analysis of the benefits from soil and water conservation practices" · "the survey and layout of regulations and policies for soil and water conservation and ecological management" · "the chargeable services concerning soil and water conservation" MCSWC at various levels are under the leadership of water administration Department and technical and professional guidance of their parent MCSWC. The headquarter station in Beijing has a mandate for national scale (macro) analysis (1:1000000), and it maintains a national, macro-scale database of soil and water conditions. The river basin and Provincial stations have similar mandates at their own scale. The national monitoring network for soil and water conservation is administrated through various level administrations. Various levels MCSWC are subjected to and managed by the corresponding water sector administrative department. The administrative management of the national MCSWC is shown in Table 1. Table 1 National Framework of Soil and Water Conservation Monitoring Monitoring Centre for Administrative Department National level Monitoring Centre for Soil and Water Conservation, Ministry of Water Resources Ministry of Water Resources Basin level Monitoring Centre for Soil and Water Conservation (7) Basin level Administrative Organization Provincial Monitoring Centre for Soil and Water Conservation Provincial Bureau of Water Resources Branch Monitoring Centre for Soil and Water Conservation in key Corresponding administrative erosion control area Organizations Monitoring Points for Soil and Water Conservation Branch Monitoring Centre The Center aims to ultimately have a national monitoring network of 7 river basin (watershed) monitoring stations, 31 provincial level monitoring stations, 175 `focused' monitoring stations, and 1625 monitoring points. A monitoring point is an instrumented area in which the aim is to record a standard minimum data set. A pilot scale network is currently being rolled out, and is due to be fully established by May '05. The pilot covers 13 western provinces, including all the Loess Plateau area, and both the Yellow and Yangtze basins. One of the Provincial stations is Lanzhou in Gansu. The pilot network thus involves 2 river basin stations, 13 provincial China Watershed Management Project Final Report (June 2006) stations and 100 focused stations. Of these focused stations, about 50 are in the Loess area, and both Qingyang and Pingliang have focused stations, and associated monitoring points. - The Gansu Provincial MCSWC was one of 13 provincial stations established in 2004. The main role of the Provincial MCSWC is the collection, interpretation, compiling, statistics, query of monitored data (including GPS, remote sensing image, air photos, observed and satellite image data) from all the monitoring points and warning of key monitored data in order to timely and precisely report high level decision makers the dynamic and varied soil and water conservation monitoring information. IThe centre is not an independent organization and is still a section under direct leadership of Gansu Provincial Bureau of Soil and Water Conservation. - Qingyang city Monitoring Centre for Soil and Water Conservation (Qingyang MCSWC) was established on Sept.2002 is an independent Shiye unit under the leadership of Qingyang Bureau of Soil and Water Conservation. WB Scientific Training Centre is another name of the organization. The centre is currently responsible for technology of SWC, training of personnel and guidance of the city, collection and management of monitoring data, undertaking monitoring, evaluation, extension, technical study tour of WB project. - Pingliang city Monitoring Centre for Soil and Water Conservation (Pingliang MCSWC) was established in 2003. The other names of the organization are Pingliang General Station of SWC, Pingliang PMO, WB Project Monitoring Centre. There are 24 staffs in the centre. The centre is the organization with second rate qualification certificate of MCSWC with facilities of PC computers, GIS software, scanner, GPS, and 2 monitoring station. It has the capability to undertake observation of runoffs and sediments of micro watershed, collection of meteorological data and monitoring of soil and water loss caused by construction projects. Bureau of Middle & Upper Reaches of Yellow River (BMURYR) The Yellow River Conservancy Commission (YRCC) is an agency of the Ministry of Water Resources, in charge of the Yellow River basin and the inland river basins in provinces including Xinjiang, Qinghai, Gansu and Inner Mongolia. BMURYR is the affiliated organization of the Yellow River Water Conservancy Commission. Its major responsibilities are planning, supervision, scientific research, demonstration and management work of soil and water conservation in 8 provinces in an area of approximately 700, 000 km2 and water related administrative work on the main stream of the Yellow River at a length of 4200 km. The YRCC, on behalf of the Ministry of Water Resources, has the responsibility for water administration in these basins. It has functions including: · Implementation, supervision and inspection of the Water Law · Carrying out an integrated management of water resources (including surface water and ground water) in the basin · Ensuring the protection of water resources in the basin · Drawing up and implementing the plan for flood control within the basin · Guiding the development of the rivers in the basin 97 China Watershed Management Project Final Report (June 2006) · Taking up and directing water and soil conservation and ecological construction and carrying out dynamic monitoring Xi An Monitoring Centre for Soil and Water Conservation (Xi An MCSWC) is under the leadership of Bureau of Upper and Middle Reaches of Yellow River and was established in 2002. This centre has linked to Zhengzhou Terminal Station and three branch centres located at Xifeng, Tianshui and Suide, The Centre has undertaken several key projects such as technology introduction project (948) on dynamic monitoring of serious soil and water loss in the Loess Plateau, remote sensing survey project on soil and water conservation in the Yellow River basin (1:250000 scale) and established 90s satellite image database, soil erosion database, regional soil and water conservation monitoring database, three dimensional visual information system, warping dam GIS and designing system of the Loess Plateau region. The centre has the capacity to provide training on GIS, RS and GPS, and has the mandate to coordinate, organize monitoring activities among the provinces in the basin of Yellow River. Xifeng Experimental Station of Soil and Water Conservation (XFESSWC) is located at Xifeng city (same city with Qingyang PMO) under direct leadership of Xi An Bureau of Upper and Middle Reaches of Yellow River, It was established in 1951. In the past 50 years, the Xifeng Station has conducted research on law of soil erosion in the gullied loess plateau, experiment, demonstration and extension of single and integrated measures of soil and water conservation. Xifeng Branch Monitoring Centre for Soil and Water Conservation (Xifeng Branch MCSWC) was established in 2004 and is situated at Xifeng City (Qingyang district) and is the monitoring centre with the first rate qualification certificate under the guidance of Xi An MCSWC. It has the capacity to undertake tasks of runoff and sediment monitoring across provinces. The staffs of the centre have rich experience in research and observation of runoffs and sediments, calculation of soil and water conservation benefits and data analysis. It has advanced monitoring equipments and facilities for SWC monitoring and skilled technicians in GIS, RS and GPS. Figure 8: National soil and water monitoring network 98 China Watershed Management Project Final Report (June 2006) Ministry of Water Resources Beijing National Monitoring Centre for Soil and Water Conservation Zhengzhou Terminal Station, Yellow Gansu Provincial Xi An Monitoring Centre for Soil and River Water Conservancy Commission Monitoring Center Water Conservation under BUMRYW Qingyang City Pingliang City Xifeng Branch Monitoring Centre Hydrological Station MCSWC MCSWC for Soil and Water Conservation Meterological Station Huanxian/Huachi Jingning/Kongdong Environmental Soil and County MCSWC MCSWC Protection Station Fertilizer Station Village Community 99 China Watershed Management Project Final Report (June 2006) Terms of Reference for Baseline Survey on Key Poverty Indicators52 1. Background of CWMP There is a critical need to assess the impact and effectiveness of watershed rehabilitation and management in the Yellow River Basin. This is due to the environmental problems and poverty situation characteristics of the Basin area, and the significance of the region for China's economy. The China Watershed Management Project (CWMP) is contributing to improved management by developing best practice models for watershed management which should have a potential for influencing Chinese government and donor approaches to watershed management and rehabilitation. CWMP has a special focus on participation and effective delivery of project benefits for poor households. In its first phase, the CWMP mainly aims to develop a comprehensive and sustainable social and environmental monitoring and evaluation system for use in assessing the impact of watershed rehabilitation and management projects. The development of this system is based upon an assessment of the monitoring carried out in the World Bank (WB) Loess 1 and 2 projects. In particular, the CWMP aims to improve monitoring to evaluate project benefits and impacts in a systematic way, focusing on the ways in which watershed rehabilitation projects can target poor and vulnerable households. Consequently, the development of monitoring and evaluation systems in CWMP will focus on the effects that the Loess 1 and 2 have had on changes in incomes and livelihoods, with particular reference to poor households. Ways in which poor households may have been disadvantaged by Loess 1 and 2 component implementation will be assessed, whilst those households missing out on project benefits will also be identified. An important aspect of developing an effective and appropriate M&E system will be to ensure participation of primary and key stakeholders, and ­ most importantly ­ village household members and women household members, in both monitoring and evaluation. 2. Objectives of Poverty Monitoring The main objective of poverty monitoring is to assess the social and poverty impacts of the watershed rehabilitation component of the Loess Project through selected poverty indicators. Analysis of poverty data will help the project implementation agencies to monitor changes of poverty status in different watersheds that can be directly or indirectly attributed to project interventions (such as terracing, building dams and gardening, etc.). A poverty survey is planned at three stages of the project: before implementation (baseline), in the middle of 52 Prepared by Liu Yonggong, CIAD, with revisions by the Team Leader 100 China Watershed Management Project Final Report (June 2006) implementation (mid-term) and after implementation (final). The poverty survey will be contracted out to a qualified agency to ensure reliable and high-quality data collection and analysis. 3. Tasks The survey contractor will complete the following major tasks: (1) Familiarize with and understand the general design structure and project contents implementation status of CWMP. This requires close study of project documents, most notably the project log frame and associated documents, the project inception report, the project schedule, annual reports during the project implementation period, and the list of indicators that have been used in internal monitoring of the CWMP. (2) According to the ToR on Poverty and Livelihood Monitoring, and referring to the operational steps and specific requirements of this manual, prepare work plan for quantitative monitoring of poverty and livelihood, the contents of which including objectives, steps, methods and expected outcomes of monitoring, indicators and methods for selecting sites, and budget of expenditures, etc., and submit to the Central PIO and Provincial PMOs for examination and approval; (3) According to the approved work plan, the basic criteria and principles for selecting monitoring sites, pilot micro watersheds and pilot villages in the project counties, that have been described in the M&E framework, need to be specified and applied in the context of a specific watershed. (4) Prepare data collection formats and questionnaires. (5) Review available secondary data and hold preparatory meetings with county and township staff to identify watersheds and villages that will be sampled. (6) Collect data at village level and household level for the piloting of indicators in villages in the four project counties, to be selected by the contractor in discussion with the staff from PMOs. For each watershed a sample of 20% natural, villages from each watershed zone will be selected, according to the criteria described in the M&E framework.. In each of these villages, households will be divided into four categories ­ very poor/poor /middle/ above average. A representative sample will be collected from each category, as described in the M&E framework. Hence, the total sample size will be around 180 households. In addition to the household data, the contractor will also collect the basic information and data required for the village level indices in each village. Both household and village indices are listed at the end of the TOR. Following on from the data collection, task team members of the contractor may decide to hold village workshops or key group interviews to corroborate their findings. (7) Systematically record field poverty data and relevant information collected, establish baselines for each indicators and report data according to data reporting sheets provided by the PIO. Compile disaggregate data sheets for natural villages and men/women where required. (8) Write full poverty and livelihood monitoring reports, including all major findings and monitoring conclusions, and submit to the Central PIO and Provincial PMOs. 101 China Watershed Management Project Final Report (June 2006) (9) Other tasks stipulated in the monitoring contract. 4. Expected Outcomes The poverty monitoring contractor will be responsible for writing a "CWMP Poverty Monitoring Report", the content of which should include: (1) Project backgrounds and objectives of M&E (2) The task of poverty survey and data collection, and the process of completion (3) Methods for data survey (4) Results of poverty survey A. Comparative analysis about village households' indicators and data B. Comparative analysis about village-level indicators and data C. Comparative analysis about micro watershed data (5) Conclusions of poverty monitoring (6) Suggestions for project implementation. Attachments of the Report: (1) Working agenda; (2) Name lists of officials and farmers interviewed during the survey; (3) Survey questionnaires and records of results; (4) Records of interview results; (5) Pictures of survey sites; (6) Important tables that cannot be included in the context of the report. 5. Competencies and Expertise Required for the Survey Contractor The poverty survey contractor must have the following competencies: (1) Being specialized divisions in the statistical department or schools of higher learning, research institutes in the project provinces or the Loess Plateau, who is competent in rural economic survey, poverty survey and PM&E, and familiar with the socioeconomic status of the project areas in the province or the Loess Plateau; (2) Having contracted for socioeconomic surveying or effects monitoring tasks in at least two international or domestic large scale poverty alleviation development projects, and having had excellent experienc e cooperating with the project owners; (3) In the task team recommended in the bid, there should be at least 3 people with the professional job titles of senior economist or senior statistician, while the other members of the team should have professional job titles of statistician or economist, etc. All the team members must have at least 5 years of experience in project M&E and socioeconomic surveys; (4) The task team members have experiences in surveying communities and farming households, and have skillfully commanded such working tools as sampling, questionnaire designing, participatory community surveys and statistics of data, etc.; (5) The contractor and the task team should systematically understand about CWMP, 102 China Watershed Management Project Final Report (June 2006) including the objectives and means of rehabilitation, livelihood improvements and poverty alleviation. 6. Schedule Stages of Work Major Tasks Time Preparatory stage Field survey stage Analyzing and integrating data Writing report Total 103 China Watershed Management Project Final Report (June 2006) 7. Annex: Poverty and Livelihood Monitoring Indicators and data collection requirements Data to be collected at village levels Indicators How data should be disaggregated: (Indicators marked in bold are recommended as Key Performance Indicators ) Watersheds Natural Villages Gender Poverty status of village (long term changes) l Number/percentage of households ü ü living below official poverty line l Total number of farming households in ü the village and classification of poverty ­ very poor households, poor households, middle households and above average households l Number of farming households with ü ü chronically sick family members (especially women) l Enrolment rate (of girl students) of ü ü the primary school in the village l Illiteracy rate of the villagers ü ü l Percentage of people going out as ü ü ü migrant labours in the population of the village l Percentage of population living in ü brick houses l Percentage of population living in ü earthen caves l Percentage of households above ü ü grain security line (Grain security line: 150 kilograms/person/year)/ Number of households without food security l Per-capita availability of terraced ü land l Per-capita availability of grass land ü 104 China Watershed Management Project Final Report (June 2006) l Per-capita availability of wood land ü Project impact indicators (direct project outcomes) 53 l Number of farming households ü ü covered by Loess 1 and Loess 2 l Per capita water consumption ü ü (liters/person/year) l Number of households with a water ü cistern l Water consumption for irrigating per ü mu of farmland (cubic meters/mu) l Number of farming households with ü ü access to safe drinking water l Number of farming households with ü ü access to electricity supply l Number of farming households without ü a toilet at home l Number of kilometers of village road ü ü constructed l Use of fuel wood from fuel wood ü ü plots (as share of total fuel consumption) l Increase of grain yields on newly ü ü terraced land l Increase of grassland available for ü ü livestock production Use of project services (as applicable): ü ü ü l Percentage of villagers participating ü ü in project training activities of Loess components, and the percentage of female participants l Percentage of villagers regularly ü ü attending Loess project meetings of the village, and the percentage of female attendants l Number of households being ü allocated newly terraced land l Number of households having ü established fuel wood plots 53 Indicators need to be customised according to project interventions 105 China Watershed Management Project Final Report (June 2006) l Number of households implementing ü soil conservation methods l Number of household participating in ü livestock projects Data to be collected at household level Incicators (Natural) Village Household level Gender level analysis analysis 54 disaggregate (Key Performance Indicators in bold) analysis Poverty status: ü l Domestic water consumption ü ü per day (liters/household) l Quantity of firewood ü ü ü collected per week (kilogram/household) l Health expenditures per ü ü household per year l Number of labor days lost per ü person per year due to illness l Education expenditures per ü ü household per year l Percentage of family debt in ü family income l Number of family members ü ü ü working as migrant labors l Number of non-agricultural ü ü labors in the family l Main food and cash crops ü ü produces l Percentage of annual food ü expenditures in annual total domestic expenditures l Number of months without ü ü food security l Percentage of annual ü expenditures on clothes in the annual total domestic expenditures 54 According to poverty levels groups 106 China Watershed Management Project Final Report (June 2006) l Dependent population of ü ü labors per household (the ratio of labors in the family : non- labor population in the family) l Number and varieties of ü livestock raised Project benefits (as applicable)55 ü l Being allocated newly ü ü terraced land; application of fertilizer and yields on newly terraced land. l Having established fuel wood ü ü plots: fuel varieties and yields. l Implementing soil ü ü conservation methods: types of methods and observed benefits. l Participating in livestock ü ü projects l Participating in training ü ü l other ü ü 55 Project benefits should be assessed for different poverty groups and in a gender disaggregate way. They would require collection of qualitative data and hence an open questionnaire. 107 China Watershed Management Project Final Report (June 2006) Terms of Reference: Baseline survey on Soil and Water Indicators 56 1. Background: CWMP has developed and finalized the system of indicators for M&E of sediment, which will become an important content for M&E of effects and impacts of the project measures implemented in CWMP, and also the basis for conducting specific M&E of sediments, the result of which will serve as one of the bases for effective management of CWMP. According to the particularities of sediment, the specific monitoring indicators will include the technical aspect and participatory aspect. In order to guarantee quality of M&E at the technical aspect, the Central PIO of CWMP decides to hire a scientific institute or monitoring organization with competent qualifications through open bidding as the contractor for M&E of sediments at the technical aspect. 2. Purposes and Tasks of Sediment M&E 2.1 Purposes of Sediment M&E The main purposes of conducting M&E of sediment indicators in the watershed are as follows: 1) To monitor and evaluate the sediment changes in the project area under the intervention of the project with the set of indicators for M&E of sediments already developed by the project, including the trends of changes; 2) According to the results of M&E, provi de basis for project management and improvement, as well as publicity of supervision and law enforcement; 3) According to the results of M&E, adjust and improve the sediment M&E indicator system of the project, including adjusting and improving the steps and methods to be taken. 2.2 Specific Tasks of Sediment M&E in the Watershed The organisation who undertakes the sediment M&E at the technical level of CWMP should effectively complete the following specific tasks: 1) Submit detailed work plan for implementation of sediment M&E (at the technical level) to the Central PIO for approval as required; The plan should include the locations for testing and observation (micro watersheds, and there should be criteria for selecting micro watersheds), time, staffing, steps, methodologies (including technical methodologies and sociological methodologies), tools and budgets, etc, and it should be submitted to the Central PIO for examination and approval; 2) According to the approved work plan and indicators for M&E of sediments, as well as relevant technologies and methods, conduct M&E of sediments with the assistance of County PMOs in the project area, and submit sediment M&E reports to Central PIO and Provincial PMOs (refer to the relevant tables hereinunder); 3) To complete other agreed tasks in the contract with Central PIO. 2.3 Capacity Requirements for Organisations: The organisation should have the following capacities and conditions: 56 Prepared by Liu Xiaoying, IRTCES 108 China Watershed Management Project Final Report (June 2006) 1) Be familiar with the ecological and social status of Yellow River Watershed, preferably being an organization from Yellow River Watershed; 2) The organisation should have at least 5 kernel members; At least 2 key technicians have participated in the Yellow River Sediment Changes Research Fund, and be familiar with the methods of calculating the water and soil conservancy benefits of water conservancy measures. 3) The organization must have Grade A or at least Grade B qualification in the field of water conservancy monitoring, or must be a hydrological organization above municipal level. 4) At least 3 staff members of the contractor hold the qualification certificates for water conservancy monitoring posts, be familiar the ecological and social status of the Loess Plateau and Yellow River Watershed, and have at least 3 years of experiences in studying or monitoring the Loess Plateau or water conservancy or ecological environment of Yellow River Watershed; 5) The contractor is scrupulous, serious and responsible in scientific research, and has certain level of authority and prestige in China; 6) Have team spirit; 7) Have relatively good skills to communicate with the government departments and local communities. 8) Whereas many factors are involved in calculating sediments, which are very complicated, it is suggested to set up a Steering Committee of 5 experts, so as to provide timely guidance. 4. Expected Results and Outputs to Be Submitted: The organization contracting for quantitative M&E of sediment changes impact should submit reports on quantitative M&E of sediments to Central PIO and Provincial PMOs, including such contents as the process, the results and suggestions. The reports will be in Chinese. 5. Input The input covers monitoring the indicators included in the list before througout the entire flood season, including field work, indoor integration of data and writing report. 109 China Watershed Management Project Final Report (June 2006) 6. Attachment of TOR Attachment 1: Indicators for Quantitative M&E of Sediment Changes Impact of CWMP Indicators on soil quality and fertility 1) Quick-acting nitrogen content (kg/ha): to be determined by Nitrate Reagent Power Colorimetry 2) Quick-acting phosphorus content (kg/ha): to be determined by Flame Photometry. 3) Quick-acting potassium content (kg/ha): to be determined by Flame Photometry. 4) Organic matter content (kg/ha): to be determined by Heavy Potassium Chromate Method. 5) PH value of soil: To be determined by Colorimetry. 6) Volume weight of soil (soil structure): To be analyzed by the Stove -Drying Method. 7) Moisture content of soil: The moisture content in unit volume of soil is to be analyzed by the Stove -drying method. Soil erosion indicators: 2 8) Erosion modulus (t/ km /yr): Volume of soil erosion in unit area of watershed in unit period time within the range of M&E. 9) Gully density (%): Length of gullies in unit area within the range of M&E. Rehabilitating benefit indicators: 10) Degree of rehabilitating (%): Percentage of rehabilitated area in soil erosion area within the range of M&E. 11) Preservation rate of the rehabilitating measure taken (%): Refers to the percentage of preserved area after taking rehabilitating measures in the rehabilitated area within the range of M&E. 2 12) Moisture-holding modulus of the rehabilitating measures (t/hm ): The difference between the flow-producing modulus before and after rehabilitating within the range of M&E. 2 13) Soil-holding modulus of the rehabilitating measures (t/hm ): The difference between the sediment -producing modulus before and after rehabilitating within the range of M&E. 14) Flood runoff reduction rate (%): Ratio of flood runoff flows produced by the same frequency of rain before and after rehabilitating within the range of M&E. 15) Volume of sediment impounded by gully construction measures (silt dam, small reservoirs, and key construction projects of gullies) (t/yr): 3 16) Runoff volume changes at the outlet of watershed (m ): Runoff volume changes at the outlet of watershed before and after taking rehabilitating measures within the M&E range. 110 China Watershed Management Project Final Report (June 2006) 3 17) Sediment volume changes at outlet of watershed (kg/m : Sediment volume changes at the outlet of watershed before and after taking rehabilitating measures within the M&E range. The indicator of new soil erosion caused by human activities: 18) Amount of new soil loss caused by human activities (t): Amount of new soil loss caused by such human activities as construction of road, and mining, etc. 19) Amount of waste dirt intercepted by human activities (t): Amount of waste dirt controlled by taking rehabilitating measures within the range of M&E. 2 20) Wind erosion modulus (t/km /yr): Amount of wind erosion on unit area of land in unit period of time within the range of M&E (for micro watershed which mainly has wind erosion). Water quality monitoring indicators: 21) Mineralization degree of water: 22) PH value of water: 111 China Watershed Management Project Final Report (June 2006) : Attachment 2 Method for Collecting Quantitative M&E Information about Sediment Changes Impacts of CWMP 1) Soil Quality and Fertility Proper number of sampling spots will be selected from typical farming households and typical parcels for monitoring, the ones with rehabilitating measures being taken and the ones without will be compared. Diagonal line sampling method will be adopted to the spots with relatively small area, while chessboard sampling method will be used to the spots with moderate area, and the "S" type sampling method will be used to relatively large area of spot. Soil samples will be taken and analyzed as per the GB6920-86 "Methodology for Monitoring and Evaluation in Loess Plateau Project with World Bank Loans. " 2) Water Quality Along with micro watershed monitoring, water samples will be taken and analyzed, and the micro watersheds being comprehensively rehabilitated and the ones unrehabilitated will be compared. The details of water sample volume and preservation methods can be referred to in GB6920-86 "Methodology for Monitoring and Evaluation in Loess Plateau Project with World Bank Loans." 3) Soil Erosion caused by Human Activities" According to the relevant spirits in "Circular on Conducting Nationwide Dynamic Sample Surveys to Soil Erosion" and the "Circular on Printing and Distributing the Implementation Plan for Surveying Soil Erosion Caused by Human Activities" by the Water and Soil Conservancy Department under the Ministry of Water Resources, implementation plans for surveying soil erosion caused by human activities and sample surveys of soil erosion will be worked out, so as to learn the situation of new areas and volumes of soil erosion, volume of disposed soil, and detained waste soil volumes caused by human activities, such as construction of road, drilling well, building house, water conservancy projects, mining, land exploitation, disposed soil etc. 4) Erosion and Runoff Sediments According to the area and distribution of soil erosion regions, runoff plots and runoff fields will be selected from each type of region, and observed by runoff and sediments. It is generally classified as monitoring moisture-holding and soil-holding benefits of individual measures and monitoring sediment benefits of comprehensively rehabilitated micro watersheds. The former is mainly monitoring runoff plots, to be supplemented with contrast gulley monitoring, while the latter is to establish runoff sediment observatory at the outlet of the watershed, and conduct comparative monitoring with unrehabilitated micro watersheds. The major monitoring items are precipitation volume, runoff volume and sediment volume. The basic principles for laying out monitoring spots: ? Mainly monitoring state-of-art micro watersheds rehabilitated in Loess 1 and 2 Projects, to be supplemented with on-surface 112 China Watershed Management Project Final Report (June 2006) representative micro watershed monitoring; ? Could basically cover the types of regions in the watershed, and control the trends of sediment changes in the region; ? Try your best to select the micro watersheds with favorable monitoring foundations, readily available monitoring conditions, and long series of monitoring, and fully take advantage of the original monitoring and analysis results; ? It should be based on the principle of necessity and possibility, minimize the monitoring costs to the greatest extent, and improve the monitoring efficiency and quality of results; ? Easy access to telecommunication and transportation. The layout of sampling spots should meet the requirements of different slopes and slope lengths, different types of erosions and different intensities of erosion, different rehabilitation measures and comparative analysis. Through multiple means and methods, such as artificial, remote sensing or GPS technologies, etc., and by comparison, statistical calculation, you will get the results. Firstly, typical observatories will be established to monitor the changes of sediments before and after the micro watershed is rehabilitated; Secondly, runoff plot observatories will be set up, and the plot area will be 5mx20m, for collecting the average data of runoff and sediment being detained by the rehabilitating measures of terraced fields, sloped farmland, fruitery, afforested land, grassland and gully rehabilitating projects. Thirdly, set up and take advantage of the existing raingages, or obtain data from the meteorological departments or hydrometric stations, so as to monitor the rainfalls of the project area. Fourthly, lay out such observation sections or observatories as silt dams, small reservoirs, backbone constructions, and gully head conservancy projects, for observing the volume of incoming sediments per flood, the volume of sediments detained, and the volume of sediments drained, etc. Fifthly, the volume of wind erosion (t) in micro watersheds mainly have wind erosion could be accessible in the forestry department, and the frequency of dust storms could be obtained from the meteorological department. 113 China Watershed Management Project Final Report (June 2006) : Attachment 3 Method for Analysis of Quantitative M&E Information about Sediment Changes Impacts of CWMP For different indicators for M&E of sediment changes impact, the information collected will be separately used to calculate / count each specific indicator, and the particulars are shown as follows: 1) Regarding calculation and analysis of moisture-holding and soil-holding modulus Soil- holding modulus (and moisture-holding modulus) are the differences of sediment- producing modulus (and flow-producing modulus) between the rehabilitated project areas and the areas without being rehabilitated by the project. Under normal circumstances, such calculation is used to analyze accumulative values of multiple precipitations monitored during the entire flood season within the project area. Under special circumstances, it is used to calculate the results monitored in certain one or two extraordinary rainstorms in the project area. For micro watersheds without laying out references, previous monitoring data of Water Conservancy Stations or Hydrological Stations in the same type of area could be based upon, and sediment-producing modulus (and flow-producing modulus) of an unrehabilitated area during the flood season or a similar precipitation could be selected for comparison and calculation. 2) The analysis about the flood-reducing and sediment-reducing effects of comprehensively rehablitating tributaries of Yellow River need to be calculated by the Hydrologically Zoned Method and Water Conservancy Zoned Method. Hydrologically Zoned Method conduct precipitation ­ flow-producing ­ sediment-producing relationship analysis, while Water Conservancy Zoned Method adopts different flood-reducing and sediment-reducing modulus of different rehabilitating measures for calculation. Refer to details of the analytical and calculating methods and formulas of the Water Conservancy Method and Hydrologic Method in "M&E Methodology" of Loess Plateau Project funded by World Bank, and the Chinese national standard on water and soil conservancy, GB/T 15774- 1996 "Method for Calculating Water and Soil Conservancy Comprehensive Rehabilitating Benefits". The results respectively calculated by Water Conservancy Method and Hydrologic Method should be verified, and it is required that the verification coefficient J between these two methods should be less than 0.2%. The formula is: J= (? ST- ? SC)/? ST, in which ? ST is the result calculated by Hydrologic Method, while ? SC is the result calculated by Water Conservancy Method. 114 Attachment 4: Reference Form of Report on Quantitative M&E of Sediment Changes in CWMP The same as Table 3 in the main context. Baseline/Interim/Final Report On Quantitative M&E of Sediment Changes of CWMP (At Micro Watershed Level) Submit to: Central PIO and Provincial PMOs Submitted by: Contractor for quantitative M&E of sediment changes impact Time of Submission: l Backgrounds and Purposes of the Task: l Indicators and methods for M&E; l Working steps (including collection and analysis of information); l Outputs; Refer to Table 4 and statements; l Evaluations and conclusions of analysis; l Mainly obtained by the expression formula; l Suggest major findings. l Attachments; ¦ Records of working schedule; ¦ Records of survey tables; ¦ Records of observation process and results; ¦ Records of working interviews; ¦ Photo records of working process; ¦ Miscellaneous. China Watershed Management Project Final Report (June 2006) Terms of Reference: Baseline on Biodiversity Indicators 57 1. Basic Context China Watershed Management Project (CWMP) has developed and identified indicators and relevant M&E flow and approaches for quantitative M&E on biodiversity impact. The M&E system is to be taken as major contents for M&E of project impact in CWMP, as well as a gist for undertaking concrete quantitative M&E on biodiversity impact, which result will be regarded as one of basis for effective management to CWMP. In terms of speciality of biodiversity, M&E indicators for biodiversity of CWMP are consist of technical aspect and participation aspect. In order to ensure quality of QMEOBI in technical aspect, central Project Implementation Office of CWMP has decided to hire qualified research institutions to undertake "Quantitative M&E on biodiversity impact of CWMP"by mean of bidding. 2. Purpose and Contents of the Biodiversity Survey 2. 1 Purposes for Biodiversity Survey which are: 1) Using biodiversity indicator system that was developed by CWMP, to monitor and evaluate biodiversity change under intervention of CWMP in project area, including trend of change; 2) According to M&E result, to provide basis for project management and improvement; 3) Based on M&E process and result, to reset and improve QMEOBI indicator system and its steps and approaches. 2.2 Concrete Assignment for the Survey Institution that undertake the biodiversity survey for CWMP should be capable of fulfilling the following concrete assignments: 1) Upon request, submit detailed work plan script for undertaking QMEOBI(in technical aspect) to the Central Project Implementation Office for approval; 2) According to approved work plan and indicators and approaches for biodiversity survey (see appendix 1-3 of the assignment script), to undertake biodiversity survey with assistance from county and township PMOs in project areas, and submit M&E report to the Central Project Implementation Office and provincial project office (see appendix 4 for report format); 3) To accomplish other assignments that are stated in contract with Central Project Implementation Office. 3. Qualification Any institution of undertaking the biodiversity survey is required to have the following capabilities and qualifications: 57 Prepared by Deng Weijie, ITAD 116 China Watershed Management Project Final Report (June 2006) 1) Know well about eco-social context of Huanghe River watershed, better be non- governmental organization (including university and research institution) in Ganshu/ Shanxi Province); 2) Has been established for at least 3 years; 3) With at least 5 core members; 4) There are at least 3 members are knowledgeable in ecology, know well about eco- environment of Huangtu Plateau and Huanghe River watershed, and have at least three years of work experience in eco-environment research and conservation in Huangtu Plateau and Huanghe River watershed. 5) There are at least 3 among members are experienced in community development, know well about participatory approach, and have more than 3 years of community development work experience. 6) Has team spirit 7) Good communication skills with local government authorities and communities. 4. Expected Result and Submitted Outcome Institution of undertaking the survey shall submit survey reports to central project implementation office (in Xi'an), and provincial project office, in which aspects such as process, result, and recommendation, and so on should be included. Reports should be written in Chinese. 5. Time Contribution 20-30 days, including field survey and indoor data sorting and report writing. 6. Appendix for the Terms of References 117 China Watershed Management Project Final Report (June 2006) Annex 1.Indicators for Quantitative M&E on Biodiversity Impact of CWMP Indicators to monitor long term changes of biodiversity in Watershed: 1) Vegetation coverage: percentage of area covered by vegetations in M&E zone; 2) Diversity Index for flora community: floral diversity in various physical geographic categories. It is indicated by Shapon-Wiener index; 3) Diversity index for fauna community: diversity of avifauna, small-sized beast, and insect species in various communities, showed by Simpson's index; 4) Richness for fauna species: categories of small-sized beast, avifauna, snake and insect species in unit area, to measure richness of species in community, which is formulated as: R(richness)=N(numbers of categories)/S(Size of area); 5) Comparative density for species: numbers of individuals of a certain species in unit area, to estimate comparative density of the species, yet count only comparative density of honeybee, butterfly, and snake species. Indicators to monitor direct project outcomes in villages and on farm plots: 6) Richness for crop varieties (in farming area): numbers of crop varieties in unit area, indicated by variety richness, equation is: R(richness)=N(numbers of crop varieties)/S(size of sample farming land); 7) Richness for economic forest species(in planting area): numbers of economic forest species in unit area, is also showed by species richness, which is formulated as: R(richness)=N(numbers of categories for economic forest species)/S(size of sample planting area); 118 China Watershed Management Project Final Report (June 2006) Annex 2. Approach for Collection of data on biodiversity 1) Transect Walk for avifauna survey: in each watershed, set up a sample route of 1000 meters long according to vegetation type in three different habitats respectively, which are village, plantation and sloping field. While walking slowly alone the route by even velocity of 1-3 KM/h, observe numbers and species of all birds within 25 meters on both sides of the route by telescope. During observation, record amount and characteristics such as body color, speculum(numbers, color), superciliary stripe, and so on of unacquainted birds, and identify their names and species later on by referring to avifauna illustrated handbooks. Survey is suggested to be taken in 7-9am in the morning, with bio- telescope and avifauna handbook as major tools, since it is peak period for avifauna activity, accordingly, data collected in this period are relatively complete and reliable. During the survey, repeat survey at each route for three times. 2) Small-sized mammal survey: adopt the same sample stand method and zonal sampling method, catch small mammals by iron rattrap. When conduct survey in human habitations, can adopt zonal sampling method, that is to place a rattrap along houses and buildings every ten meters, a sample zone is usually as long as 200 meters. While at grassland, forestland, and farming land, should use sample stand method, set sample with size of 150mX150m and shape of grid, place a rattrap every 10 meters in sample grid, that is 256 rattraps in total in each grid. Usually two sample grids are randomly selected in each habitat and are at the same latitude(with ±50 m difference) if they are located in the different habitats of the same site. The average numbers of rattraps in each habitat is usually 1024. Traps are better placed at 17:00-18:00 in the afternoon with fresh peanuts as baits, and are checked at 8:00-12:00 in the morning of the following day, with one check-up per day during two-day trap placement in each sample. When mammals are caught by clean rattraps, new baits should be refilled to replace that eaten by animals. Traps that caught animals should have been cleaned in flowing water for at least 12 hours. 3) Survey for insect: Adopt sample stand method. Select suitable grassland, farmland, and forestland (20mX30m each), and catch insects at 10:00-12:00 and 13:00-15:00 when insects are move actively in high temperatures. Identify insect specimens according to illustrated handbook. Then do diversity analysis with above approach. 4) Sample stand method for Foliage: Sample stand method is the most often used survey approach in phytoecology, which theoretical foundation is random sampling principle of probability theory. When samples reach certain numbers, their representative of the whole is acceptable. There are three key points in foliage diversity survey by using sample stand method: the first, randomicity of selecting sample stands. The second, samples should reach enough amount, which is determined by heterogenization of holistic watershed. The third, size of sample is not fixed, for instance, a sample of grassland can be either 1m X 1m or 2m X 2m. When setting sample, need to differentiate differences in terms of altitude, gradient, slope leaning direction, and distance to farmland, road, and village. Size of sample need to be determined by "Species-area Curve" in advance. The first step of survey is to select a route in M&E area, and mark survey site on topographic map, then confirm size of sample, which is 10m X 10m for arbor community, 5m X 5m for shrub community, and 1m X 1m for meadow community, the coming step is to count amount of species and individuals of each species in each sample. Take whole individual plant back to survey station for further identification if there is any unacquainted species. Meanwhile, estimate size proportion of trees and herbages' projection to sample size, to compute vegetation coverage. Community type and human's intervention should be narrated as well. Information such as coordinates and altitude can be measured by hand-carry GPS. Sample size and projection measurement can be recorded in the table below. 119 China Watershed Management Project Final Report (June 2006) Size and Numbers of Sample Stands Category Size Standard( m2) Numbers of Total area of sample stand sample stands( m2) Arbor(economic 10m×10m forest) Shrub(crop) 5m×5m herbage(crop) 1m×1m 120 China Watershed Management Project Final Report (June 2006) Annex 3. Approach for Analysis of Data on Biodiversity 1) a Vegetation coverage: including forest l nd coverage and meadow coverage, which is percentage of area covered by trees(arbor and shrub)or herbage projection in sampling area; 2) Diversity index for Flora Community: caculated by Shannon-Wiener index, S 1 H ' = 3.3219(log10 N - N n log i 10 ni ) i =1 N stands for sum of individuals of all species, ni stands for sum of individuals of species;, S is numbers of species; 3) Richness of crop variety (in farming area): R(richness)=N(numbers of crop varieties in sample farming land)/S(size of sample land); 4) Richness of economic forest species (in planting area): R(richness)=N(numbers of economic forest species in sample land)/S(size of sample land); 5) Diversity index for fauna community: Use Simpson's index to count species diversity of avifauna, small-sized animals and insects in various communities respectively. Simpson's index=probability that two randomly selected individuals belong to different species=1-probability that two selected individuals belong to the same species. If individual proportion of a certain species i in its community is showed by Pi, accordingly, the combined probability of two randomly selected individuals of the same species is ( illustrated as [ Pi) ( Pi) ] , or( Pi) 2. The sum of probabilities of all species in the community is Simpson's index. O stands for Simpson's index, Pi is proportion of individuals for species i in community. 6) Richness of fauna species : respectively calculate categories of small-sized beast, avifauna, snake, and insect species, and so on in unit area, which can be showed by a equation: R(richness)=N(numbers of small-sized beast, avifauna, snake, and insect species)/S(size of sample); 7) Comparative density of species: respectively compute numbers of species of honeybee, butterfly, and snake in unit area, which can be formulated as: D( density) =N ( numbers of individuals of a species in sample site ) /S ( Size of sample) 121 China Watershed Management Project Final Report (June 2006) Annex 4. Reporting Format for Biodiversity Survey Baseline/Mid-term/Final Report for Quantitative M&E on Biodiversity Impact of China Watershed Management Project(for Small Watershed) submitted to? Central project Implementation office? PIO? ? Provincial Project Management Office? PMO? Submitted by? Contract institution for quantitative M&E on biodiversity impact Time for submission ? l Background and goal for mission; l Indicators and approaches for M&E; l Work steps (including information collection and analysis); l Output (refer to comparation and analysis table of biodiversity impact) l Analyze evaluation, conclusion l Main findings and recommendations l Appendix n Record of work schedule; n Record of Questionnaire; n Record of observation process and result. n Record of interview; n Photo record of work process; n others 122