WATER P-NOTES ISSUE 34 APRIL 2009 47933 Monitoring and Evaluating Agricultural Water Management Projects Better Tools for Better Results I ncreasing agricultural production is necessary to feed growing populations, raise the incomes of Box 1. Typical components of AWM poor farmers, and boost national export revenues. projects But in poor, water-scarce areas of the world, it is impossible to raise production without first finding · Soil and water conservation in watersheds ways to get more out of limited supplies of water, · Small-scale irrigation the ultimate scarce resource. Increasing the effi- · Groundwater development and management ciency with which water is used is one of the chief · Construction, rehabilitation, modernization, goals of agricultural water management (AWM). and upgrading of irrigation and drainage sys- The World Bank is engaged in a wide variety of tems, flood-protection systems, and dams AWM projects, some common features of which are · Formation and support of water-user associa- detailed in Box 1. tions and support to water users for greater Continual improvements in the efficiency of agricultural production water use depend on close monitoring and care- · Improved management, operation, and mainte- ful evaluation. But despite the scale and scope of nance of irrigation and drainage systems AWM projects around the world, and their critical- · Better management of water resources ity to global efforts to reduce rural poverty and increase food security, the monitoring and evalu- · Reform of water sector agencies ation (M&E) of AWM projects need considerable · Capacity building and training, drafting of leg- improvement. islation, improving gender equity in rural areas, and protecting the environment. A 2006 review of the World Bank's AWM portfolio by the institution's Independent Evalua- tion Group (IEG) identified several areas where M&E was not being used to full effect. A majority of AWM projects lacked concrete information on the short. Only one in nine AWM projects was designed progress being made toward project objectives. In in such a way as to permit rigorous impact assess- fact, in many cases, no evaluation at all was taking ment. The report stressed the need for greater place. Only one in three AWM projects established emphasis on outcomes and impact measurement in baseline data before the project began, and fewer project design. It also concluded that the effective than half established baselines prior to project implementation of projects was routinely compro- completion. As a result, it was often difficult to mised by inadequate supervision. assert with any degree of certainty whether a project In response to the IEG report, the Agricultural achieved its goals or not, or when and where it fell and Rural Development Department at the World This note highlights important information from the "Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects," Report 44799, Agricultural and Rural Development Department, World Bank, Washington, DC. Readers may download the complete cdrom toolkit from www.worldbank.org/water. WATER P-NOTES Bank developed the "Toolkit for Monitoring and structures; (iii) the creation and support of water Evaluation of Agricultural Water Management user associations; and (iv) capacity building in water Projects." The toolkit enables project managers management. Second, projects in which AWM to incorporate M&E into their projects, beginning makes up less than half of the overall development with the design phase, allowing them ultimately to apparatus should focus M&E on implementation demonstrate convincingly the outcomes and impact and process. These types of process projects should of their projects. It also helps users determine how be results-oriented and flexible, so that they can best to monitor and evaluate specific projects, how adapt to the competing demands of non-AWM to ensure the optimum collection and analysis of actors. Finally, M&E is strongest when there is a relevant data, and how to determine the reasons for participatory quality to data collection and evalua- the success or failure of a project. It also contains tion, so that stakeholders are invested in the proj- some general principles that contribute to a greater ect's successful implementation and aware of its overall understanding of the challenges facing successes and failures. AWM projects and the power of M&E to improve project outcomes. M&E must contend with challenges in most AWM projects. For example, the presence of many The toolkit is designed for seven types of users: stakeholders with competing interests will compli- World Bank managers and task teams; government cate the assessment of project outcomes. Often, departments and agencies; managers and staff too, multiple projects are being undertaken in a responsible for implementation or management given region, with varying timelines, which may of AWM projects; farmers and other stakeholders in the project area; consultants and other exter- nal organizations providing assistance on project design, implementation, and management; wider stakeholders from civil society with an interest in Box 2. Monitoring and Evaluation: project outcomes; and other financing agencies definitions and concepts involved in AWM. Monitoring--the continuous gathering of informa- tion during a project's implementation--informs Better M&E Can Improve the Design, stakeholders of the project's progress and suggests where adjustments may be needed. The data accu- Implementation, and Assessment mulated through monitoring often make it easier to assess the outcomes of the project and the sources Good M&E, as described in Box 2, can improve of those outcomes. project design in several important ways. First, Evaluation, by contrast, assesses the outcomes and M&E can help to establish the causal connec- impacts of a project relative to its objectives and tions between the logical framework of an AWM expectations and helps explain variations between project and the emphasis on results analysis that them. Good evaluations allow project managers to has become an essential part the World Bank's adjust funding allocations in light of comparisons approach to development. Establishing parameters between project components and across projects. to be used in the M&E process, one of the nine key They also make it possible to determine the most cost-effective ways of realizing the objectives of a steps in the toolkit's methodology (Figure 1), also given policy, whether in AWM or another area. assures that the project has realistic, achievable and measurable objectives--the key to assessing its M&E are complementary aspects of project design degree of success. Finally, the collection and analy- and assessment, in that monitoring raises questions for evaluation, while evaluations flag new monitor- sis of relevant project data provides an indicator of ing needs. M&E also can be used to diagnose and management capacity. address specific problems of project implementa- To improve M&E design, the toolkit makes sev- tion. In process-oriented projects, in particular, eral recommendations for future projects. First, M&E where the goal is to find the best way to meet proj- ect objectives, M&E can provide direction on how to should extend beyond technical matters to (i) the get there. In "blueprint" projects where the route is institutional aspects of AWM projects; (ii) the civil specified in advance, M&E can provide information works components of projects, such as improve- for reviewing the project's relative success. ments to managerial, operational, and maintenance 2 ISSUE 34 · APRIL 2009 Figure 1. Steps in planning a results-based M&E system Conduct a Agree on outcomes Identify baselines Plan monitoring, Identify necessary readiness and objectives and plan data data analysis, conditions and assessment collection and communications, capacities analysis and reporting 1 2 3 4 5 6 7 8 9 Establish the Select indicators Select results Plan critical purpose and and evaluative targets reflection and scope of M&E framework interim evaluation Source: Adapted from Kusek and Rist 2004. affect the ability of managers to accurately assess development focuses on causal connections that the success of a specific project. AWM projects are can be clearly established (using logical frame- also becoming increasingly decentralized, making work analysis). Following the classic "if, then" it more difficult to assure that a specific project is logical paradigm, a particular project's design implemented, managed, and monitored in the way should predict that certain outcomes will occur if it was intended. the project's various components are implemented The wide variety and sheer number of AWM properly. projects in which the World Bank is engaged are Implementation monitoring, by contrast, mea- matched by a corresponding range in the techni- sures the effectiveness and efficiency of the project's cal and management capacities of individual and process, paying close attention to the management institutional actors, which means that projects must information systems put in place during and after be tailored to the specific constraints of a project the project. The emphasis therefore is not on the area. Conditions external to the project itself may narrow technical qualities of the project, which are compromise project effectiveness--and inevitably relatively easy to measure, but instead on manage- reality is more complex than can be accounted for ment personnel and institutions. in a project's design. Nevertheless, M&E provides an important tool for determining a project's effec- Results monitoring is more complex than imple- tiveness in achieving articulated objectives. mentation monitoring. The complexity arises in part from measurement challenges, such as establishing a pertinent time horizon for evaluating a project's Results and Implementation impact. Additional complexity derives from the dif- Monitoring: New Standards for ficulty of attributing changes directly to the project, AWM Project Design given the many variables external to the project that may affect its outcomes. Results monitoring measures the extent to which These challenges are worth overcoming, how- a given project contributes to the overall objec- ever, and AWM projects should be designed to tives of AWM, as well as the extent to which actual achieve definite, measurable outcomes that can be outcomes match the outcomes anticipated in the assessed in light of the five criteria of logical frame- project's design. A results-based approach to work analysis: 3 WATER P-NOTES · The impact that can be directly attributed to the The M&E toolkit helps managers ensure that project AWM projects are designed with these criteria in · The relevance of the project to meeting over- mind. The ultimate goal, of course, is to see that arching policy objectives World Bank projects achieve the most beneficial impact for their intended recipients--a goal that · The effectiveness of the project in meeting is more likely to be achieved where rigorous M&E those objectives techniques permit timely adjustments and cor- · The relative efficiency of the project in terms of rections to ongoing projects and improvements maximizing impact while minimizing cost, and to the design of future projects. In a world where agricultural demand is constantly increasing and · The sustainability of the project--that is, the the resources available to meet that demand are likelihood that it will continue to provide benefits stretched to the limit, M&E are invaluable tools for once the external funding has been eliminated. making the most of what we have. The Water Sector Board Practitioner Notes (P-Notes) series is published by the Water Sector Board of the Sustainable Development Network of the World Bank Group. P-Notes are available online at www.worldbank.org/water. P-Notes are a synopsis of larger World Bank documents in the water sector. 4 THE WORLD BANK | 1818 H Street, NW | Washington, DC 20433 www.worldbank.org/water | whelpdesk@worldbank.org