Environmentally and Socially Sustainable Development Unit Europe and Central Asia Region Good Practices in Monitoring & Evaluation and Management Information Systems for Competitive Grant Programs in ECA Countries October 2003 Jitendra Srivastava Jacob Kampen Gary Baker Derek Byerlee Meeta Sehgal The World Bank Copyright © 2003 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. All rights reserved Manufactured in the United States of America The Europe and Central Asia Environmentally and Socially Sustainable unit (ECSSD) distributes this working paper to disseminate findings of work in progress and to encourage the exchange of ideas among Bank staff and all others interested in development issues. This paper carries the name of the author(s) and should be used and cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. For submission of comments and suggestions, and additional information, including copies of this report, please contact Sharifa Kalala at: 1818 H Street N.W. Washington, DC 20433, USA Email: skalala@worldbank.org Cover photos by:­ Rufiz Vakhid Chirag-Zade -A couple farming in Azerbaijan, T.V. Sampath - a group examining rows of sugar beet at Sharur raion in Shafag farm off Nakhichevan, AR of Azerbaijan & Crystal Graphics -woman at computer and hand writing on paper. Printed on recycled paper TABLE OF CONTENTS Foreword ............................................................................................................................................... ...v Acknowledgements......................................................................................................................................vi Glossary ....................................................................................................................................................vii Definitions ................................................................................................................................................viii Executive Summary...................................................................................................................................... 1 1. THE IMPORTANCE AND ROLE OF M&E AND MIS IN CGPs. ............................................... 3 (a) Competitive Grant Programs in the ECA Region ..................................................................... 3 (b) Structure of CGP Management ................................................................................................. 4 (c) Importance of M&E in CGPs ................................................................................................... 5 (d) Levels of M&E in CGPs ........................................................................................................... 5 (e) The Critical Link Between M&E and Timely Flow of Funds................................................... 6 (f) The Importance of Information Management and Utilization in CGPs .................................... 6 2. THE BASICS OF MONITORING AND EVALUATION OF CGPs............................................. 7 (a) Definitions................................................................................................................................. 7 (b) The Functions of an M&E System............................................................................................ 8 (c) The Responsibilities for M&E in a CGP................................................................................... 9 (d) Financial Management of M&E in a CGP ................................................................................ 9 (e) Procurement and M&E in a CGP.............................................................................................. 9 (f) Special Challenges in M&E for CGPs .................................................................................... 10 3. A FRAMEWORK FOR M&E SYSTEMS IN CGPs .................................................................... 11 (a) CGP/Sub-Project Design and M&E........................................................................................ 11 (b) Sub-project Implementation Plans .......................................................................................... 11 (c) Monitoring Tools - "Key Performance Indicators" and "Milestones".................................... 11 (d) Tranche Release Requests....................................................................................................... 13 4. MONITORING IMPLEMENTATION OF CGP SUB-PROJECTS............................................. 16 (a) Routine M&E and Sub-project Progress Reporting by Grantees............................................ 16 (b) Sub-Project Completion Report .............................................................................................. 16 (c) Technical, Financial and Procurement Reporting ................................................................... 16 (d) Data Quality, Evaluation and Analysis ................................................................................... 17 (e) Supervision and over-sight by the CGP Secretariat ................................................................ 17 (f) Sub-project Visits.................................................................................................................... 18 (g) Problem Sub-projects .............................................................................................................. 19 (h) Evaluation/Grading of CGP Sub-Projects............................................................................... 19 (i) Financial and Technical Audits............................................................................................... 20 iii 5. M&E AT THE CGP PROGRAM LEVEL .................................................................................... 21 (a) Monitoring Indicators.............................................................................................................. 21 (b) Sub-project Ratings................................................................................................................. 21 (c) Special Studies and Reviews................................................................................................... 21 (d) Sharing Experience Across Sub-projects ................................................................................ 22 6. SETTING UP THE M&E SYSTEM.......................................................................................... 23 (a) Key Issues to be Addressed..................................................................................................... 23 (b) Staffing an M&E System ........................................................................................................ 23 (c) Staff Training .......................................................................................................................... 23 (d) Monitoring M&E Performance and Fine-tuning..................................................................... 24 7. THE MIS IN A CGP...................................................................................................................... 25 (a) The Basics ............................................................................................................................... 25 (b) Essential Aspects of MIS Establishment and Operation ......................................................... 26 (c) What Type of MIS is Required ...............................................................................................26 (d) Design Requirements .............................................................................................................. 27 (e) Development and Installation.................................................................................................. 29 (f) Cost Aspects............................................................................................................................ 30 (g) Monitoring MIS Performance and Organizing Maintenance/Upgrading................................ 30 8. IMPACT ASSESSMENT OF CGP SUB-PROJECTS............................................................... 32 (a) Impact Assessment.................................................................................................................. 32 (b) A Practical Approach to Impact Assessment ........................................................................... 33 (c) Beneficiary Assessment .......................................................................................................... 35 (d) Assessing Institutional Impacts............................................................................................... 36 Annexes: Annex 1: Detailed Outline for CGP Sub-project Proposals.................................................................... 37 Annex 2: Templates of Monitoring Forms Required for M&E ............................................................. 42 Annex 2b: An example of the Completed Monitoring Forms ................................................................. 44 Annex 2c: Secretariat's Sub-Project Assessment and Comment Form.................................................... 46 Annex 3: Format for a CGP Sub-Project Completion Report.................................................................... 47 Annex 4: M&E in the Nepal Hill Agricultural Research Project .............................................................. 48 Annex 5: Key References........................................................................................................................... 49 List of Boxes: Box 1: The M&E System Used in WB-Financed Projects....................................................................... 7 Box 2: Ecuador-Program for Modernizing Agricultural and Livestock Services; Sub-project Visits ....................................................................................................................... 18 Box 3: Ecuador PROMSA: Project "Alert Status"................................................................................. 19 Box 4: Sharing Experience Across Sub-projects.................................................................................... 22 Box 5: Ensuring Adequacy and Usefulness of an M&E System at CGP level...................................... 23 Box 6: Requirements for an MIS to Monitor CGP Implementation for the Agricultural Research Council (ARC), Ministry of Agriculture and Forestry, Croatia.................................................. 29 Box 7: Ecuador- Program for Modernizing Agricultural and Livestock Services ................................. 35 Box 8: Impact Evaluation in PRONATTA Colombia............................................................................ 35 List of Tables Table 1: Key Performance Indicators for a CGP Sub-Project on Improved Vegetable Varieties...................................................................................................................................... 12 Table 2: Time-bound Schedule of Tranche Payments for a Hypothetical Maize Improvement CGP Sub-project .................................................................................................. 13 Table 3A Example of a Completed Tranche Release Request Form ....................................................... 14 Table 3B: Explanation of Sub-project Implementation Shortfalls to be Attached to Tranche Request Release Form.............................................................................................................................. 15 List of Figures Figure 1: Flow of Net Benefits from Research .......................................................................................... 33 Figure 2: Adoption of Technologies .......................................................................................................... 34 FOREWORD There is increasing use of "Competitive Grant Programs" for financing agricultural support services, research, community development programs and environmental projects worldwide. This trend is reflected in the Bank's Eastern Europe and Central Asia (ECA) region, where the number of countries using Competitive Grant Programs (CGPs) has been steadily growing. CGPs for agricultural knowledge systems are under implementation in Romania, Georgia, Macedonia, Turkey, Armenia, Croatia, and Azerbaijan. Other projects with CGP financing are under preparation in Albania, Croatia (second project), and Kazakhstan. The guidelines provided in this paper have been developed by drawing on the experiences and lessons learned in implementing Competitive Grant Programs not only in Eastern European and Central Asian countries but also in other regions served by the World Bank. The paper is one in a series of training manuals developed under the Regional Initiative on Reforming Agricultural Knowledge Systems in Europe and Central Asia. It served as a basis for a regional workshop held in Rovinj, Croatia, in September 2002. The findings and recommendations developed at that workshop have assisted in the finalization of this document. This "good practices" paper focuses primarily on the monitoring and evaluation of CGPs, a key element in identifying and resolving implementation problems, in measuring outcomes and improving future performance. I believe the document will be very useful for grant recipients, staff of Project Implementation Units and World Bank Task Team Leaders involved with Competitive Grant Programs in Europe and Central Asia and possibly in other regions. Laura Tuck Director Environmentally and Socially Sustainable Development Unit Europe and Central Asia Region ACKNOWLEDGEMENTS The authors would like to express their sincere appreciation for the useful comments and suggestions received from staff in several departments of the Bank that contributed significantly to the development of this document. Sincere thanks are extended to Naushad Khan whose keen insights and thoughtful analyses proved invaluable in the preparation of this good practices paper. The paper also greatly benefited from the contributions of Derek Byerlee and Gary Alex (Agricultural Rural Development Department) whose technical expertise helped to strengthen the document at every stage of development. In addition, the authors wish to thank Aleksander Nacev for his pragmatic ideas that served to ensure that the good practices recommended in the document were appropriate for the specific environment of ECA countries. The authors are also grateful to Irina Iacovlenko for translating the document into Russian and Sharifa Kalala for processing the paper. The support and guidance provided by Marjory-Anne Bromhead, Sector Manager and Laura Tuck, Director, Environmentally and Socially Sustainable Department were critical in developing this document. Financial support from Europe and Central Asia regional management, through the "Regional Initiative on Reforming Agricultural Knowledge Systems in Europe and Central Asia", made it possible to undertake this important exercise and the authors wish to express their appreciation for their continued support. GLOSSARY AKIS Agricultural Knowledge and Information System ARC Agriculture Research Council of Croatia CAC Central Asian Countries CGB Competitive Grants Board CGP Competitive Grant Program (or Project) CGS Competitive Grant System ECA Europe and Central Asia ECSSD Environmentally and Socially Sustainability Development ITQ Invitation to Quote KPI Key Performance Indicator MIS Management Information System M&E Monitoring and Evaluation NGO Non-Governmental Organization OVI Objectively Verifiable Indicator PMU Project Management Unit PIA Project Implementation Agency SOE Statement of Expenditure TTL Task Team Leader WB World Bank DEFINITIONS Definitions Examples Inputs Resources needed to carry out the activities Scientists' time, technicians' time necessary to implement the (sub-)project. equipment, operating costs, travel costs, etc. Activities Implementation actions taken to produce the Research trials, surveys, training, outputs or products from the project and development of new methodologies, generate the desired outcomes and impacts construction of facilities, etc. Outputs Direct deliverables that the project is obligated Progress reports, research studies, to produce and that are within the control of the manuals, new varieties developed grantees to deliver as a result of project from breeding programs, etc. activities. Outcomes The initial changesusually in target group Adoption of new technologies (such behaviorthat are induced by the outputs of as improved seeds); investment in the project. processing and production; etc. Impact The ultimate changes in social indicators Income increases; deforestation resulting from the outcomes of the project. reduction; improved water quality; reduction in malnutrition; etc. Monitoring Routine collection of information to track Routine reporting; field visits; sub- implementation progress, measuring efficiency, project supervision, client surveys, and questioning "whether the project is doing etc. things right." Evaluation Analyzing information, ex-post factual Project performance ratings; policy assessment of effectiveness and impact, decisions to continue or change confirming project expectations, measuring implementation plans; resource impacts, and questioning, "whether the project allocation decisions relating to is doing the right things." program or sub-project financing; etc. Logframe A logframe in a (sub)-project presents a sound Aid for (sub)- project design, cause-and effect link from inputs and activities implementation, and M&E found in to outputs to outcomes to goals (sub)-project concept document, and (sub)-project proposal document Objectively Describes the expected inputs-outputs- Amount of technical assistance in a Verifiable outcomes ­impacts concerning a (sub)-project. (sub-project), multiplication of Indicators They should be objectively verifiable in terms certain quantity of seed, adoption of of quality, quantity and time. new technologies in a specific time and specific area. EXECUTIVE SUMMARY Introduction To assist countries in developing successful Competitive Grant Programs, the World Bank through the "Regional Initiative on Reforming Agricultural Knowledge Systems in Eastern Europe and Central Asia," has organized a series of training workshops for country staff charged with Competitive Grant Program (CGP) implementation. These have provided a useful forum for the exchange of ideas and experiences, identification of common issues and constraints, and development of better practices for the design and implementation of CGPs. At the regional workshop on "Good Practices in Procurement, Financial Management and Disbursement of CGPs" held in November 2001 at Baku, Azerbaijan, workshop participants expressed serious concerns about the adequacy of monitoring and evaluation arrangements in their CGPs. It was therefore agreed that the next workshop should focus on both the monitoring and evaluation of CGPs and on the establishment of an effective management information system. This workshop was held in Croatia in 2002; the findings and recommendations developed at that workshop have assisted in the finalization of this paper The achievement of CGP objectives depends on the quality of implementation and the results achieved by the sub-projects. This often depends in turn on the commitment of the sub-project grantees to implement the work according to the agreed contracts and also on the thoroughness of supervision and guidance. The quality of the latter is contingent upon having available up-to-date information concerning the implementation of the sub-project. It is therefore important that sub-projects are monitored and evaluated, so that any problems encountered may be quickly resolved and lessons learned fed back into the implementation process. A first step in the design of a CGP for agricultural knowledge services is to agree within the country on broad research and extension priorities adapted to new farm systems and changing market realities. A second step is to establish a governance mechanism for receiving proposals and awarding contracts. A working Secretariat under an Advisory Board is usually established as the governance mechanism to ensure that agreed procedures are being followed by the grantees. The Secretariat, with assistance from grantees, is also responsible for monitoring and evaluating the performance of all CGP sub-projects It is necessary to ensure good quality monitoring and evaluation particularly in the early stages of a sub-project when the CGP Secretariat and the grantees gain experience. Monitoring refers to: routine collection of information, tracking implementation, measuring efficiency, and questioning "whether the project is doing things right." "Evaluation" is generally defined as: analysing information, ex-post assessment of effectiveness and impact, confirming project expectations, measuring impacts, and questioning "whether the project is doing the right things." M&E applies at three distinct levels in World Bank (WB)-financed projects involving a CGP: (i) M&E of sub-projects (this is the shared responsibility of the sub-project grantee and the CGP Secretariat); (ii) M&E of the CGP as a whole (by the Director of the CGP Secretariat and/or the Head of the Project Management Unit); and (iii) M&E for a WB-financed project involving a CGP (by the national counterpart agency implementing the project). The primary focus of this "good practice" paper is on the M&E of CGP sub-projects. Key Components of a Successful M&E and MIS The cardinal rule in establishing a practical, functional M&E system for a CG program is to keep it simple. It is best to start with a few important indicators and milestones and build from there as requirements dictate. There are four key components of an effective monitoring, evaluation and management information system: First, a clear (sub)-project/program design. Such a design needs to have explicit objectives and a time-bound implementation plan with key milestones for all activities or a logframe with specific indicators and targets. This ensures that the parties that have committed to sub-project implementation are fully aware of their obligations to report accurately and in a timely manner to the CGP Secretariat in order to receive continued funding for implementation. Second, milestone-based implementation progress reporting. For CGP sub-projects, requests for tranche disbursement are generally tied to milestone-based progress reports. This reporting usually covers both expenditures (concerning procurement and other inputs) and technical progress. This routine reporting ideally will also include some information on the quality of outputs, the number of beneficiaries affected and preliminary results. The progress reports by sub-project grantees, tied to tranche disbursements where appropriate, should provide adequate information on implementation progress which are usually measured against agreed indicators. . Third, frequently conducted progress reviews and impact studies by the Secretariat. These draw on sub-project implementation progress and completion reports. The impact studies involve field visits and where possible making use of beneficiary assessments to evaluate sub-project progress and achievements in view of CGP objectives. Special reviews to confirm progress or impact studies for a sample of CGP sub-projects should be carried out. However, sub-project impacts should be assessed in all completion reports prepared by sub-project grantees. Fourth, a Management Information System to manage and analyze data on the sub-projects or program. It is essential that the M&E data collected from each sub-project is entered into a data management system so that it can be fully utilized by the Project Management Unit and in particular by its Technical Secretariat. A high quality and efficient management information system will enhance collection, storage, management and analyses of the large volume of data and information generated in the context of implementing a CGP. A well-performing M&E system and MIS will enable the Secretariat to monitor implementation of sub-projects according to the agreed implementation plans and promote growth in areas found to be successful while drawing lessons from sub-project failures. The Secretariat should make regular field visits to sub-projects, particularly in the early stages as a check on the quality and accuracy of grantee reporting and to provide guidance. An important function of an M&E system is to facilitate the resolution of "problem" sub-projects. With a weak M&E system, problem sub-projects may continue without being detected. "Rating" systems that summarize the performance of sub-projects can facilitate this. Special provisions are often required to ensure that sub-projects do not work in isolation because this could cause problems with maintaining program quality, replication, and accessing benefits from economies of scale. Thus, it is necessary for CGP management to ensure close linkages between grantees implementing sub- project and to create opportunities for the regular exchange of experiences. CHAPTER I THE IMPORTANCE AND ROLE OF M&E AND MIS IN CGPS (a) Competitive Grant Programs in the ECA Region Worldwide there is increasing use of competitive funding mechanisms or "Competitive Grant Programs"1 (CGPs) for financing agricultural support services (research and extension), and also for community development programs and environmental projects. This trend is reflected in the World Bank's (WB) Eastern Europe and Central Asia (ECA) region, where the number of countries using CGPs for funding these types of projects has been steadily growing. Under these CGP programs, funds are allocated competitively to private and/or public research and development institutions to implement contractually agreed activities. CGPs are being implemented either as freestanding projects or as components of projects financed by the WB in Romania, Georgia, Macedonia, Turkey, Armenia, Croatia, Albania and Azerbaijan. Such CGPs enable governments to address priority research, extension and other rural development needs with a focus on adoption and impact and simultaneously to "jumpstart" the agricultural services reform process. CGP sub-projects are intended to foster innovation and efficiency by subjecting financing proposals to competitive processes and selection. They encourage institutional pluralism and competition by bringing institutions together (from both the public and the private sector). The flexibility in CGPs can also be useful in initiating the reform of Agricultural Knowledge and Information Systems (AKIS) by funding required studies, pilot projects and other preparatory activities. A successful CGP is dependent on a number of factors including: · Identification of realistic objectives and focused priorities; · A well-trained program management team (typically including a CGP Secretariat (the Secretariat), directly responsible for implementation and administration of the CGP, a Project Management Unit handling overall procurement and accounting and a "Board of Directors" comprised of stakeholders, for oversight); · An independent and transparent peer review process; · A smooth flow of funds and proper accountability; · Efficient and effective fiduciary procedures, including financial management, procurement and auditing; and importantly, · A well-defined and organized monitoring and evaluation (M&E) and Management Information System (MIS). 1The Competitive Grant activities as a whole are referred to as a CG "Program", as this is often a broader, longer- range financing mechanism, even though it might be currently financed entirely as part of a short-term WB-financed project. This manual will generally use the term "competitive grants program" or CGP, although it is recognized that most sub-projects are implemented under contractual arrangements rather than as "grants" and that many programs involve co-financing of sub-projects. (b) Structure of CGP Management Implementation of a CGP relies on a number of key institutions and their associated staff, each with separate roles and responsibilities and needs for information. Success of CGPs often depends on maintaining a clear separation between different functions, such that responsibilities remain clear. This requires however, a sound and easily accessible information base to facilitate interaction and informed discussions between the different staff associates. The names of the CGP-related institutions and frameworks within which the concerned staff work differ from program to program, and country to country, but typically these would include: · A Project Management Unit responsible for the WB-financed project as a whole; · A CGP Board of Directors in charge of policy direction for implementation of the overall CGP; · A CGP Secretariat2 specifically in charge of overseeing CGP implementation; · A Technical Evaluation or "Peer Review" Panel to examine proposals submitted for CGP funding; and, · Independent sub-project grantees. The specific responsibilities of these entities are as indicated below: A PMU handles all activities specific to the implementation of a WB-financed project of which a CGP can be a component; this includes progress reporting, procurement of goods and services, financial management and accounting, M&E (for a CGP through the Secretariat), ensuring timely project financial audit, etc. A Board of Directors (or "Advisory Board", "Coordinating Committee" or sometimes simply the "CGP Funding Agency") is responsible for policy direction and overall supervision of the CGP. The Board provides final approval of funding for sub-project proposals that have been recommended for financing by the CGP Secretariat. The Board also monitors achievement of the overall CGP objectives mainly, but not exclusively, through a review of reports provided by the CGP Secretariat. The Board may at its discretion decide to meet with sub-project implementing entities and/or visit selected sub-project sites. A CGP Secretariat generally has responsibility for over-all management and administration of the CGP, including: day-to-day administration; coordination of activities; publishing information concerning the availability of CGP funds for research and extension grants; review and initial screening of proposals submitted; overseeing sub-project selection and contracting; facilitation of financing; M&E; and management of all information regarding the program. The CGP Secretariat is responsible for reporting on over-all CGP progress and therefore must closely monitor performance of all the individual sub- projects and take corrective action where required. The Secretariat also has first-line responsibility for the impact evaluation of CGP sub-projects and for ensuring the preparation of a final CGP "Implementation Completion" Report. A Technical Peer Review Panel is established for most CGPs. The Panel's main function is the technical evaluation of proposals submitted to the CGP for funding. The Panel's responsibilities are often limited to examining and evaluating proposals pre-screened by the CGP Secretariat and making recommendations for project approvals and allocations of funding. However, in some CGPs, the 2In World Bank-financed programs, the CGP Secretariat is often part of the Project Management Unit. Technical Panels or individual members remain "on-call" to assist with routine or special evaluations of sub-project implementation progress and impact when required. CGP sub-project grantees include a wide range of public and private (for-profit and not-for- profit) organizations, producer and community groups, universities, research agencies, and other organizations that submit proposals and are awarded contracts to implement CGP sub-projects financed by the CGP3. They are responsible for sub-project implementation and regular progress reporting, including the preparation of a "Sub-Project Completion Report." (c) Importance of M&E in CGPs Satisfactory implementation of a CGP and the longer-term viability of the concept itself depend substantially on the quality of implementation and results achieved by the sub-projects. This is primarily determined by the commitment of the sub-project teams to implement the work according to the agreed contracts and also by the supervision and guidance received. The quality of the latter largely depends on having up-to-date information concerning sub-project implementation. It is important that approved sub- projects are closely monitored and evaluated and that the information collected is analyzed, so that any problems encountered (in adequacy and continuity of funding, procurement arrangements, timely sub- project implementation, technology dissemination, audit arrangements, reporting, or other functions), can be quickly diagnosed and resolved. Thus, high quality and timely monitoring and concurrent evaluation are essential to a successful CGP; they are particularly important in the early stages when both the CGP Secretariat and the CGP grantees lack experience. Monitoring and identifying problems and successes often provides good opportunity to "learn by doing." Because the sub-project grantees are expected to provide the basic data for overall M&E, it is crucially important that they understand that the goals of M&E are to support quality implementation and not "to find fault" with the implementers. CGP management is also responsible for conducting impact assessments at the end of sub- projects. There are two reasons to carry out impact assessments. First, the information provides important feedback to CGP management on the overall performance of its program and sub-project portfolio that can be used to adjust program strategies and priorities. Second, evidence of impacts is important in future dialogues with financing agencies in order to consolidate, institutionalize and possibly expand the CGP, as a permanent component of the national AKIS. Approaches to impact assessment are further discussed in chapter VIII. (d) Levels of M&E in CGPs M&E applies at three distinct levels in WB-financed projects involving a CGP: · M&E of sub-projects is the shared responsibility first of the sub-project grantee and secondly the CGP Secretariat; · M&E of the CGP as a whole is the responsibility of the Director of the CGP Secretariat and/or the Head of the PMU, and the Board of Directors or similar organization with oversight responsibility for the program; and, · M&E for a WB-financed project involving a CGP is the responsibility of the national counterpart agency charged with implementing the project (often this will be the concerned Ministry/PMU), with supervision by WB staff. 3CGP-financed activities are generally called "projects", but in this manual they are referred to as "sub-projects" to differentiate them from WB-financed "Projects". The primary focus of this "good practice" paper will be on the M&E of CGP sub-projects which are the responsibility of the CGP Secretariat. However, M&E of sub-projects provides substantial input to the M&E for the CGP Program as a whole and the paper will to some extent also discuss the methodologies, approaches, and the framework for M&E of a CGP. The paper will, however, not go into detail on M&E for WB-financed projects as such. (e) The Critical Link Between M&E and Timely Flow of Funds In CGP operating procedures it is common practice to establish a close linkage between M&E of sub-project implementation and approvals for funding. Sub-project proposals approved for funding by the CGP Board normally receive a first funding installment or "tranche" at the time of contract signature (or sometimes, at an agreed later date). Subsequent fund releases are generally linked to sub-project reporting with signed evidence of acceptable progress in implementation and achievement of specified "milestones." The final tranche or payment for sub-project expenditures is frequently linked to the submission of the sub-project "completion report." The effectiveness of the CGP Secretariat in accurately assessing technical quality, the achievement of agreed "milestones," and procurement/financial progress of sub-projects at the time of each new request for disbursement of a funding "tranche," are important to the quality and timeliness of CGP sub-project implementation. (f) The Importance of Information Management and Utilization in CGPs The administration of a CGP produces a very large amount of data and information in terms of outputs from the reporting and M&E systems. Thus, a high quality and efficient "Management Information System" (MIS) is needed to collect, manage and analyze such a large volume of data and information. Aspects concerning the role and the establishment of an MIS in a CGP are discussed in chapter VII. CHAPTER II THE BASICS OF MONITORING AND EVALUATION OF CGPS (a) Definitions Monitoring and evaluation are important exercises to ensure success of programs. The WB M&E system for monitoring its portfolio of development projects are somewhat analogous to those needed for M&E of a competitive grant program that is funding diverse sub-projects. An M&E system for WB projects is necessary and critical as the Bank finances a large number of diverse projects in many dispersed locations. The WB M&E system must track the performance and impact of all individual projects and provide objective evidence of the quality of implementation and impact of the portfolio as a whole (see Box 1). M&E systems provide information on the progress of projects, evidence on effective use of goods and services financed, and signal early indications of outcomes. Box 1: The M&E System in WB-financed Projects The World Bank requires a high quality M&E system for its portfolio of approximately 1800 projects in various sectors of over a hundred countries. The WB M&E system is dependent on sound project designs that establish clear objectives, defined targets and "objectively verifiable indicators" (OVIs) set out in "logical frameworks"4. The WB M&E system provides a basis and framework for: · Routine financial and technical progress reporting by project implementation units and grantees; · Regular supervision visits by Task Team Leaders (TTLs) and others to jointly review progress with grantees and report on project key performance indicators (KPIs); · Regular ratings of projects as to implementation progress and achievement of development objectives and assignment of "flags" for those where problems are encountered; · A more substantive "mid-term review" of project progress to confirm project design and address any needs for redesign; · A project implementation completion report to document results, assess sustainability and development and institutional impact, and evaluate implementation performance; and, · Implementation "audits" and impact studies carried out on an occasional basis for projects several years after their completion to evaluate final impacts and lessons learned from the project experience. PMUs and project implementing agencies such as Government Ministries are primarily responsible for much of the M&E work in WB-financed projects, with WB TTLs participating in this work during supervision visits and providing regular assessments of performance and potential to meet the stated project objectives. The WB's Operations Evaluation Department carries overall responsibility for M&E in the WB. On an institutional level, the WB also monitors global changes in economic growth, poverty levels, food security and environmental conditions. There is frequently confusion with definitions, but in WB-financed projects, "monitoring" is generally seen as distinctly different from "evaluation." Monitoring refers to: routine collection of information, tracking implementation progress, measuring efficiency, and questioning "whether the project is doing things right." "Evaluation" is generally defined in terms of: analyzing information, ex- post factual assessment of effectiveness and impact, confirming project expectations, measuring impacts, 4Note: A logical framework or "logframe" of a (sub-)project presents a cause-and-effect link from the project's inputs and activities to outputs to outcomes to goals. and questioning "whether the project is doing the right things."5 Perhaps the most useful distinction is that: · Monitoring in essence is a continuous collection of data on a project to track implementation progress and collect information as a basis for evaluation of performance and decisions concerning the implementation of the project (making adjustments, termination, etc.); and, · Evaluation is also a continuous process of analysis of a project's performance and progress towards its objectives. It starts during project implementation and continues after project completion with ex-post impact evaluation, which feeds into decisions on future projects. (b) The Functions of an M&E System An M&E system provides information that is used in various ways in managing CGP sub- projects and the CGP as a whole, including follow-up after completion. Most current systems for M&E distinguish between: "implementation monitoring," "concurrent evaluation," and once a sub-project, project or program is completed, "ex-post evaluation." (i)Implementation monitoring at sub-project level is the responsibility of the CGP sub-project grantees and CGP Secretariat. It provides regular feedback on program/project implementation to identify potential problems early-on and allow for corrective action. It also provides a basis for approving releases of funding linked to the achievement of "milestones." (ii)Concurrent Evaluation of sub-projects is carried out by the grantees and the CGP Secretariat. The Secretariat may undertake a more extensive review and evaluation of the sub-project, using its own staff or extension staff, farmers associations, NGOs, or qualified individuals. The CGP Secretariat, sometimesbut not alwaysthrough contracting an outside independent agency (research institute, consultant, local university, an IARC, etc.), periodically undertakes special sub-project progress reviews and "quick response studies" of a sample of sub-projects with special focus on those that have encountered problems in implementation (and/or where unusual impact seems to have been achieved). These reviews provide for an objective, examination of CGP-funded activities and often result in recommendations to address problems and/or, to capitalize on new opportunities. (iii) Evaluation of completed CGP sub-projects. It is important to document the final accomplishments and experience gained under each CGP-financed sub-project. Grantees are typically required to prepare a "completion report" at the end of each sub-project, covering the entire implementation period and giving particular attention to documenting the ultimate impact and achievements of the sub-project, including lessons learned. The CGP Secretariat usually also undertakes an ex-post evaluation of all (or a sample of) completed CGP-financed sub-projects. Information from completion reports should help in the design of future sub-projects and CGP operations as a whole, in order to achieve greater impact on overall goals such as increased productivity, poverty reduction, environmental protection, and institutional reform. 5Together M&E can be described as "a continuous process of collection, analysis and use of data." (Alex and Byerlee, 2000) (c) The Responsibilities for M&E in a CGP Standard practice places responsibility for M&E in CGPs under the CGP Secretariat and the CGP Board. The CGP Secretariat is usually responsible for M&E of individual sub-projects financed by the CGP and for providing information to the CGP Board for it to use in evaluating the performance of the CGP as a whole. However, in reality, all Secretariat and PMU staff and stakeholders involved with program and sub-project planning and implementation share responsibility for M&E. Up to an agreed ceiling, specified in the sub-project contract, procurement of materials and supplies, services of consultants, transport, etc. are the sole responsibility of the sub-project grantees. Involvement by the sub- project grantees also helps in educating them to the main objectives of the program and instills a focus on "managing for results." At the same time however, sub-project grantees need to be fully aware that the CGP Secretariat is responsible to monitor implementation (both financial and technical), of each CGP- financed sub-project. (d) Financial Management of M&E in CGPs Usually, the management of the financial resources allocated in support of CGP-funded sub- projects is delegated to the sub-project level. The borrower, the CGP Secretariat and ultimately the recipient of the grant are responsible to ensure that the funds disbursed are used only for the purposes for which the grant was provided. The Secretariat will therefore require all grant recipients to: (i) maintain a financial management and reporting system, (including records and accounts in a format acceptable to the Secretariat), that accurately reflects the operations, resources and expenditures of the sub-project; and (ii) submit sub-project financial statements for each calendar year or more often as may be required. Financial books, records, grant project financial statements and any relevant substantiating documents (such as original bills, invoices, receipts, etc.), need to be retained by the grant recipient for a period of at least two years after the WB close of the overall project. Because the CGP-funded sub- projects are located at widely dispersed locations and implemented by a broad range of agencies it is essential that adequate training is provided to the grant recipient's accounting staff before sub-project implementation starts. Reporting formats and frequencies should be agreed in advance of contract signature and be included in the contract. (The existing manual in the series on "Good Practices in Procurement, Disbursement, and Financial Management for Competitive Grant Programs in ECA Countries" provides guidelines in this regard. (e) Procurement and M&E in CGPs Conduct of procurement in a timely, economic and efficient manner is necessary to the achievement of the objectives of WB-funded projects, including CGPs and the sub-projects they finance. In a WB-funded project with a CGP component, procurement M&E takes place at three levels: (i) WB project, (ii) CGP; and (iii) sub-project. At the WB-financed "project" level, procurement monitoring is the responsibility of the Secretariat (PMU or similar entity) with the participation of the WB TTL and supervision missions. If necessary, the PMU may hire external assistance (such as an individual specialist or a consulting firm with experience and specialization in M&E), to advise on procurement monitoring for the whole project. At the CGP level, the major procurement consists of the award of contracts for sub-project implementation. Standard reporting systems for this process provide the basis for procurement monitoring. If a CGP requires substantial additional procurement (i.e., for promotional activities, training, monitoring and impact studies, etc.) additional reporting systems need to be set up to track such procurement. At the CG sub-project level ensuring quality procurement monitoring is difficult because of the high number of sub-projects scattered over a large project area, sometimes involving hundreds of grantees. The Secretariat must almost always organize procurement training for sub-project grantees, with emphasis on procurement methods and procedures for preparation and submission of procurement reports. Simple information collection formats, including information relating to procurement procedures should be designed in advance so that CGP beneficiaries can easily complete the specified forms and submit the required information to the Secretariat. Submission of these reports is normally linked to the requests for tranche payments, and the final disbursement of funds to the sub-project completion report. (f) Special Challenges in M & E for CGPs Successful M&E systems for CGP sub-projects must address a number of challenges. Accurately monitoring what is happening in a very large number of diverse small projects and, importantly, taking appropriate actions in a timely fashion, is inherently difficult. CGPs increasingly target diverse objectives including: increased incomes, poverty reduction, conserving environmental resources, promoting food security, increasing exports, enhancing economic growth, reducing social conflict, and others; methods of monitoring these impacts differ widely. Many variables outside the control of a particular CGP-funded sub-project can influence outcome indicators and make it difficult to assess the true impact of a sub-project. For example, weather conditions, overall trends in national economic growth, governmental policies, changes in institutions, natural disasters and other uncontrollable factors can significantly affect the indicators being monitored for CGP-financed sub-projects. An effective M&E system requires indicators that are as independent of these other factors as possible, but, since external influences are unavoidable, in-built safeguards in the form of appropriate monitoring and reporting on these external factors are essential to make sure they are taken into account to avoid unfair negative evaluations of sub-projects. CHAPTER III A FRAMEWORK FOR M&E SYSTEMS IN CGPS (a) CGP/Sub-Project Design and M&E M&E is a process of comparing what is happening with what was expected and with what is desired. If the CGP program or its sub-projects lack clear objectives or explicit time-bound implementation plans, it is difficult to assess performance or decide objectively on the release of additional funding. M&E in a CGP depends on clarity in the program and sub-project objectives and a clearly formulated, time-bound and implementation plan. Thus, the CGP and the sub-project objectives and implementation plans provide the starting point for M&E systems. An example of a CGP sub-project proposal format currently in use in ECA is illustrated in Annex 1. The sub-project proposal format should allow potential grantees to state clearly and in quantitative and qualitative, time-bound terms, the goals, objectives, intended outputs, expected outcomes, required inputs and financing, and activities to be performed to satisfactorily implement the sub-project. These can be reflected in a "a time-bound implementation plan" that summarizes how the sub-project is to be implemented. The relevant data (specified in Annex 1) should be transferred into monitoring templates (illustrated in Annex 2) and entered into the MIS system to facilitate the M&E process. (b) Sub-project Implementation Plans As indicated earlier, a detailed and time-bound work program or implementation plan is a key part of a CGP sub-project design. An implementation plan will include all the important activities under the sub-project, the corresponding inputs and outputs/deliverables and the actual time at which these are to be achieved. This might be summarized in a table or flow chart that visually illustrates the steps in implementation and the time-bound phasing of activities. Outputs should be selected in such a way that it is feasible to objectively monitor their attainment. Although there will usually be a generally prescribed outline for implementation plans in the CGP instruction materials enclosed with the sub-project proposal forms (see Annex 1), due to the wide diversity in sub-projects, it will be important that the final implementation plan is specifically tailored to each sub-project and that it becomes part of the agreed contract between the grantee and the CGP Secretariat/Board. Because it may often be necessary to adjust the implementation plan due to external circumstances (e.g., adverse weather, illness of contractors, or even tranche payment delays), particular milestones and the agreed tranche payment schedule may sometimes need to be re-negotiated. A CGP Implementation Manual needs to include guidelines on how changes in the sub-project implementation plan and in the schedule of payments will be requested and approved. (c) Monitoring Tools "Key Performance Indicators" and "Milestones" Two types of monitoring tools derived from the sub-project design normally are the basis for M&E: these are: (i) Key performance indicators: M&E systems rely fundamentally on the use of monitorable indicators set out in the sub-project design to track implementation progress and established Key Performance Indicators (KPIs) describe the expected inputs, outputs, outcomes and impacts of a sub- project. The number of indicators should be kept as low as possible to avoid overburdening reporting systems. Indicators must be identified in unambiguous terms; they should be objectively verifiable. Indicators generally should be defined in terms of quantity, quality and time (QQT). For a 3-year hypothetical CGP sub-project in agricultural research and extension that aims at enhancing smallholder farm incomes through the development, testing and facilitation of widespread adoption of new vegetable varieties, possible KPIs could be as illustrated in Table 1. Table 1: Key Performance Indicators for a CGP-financed Sub-project on Improved Vegetable Varieties Input · 3 months of technical assistance used to evaluate potential vegetable crops and varieties within six months of sub-project start-up. Input · 20 metric ton capacity seed processing equipment procurement completed within nine months of start-up. Output · 2 vegetable varieties suitable to local agro-ecology identified within 1 year from start- up. Output · multiplication of 2000 kg of seed of improved varieties within 2 years from start-up. Output · 50 extension staff and 20 farmer groups, each of at least 10 members, trained in vegetable production technology within 3 years from start-up. Outcome · at least 10,000 farmers adopt new varieties within 5 years from start-up. Impact · average income from vegetables in the region served increased by 10 percent within 6 years from start-up. (ii) Implementation "milestones": For CGP sub-projects, specific and substantial progress achievements or "milestones" are often used as the basis for release of payment tranches. These milestones represent success in achieving for example the acquisition of specific inputs, delivery of particular outputs, or timely completion of a specified activity identified in the sub-project implementation schedule. The milestones are important to technical monitoring of sub-project progress and as justification for disbursement of subsequent tranche payments. Table 2 lays out a simple set of milestones for the hypothetical CGP-funded on-farm maize improvement sub-project. The number of tranches under the project is clearly defined and the date that each tranche is eligible for payment is indicated. The table provides a specific description of the milestones that need to be achieved and certified by the Secretariat before a tranche payment can be made effective. The final column of the table reflects the funds budgeted for each tranche payment. Thus, the flow of funds depends on effectively monitoring what is happening in the sub-project, often at the field level. Table 2: Time-bound Schedule of Tranche Payments for a Hypothetical Maize Improvement CGP Sub-project Tranches Date Milestones achieved; conditions for tranche release Amount (US$) Tranche 1 Oct. 15, 2002 Signature of contract 14,500 Tranche 2 July15, 2003 Achievement of agreed outputs: (a) 24 on-farm trials set 10,000 up; (b) 3 farmer field days organized; and (c) the project progress/M&E report timely submitted. Tranche 3 Dec. 15, 2003 Achievement of agreed "end of project" outputs: (a) 24 5,500 on-farm trials completed and analyzed; (b) 2 additional farmer field days organized; (c) end-of-project stakeholder workshop held; and (d) project completion report submitted. TOTAL $30,000 CGP M&E systems track inputs and outputs (or milestones) to see: (d) Tranche Release Requests At the time of contract negotiations, the time-bound milestones, achievement of which "trigger" the release of additional funding tranches, should be agreed, and the completion and/or initiation of certain sub-project activities. For example, in case of the maize improvement trial of Table 2, the initial fund release at contract signature of e.g., US$14,500 which should be sufficient for all expected sub- project expenditures required to be paid until the first milestone is achieved. Completion of the third activity then becomes the trigger to release the next financing tranche of US$10,000. A final budget release of say US$5,500 could be linked to the submission of the final sub-project (completion) report. A brief, factual format exclusively referring to milestones that can be quickly reviewed and examined is required. A sample framework for tranche release approval in the context of a hypothetical CGP sub-project on the expansion of quality apple production, is given in Table 3A; this provides the essential information on the satisfactory completion of activities funded under the first tranche and is necessary for requesting funds for the second tranche. An example of an explanatory form for shortfalls in achieving agreed KPIs is given in Table 3B. A new tranche release request needs to be completed each time a (set of) agreed milestone(s) or KPI(s) has been achieved. The number of tranches can vary depending on the duration of the sub-project but should for practical purposes probably not exceed 5, even for a three-year sub-project. The agreed format for tranche releases request must be attached to each final CGP sub-project contract. Because of the wide range of types of sub-projects funded under CGPs, it is necessary that the grantees themselves draft this format and that it is specifically approved at the time of contract negotiations. The final contract will therefore provide the basic format according to which the progress of the sub-project can be measured (see Annex 2 ) Table 3A: Example of a Completed Tranche Release Request Form for a Sub-project on the Expansion Of Quality Apple Production. Activities Start Date End Date Quantifiable Indicators Financial Procurement **Indicators eh Monitoring Monitoring Verified by Secretariat Planned Actual Planned Actual Key Was Main Main Items Office Field Tranc Performance indicator of Items procured of de Indicator achieved. tneps expected (Yes, No, to be No. undsF nding Partially) allocat undsF procured * Category spe 1 Testing 10 apple varieties on 1.01.02 1.01.02 30.6.02 30.6.02 3 testing fields x Yes 4900 4795 Goods Salary of 3 Salary of 3 x resistance to viral diseases 2500 sqm=7500 1000 specialists specialists for 3 sqm; - Works for 3 months; field grafting of 10 3795 months; field work varieties of virus work free apple trees; 2 The elimination of the varieties 1.09.02 1.07.02 1.10.02 1.09.02 2 varieties x 1000 Yes 3650 3680 Goods Salary of 3 Salary of 3 X identified as infected with latent of diseased trees 1000 specialists specialists for 3 viruses removed; Works for 3 months; months; field 2680 field work work 3 Obtaining tree grafts free of viral1.01.02 1.01.02 1.06.02 1.06.02 Obtaining 30,000 Yes 25100 25250 Goods rented 1 - rented 1 X diseases grafts of virus free 20000 tractor, 3 tractor, 3 trees. Planting o f Works atomizors + atomizors + 30,000 grafts; 8 3000 spare parts, spare parts, maintenance crews grafting grafting in the field Trip 2250 material material 4 Readying the trees and planting 1.03.02 1.03.02 1.10.02 1.10.02 Harvest the virus Partial 52350 45000 Goods 1 computer, 1 computer, 1 x free grafts of (12100) 40000 1 printer, 30 printer, 30 tool 15,000 trees; Works tool kits, kits, irrigation Planting 15,000 5000 irrigation equipment, 1 disease free trees in equipment, 1 tiling machine, the field. planting 1 planting machine, 1 machine, tilling rented 1 machine, tractor. rented 1 tractor. Status of project: Highly Satisfactory, Satisfactory, Marginally Satisfactory, Unsatisfactory, Alert. If the indicators are partially completed or not completed at all, provide explanations on the attached form 14 Table 3B : Explanation of Shortfalls in Sub-project KPIs Monitorable Reason for partial or non-completion of Action Taken/To be Taken to Rectify the Situation Indicator No indicators Only 12000 Apple seedlings were planted due to: Soils were ploughed and prepared and new seedlings were purchased and are maintained at the nursery. Lack of quality grafts for a 2ha area. When the weather conditions improve they will be Adverse weather conditions (heavy rains) were transferred to the 2ha site. also a problem Explanation of Expenditure Allocated versus Spent No Problems Difference in Proposed Procurement versus Actual Procurement No problems On GENERAL COMMENTS ON PROBLEMS GENERAL COMMENTS ON SUCCESSES completion of specific tranche Heavy rains delayed the project The project will eventually meet all criteria and goals. The lack of quality seedlings in the region A stronger relationship between researchers and farmers is developing. The lines of communication between project and PMU are strong This was prepared by:Sub-project leader_______Date: 4/22/03_______ Information certified by:CGP Secretariat_________Date:_4/28/03_ Decision: No serious implementation problems; tranche release should proceed. 15 CHAPTER IV M&E OF IMPLEMENTATION FOR CGP SUB-PROJECTS (a) Routine M&E and Sub-project Progress Reporting by Grantees Whenever possible, reporting formats should be standardized across sub-projects to simplify data management for the CGP Secretariat. A reporting / monitoring format should be established before any grant agreement is signed (Annex 1). A sub-project's performance should be tracked by the time-bound implementation of the expected inputs, activities, and outputs which are entered into monitoring templates prior to the grant agreement. The monitoring template should then be completed at the conclusion of each sub-project tranche. Thus, providing an accurate account of expected inputs, activities and outputs versus actually inputs activities and outputs. The completed template notifies the CGP Secretariat of anything short of full activity completion and provides the grantee an opportunity for narrative explanation as to the successes and failures in reaching each milestone. Also, the templates offer the grantee an opportunity to discuss actions taken or to be taken to resolve any problems related to a particular milestone. (The monitoring templates can be seen in Annex 2) (b) Sub-Project Completion Reporting At sub-project completion, there is a requirement for a report on the entire sub-project experience, including especially the final outputs and where appropriate, impact on productivity, farm incomes, environmental effects, etc. A draft format for a Sub-project Completion Report is attached as Annex 3. (c) Technical, Financial and Procurement Monitoring Monitoring of sub-project progress assembles data relating to the sub-project milestones (technical, financial and procurement), and determines the degree to which it is being effectively implemented. The sub-project design and implementation plan provides the basis for such monitoring (Annex 1 Table 2). Technical monitoring is needed to confirm sub-project efficiency in implementation, production of outputs, and achievement of milestones; effectiveness in the quality of implementation and outputs; and the relevance and sustainability of outcomes. Technical monitoring must consider both expected and unexpected outcomes of the sub-project. Financial monitoring focuses on financial management and accounting. Financial monitoring must verify that accounts and records are timely and accurate and that funds are managed as required by the sub-project agreement. Review of the use of funds must confirm that funds are used only for approved purposes for implementation of the sub-project. Monitoring identifies any bottlenecks or delays in release and use of funds and identifies solutions to any problems encountered. Such monitoring provides an opportunity for budget review to verify whether the established budget is adequate or whether changes will be needed. Procurement monitoring tracks the purchase and delivery of inputs, especially those inputs critical to the success of the sub-project. Procurement monitoring verifies execution of time-bound equipment procurement, implementation of consultancy contracts, completion of training courses and study tours, and delivery of other goods or services necessary for implementation of the sub-project. Procurement monitoring verifies the quality of inputs to ensure that these are adequate, but not excessive, to sub-project requirements. Procurement monitoring also relies on and confirms the accuracy of inventory records and systems. 16 (d) Data Quality, Evaluation and Analysis Data quality: The quality of the data/information collected in technical, procurement and financial monitoring must exhibit: Reliability, as to the extent to which the data collection approach is stable and consistent across projects and time; Validity, as to the extent that the data clearly and directly measures the performance indicators or aspects intended to be measured; Timeliness, as to how soon the data is available for review and inspection; and, Cost, as to the financial and management time required to obtain the data. Data evaluation and analysis: The evaluation and analysis of reported information/data should: Provide information on the main achievements of CGP-funded sub-projects both in quantitative and qualitative terms, and in relation to the agreed time schedule; Give information on the status of sub-projects or sectors of the sub-project portfolio; Specify any problems occurring within a sub-project or a group of sub-projects; Submit suggestions that might promote any improvements in the sub-projects, programs, or relevant policies; and, Provide information over time on either positive or negative trends in a sub-project, a group of sub- projects or program. The data is retrieved from the sub-project tranche release request forms (Table 3A and 3B). But sub-project reporting by the staff responsible for implementation is often subject to limitations. There can easily develop a tendency to report only "good news." This can be seen in some M&E systems where all indicators end up being reported as "100 percent complete." Such reporting is not useful to the grantee nor to the CGP Secretariat. Through training, a collegial relationship that encourages truthful reporting needs to be developed. This will only help to a certain degree; to complement and improve self- monitoring and reporting, regular supervision by the Secretariat is essential in order to maintain reliable reporting by grantees. (e) Supervision and over-sight by the CGP Secretariat In CGP sub-projects, a significant number of indicators are monitored and evaluated by the Secretariat: financial, managerial, physical and technical. Monitoring reports are reviewed by the Secretariat, and if found satisfactory concerning the achievement of the agreed milestones, submitted to the financial officer for payment of the next tranche. Analysis of the information also should provide opportunity to promote growth in areas found to be strong and successful while simultaneously drawing lessons from possible early project failures. The Secretariat should also make regular field visits to the CGP-funded sub-projects. This is particularly important in the early stages as a check on the quality and accuracy of sub-project level reporting and to provide guidance in implementation. The Secretariat may also decide that for specific sub-projects there is a need for additional reviews of progress. In that case, sub-project progress review and preparation of the report can either be by someone from within the Secretariat or by someone appointed by the Secretariat. If agreed milestones are only partially met (see Table 3B), the Secretariat must make a judgment decision on further disbursements. Normally, in case sub-project implementation is at least "marginally satisfactory," routine disbursement of the next tranche would be expected. In case a sub-project remains in the "unsatisfactory" category for more than one reporting (tranche release) period, notwithstanding attempts to bring it back on track, it normally should be closed. However, depending on the seriousness of the shortfall, the Secretariat may decide on several alternative options such as: (i) continued tranche release without special actions in case the shortfall is considered minor and expected to be corrected soon; (ii) an immediate independent review of progress while releasing (part of) the next release; (iii) 17 temporarily delaying further disbursement until a review has been completed and satisfactory action been taken; and, (iv) in cases of confirmed breach of contractual obligations or mis-procurement, immediate cancellation of the sub-project possibly with actions for fund recovery. (f) Sub-project Visits Sub-project visits are of importance in monitoring CGPs as these enable the CGP Secretariat to confirm the accuracy of sub-project reporting and to identify difficulties. The frequency, duration, and procedures for sub-project visits will depend on specific sub-project circumstances. Generally, a subproject visit within the first six months after sub-project approval is desirable. Sub-project visits can be important to developing and strengthen capability of grantees, as well as to ensuring smooth implementation of sub-projects. In Nepal the "HARP" sub-project (Annex 4) used the term "monitoring and support visit" to emphasize the Secretariat's contribution to developing capacity for sub-project implementation. In Ecuador's CGP funded "Program for Modernizing Agricultural and Livestock Services" (PROMSA) routine "project monitoring, evaluation, and support visits" are carried out approximately twice a year, enabled the Secretariat to address technical and administrative problems and accurately rate a number of sub-project implementation criteria ( Box 2, next page). The composition of the sub-project visit teams will vary depending on the needs and circumstances. Often project officers from the CGP Secretariat can independently carry out the monitoring visits. This has the advantage that the project officers are thoroughly familiar with the CGP program. In other cases, contracted specialists (from local institutions, universities, etc.) will be used while they may not have full knowledge of the CGP program, they provide flexibility, a fresh and objective view of sub-project activities, and specialized technical knowledge of the subject matter. How ever, there should be clear reporting procedures for these visits. Whoever makes the sub-project visit, it is important to have clear procedures established in sub- project visit terms of reference. Sub-project visits will generally be closely coordinated with the sub- project grantee and involve full participation of key implementing staff and sub-project stakeholders. Sub-project visits should document observations and recommendations and include these in the sub- project file and/or database. However, such reports should be brief and recorded in a standard form; lengthy reports from these visits are seldom useful or appropriate. Box 2: Ecuador- Program for Modernizing Agricultural and Livestock Services Sub-project Visits A Competitive Fund under Ecuador's "Program for Modernizing Agricultural and Livestock Services" (PROMSA) finances 114 research projects and 14 "strategic alliance" projects to strengthen institutional linkages for research and education. Costs of projects of a duration of 1-3 years, generally range from $40,000-75,000 for research projects and from $200,000-500,000 for strategic alliances. Project visits are a key element to smooth functioning of the PROMSA CGP and to developing capacity within the research system. Project Monitoring, Evaluation and Support Visits: The Competitive Fund Implementation Unit (or Secretariat) has Project Officers (sometimes posted at a regional office) who are senior researchers and who make an average of two visits a year to each project. These project monitoring, evaluation and support visits provide technical and administrative support to projects and provide an opportunity to monitor and evaluate implementation performance. Project visits are generally for one day or less by one Project Officer, but may involve other specialists if there are special problems or issues to be addressed. These routine visits serve "formative" or "implementation" evaluations in which Project Officers (and other specialists, if participating), rate projects on a number of criteria of implementation efficiency and quality on a scale of 1 (unacceptable), 2 (marginal), 3 (satisfactory), and 4 (very good). The average rating becomes the final grade for the project. 18 Final Project Evaluation Visit: The last visit to the project by the Secretariat comes at or near the time of project completion. This final visit generally comprises a one-day visit by a Project Officer with participation of a socio-economist and/or technical specialist depending on the size and complexity of the project. When needed, the visit will include consultations with the project target group and the "Reference Group" and visits to field sites. During this visit, the "Final Visit Team" rates the project on sets of 6-8 criteria for "implementation efficiency and quality" (reflecting efficiency in delivery of outputs), "potential for impacts," "capacity development," and "preliminary adoption of results and linkages with the target group" (reflecting potential for future impacts from the project). An additional set of indicators is rated reflecting "quality of financial management". Ratings indicate: 1 (poor), 2 (acceptable), 3 (good), and 4 (very good). The ratings for the different sets of criteria are consolidated for the project to give an over- all rating of poor (1.0-1.8), acceptable (1.8-2.5), good (2.5-3.2), and very good (3.2-4.0). (g) Problem Sub-projects An important function of a CGP M&E system is the identification of "problem sub-projects" and the resolution of problems. Most sub-projects will encounter some problems (often due to external factors such as the weather or unexpected economic changes). When such serious problems occur the M&E system should assign a "warning flag" to the sub-project until the problem is corrected. Depending on the seriousness, sub-projects with a warning flag might not be eligible for subsequent financial disbursements or might receive additional visits or other technical assistance. However, there needs to be a defined system for resolving sub-project implementation problems and eliminating the "problem sub- project" status. If sub-project grantees do not address problems on a timely basis despite repeated requests to do so, the sub-projects should be terminated. Ecuador's PROMSA project (Box 3) has a well-developed system of warning flags ("alerts") and criteria for when such flags are assigned. Box 3: Ecuador PROMSA--Project "Alert Status" Ecuador's PROMSA Competitive Fund employs a system of "Alert" signals or "flags" that indicate when a project has become a "problem project." There are four ways a project can be flagged as being in alert status: · Quarterly or annual reports overdue by more than a month; · Less than 80 percent of the milestones achieved for the quarter; · A project visit rating of less than 2.5 out of four (see Box 2); and/or · A serious problem in the quality of implementation. While in alert status, a project is not eligible for additional payments from the CGP. If the project does not clear its alert status by correcting deficiencies or rescheduling milestones within a specific time frame, the project is cancelled. (h) Evaluation/Grading of CGP Sub-projects The outcomes of CGP Secretariat oversight of sub-projects need to be formally recorded. Such ratings may be based on an evaluation of progress and performance by the concerned grantees, sometimes verified by CGP Secretariat staff or especially appointed technical reviewers. The ratings may also be "participatory" evaluations involving the clients served by the sub-project in assessing the quality of program implementation. In such summary reviews, sub-projects are often graded into one of four categories that meet (or fail) certain specific expectations such as: 19 Highly · Very good technical quality and/or large actual and/or potential benefits. Satisfactory Satisfactory · Achieving most objectives with likelihood of significant impacts. Marginally · Is somewhat behind schedule or weak in technical quality, but with additional Satisfactory effort should be able to achieve most objectives. Needs special attention. Unsatisfactory: · Is performing much below expectations or has serious problems and requires mitigative measures if it is to produce useful results. May need to be closed Ratings should be assigned on a routine, predictable basis and with clear criteria for ratings at different levels. There may be opportunities to rate a number of criteria of sub-project implementation and then combine these in an overall rating for the sub-project. The rating should be shared with the grantee along with the justification for the rating. This provides an opportunity for frank discussions of any shortcoming and successes of the sub-project with the concerned implementing staff. If the outcome of a sub-project is considered highly satisfactory, significant efforts should made to provide publicity of the result of the sub-project. Dissemination and replication of the sub-project over a wider area may be encouraged. Also in the portfolio evaluation process, problems and challenges encountered in implementing certain categories of sub-projects should be clearly identified and the lessons for other ongoing and future projects drawn. (i) Financial and Technical Audits As in the case of Bank-financed projects, the PMU, the CGP Secretariat, the grantee responsible for the CGP, or the WB, may decide to organize financial (and technical) sub-project audits (chapter II d). All financial and procurement documentation concerning sub-project implementation therefore needs to be maintained for a specific period in order to be available for audit. It should be made clear to all project grantees that an unsatisfactory or incomplete audit frequently results in cancellation of the concerned sub- project and possible disqualification of the grantee to compete for future sub-projects financed through the CGP. 20 CHAPTER V M&E AT THE CGP PROGRAM LEVEL Responsibility for M&E at CGP level is foremost a responsibility of the CGP Secretariat, under the direct supervision of the CGP Board. The monitoring of the quality of implementation and performance of the CGP as a whole is determined by the aggregated sub-project performance; it is therefore important that the status of all sub-projects can be easily aggregated and presented in summary fashion. (a) Monitoring Indicators With regard to CGP performance, a CGP Implementation Plan should identify program objectives, key "milestones" and indicators for the program. At CGP level, monitoring indicators of outputs may be as follows: · Number of sub-project agreements signed in a given period; · Total sub-project amount disbursed/contracted in a given period(committed amount); · The amount disbursed against sub-projects; · Number of sub-project implementation training sessions held for project staff and sub-project beneficiaries; · Number of sub-project supervisions undertaken by the Secretariat; · Number of procurement disputes resolved; · Number of successfully implemented sub-projects, i.e., the sub-projects which achieved their stated objectives; · Etc. Examples of M&E approaches in CGPs are given in Box 2 and in Annex 4. (b) Sub-project Ratings In preparing summary reports on the status of a CGP portfolio of sub-projects, it is often useful to be able to characterize the portfolio on the basis of average ratings for sub-project performance or by numbers of sub-projects in a particular category; grading often goes from unsatisfactory to highly satisfactory (see chapter IV h). A CGP Board may instruct the CGP Secretariat to prepare such a summary statement for example on a half-yearly basis or for an upcoming WB supervision mission. Such a summary report on sub-project ratings might include assessments on a variety of sub-project characteristics, such as quality of technical support, client participation, involvement of women or minorities, cost-effectiveness, etc. (c) Special Studies and Reviews There can be many reasons for the Secretariat or even for the CGP Board to organize special studies concerning a specific group of sub-projects. The results of special studies are usually shared with the Board and communicated to the WB, and may result in particular measures being taken such as amending the contract procedures, applying lessons learned (e.g. speeding up disbursements) to other ongoing sub-projects and/or future projects, taking special measures for ensuring effective technology dissemination and adoption, etc. 21 (d) Sharing Experience Across Sub-projects In the case of decentralized programs and competitive grant programs, special provisions are often required to ensure that sub-projects do not end up working in isolation. There is, however, sometimes little incentive for sub-projects to share either successes or failures with one another, as they are competing against each other for funding. Thus, it is necessary to ensure close linkages between sub- project grant winners. Two different strategies that have been employed by CGPs to encourage "shared learning" are summarized in Box 4. Box 4: Sharing Experience Across Sub-projects Consultative Fora: When a number of sub-projects are working on similar activities, regular meetings to share experiences and plans, and to design collaboration in various ways can be helpful. In Venezuela, the coordinating unit for the CGP organized meetings and technical support for all sub- projects working on the same commodities (i.e., coffee, vegetables, rice, dairy, etc.). In Bangladesh, a rice research program organized an "Uptake Forum" for all sub-projects working to improve dissemination of improved rice technologies. In Colombia, research and training sub-projects working on similar issues meet regularly. Grouping sub-projects by activity or objective is found useful to monitor and evaluate performance and impact within and across different types of sub-projects. Peer Reviews: In an outgrowth of the Uptake Forum in the Bangladesh rice research program, grant winners collectively carried out joint assessments of individual sub-projects, providing informed and objective evaluations of activities and maximizing the shared learning between sub-projects. As a result, good practices from the various sub-projects are being introduced in other sub-projects and all are converging towards a standard approach to implementation. 22 CHAPTER VI SETTING UP THE M&E SYSTEM (a) Key Issues to be Addressed An M&E system should be designed to meet the specific needs of the individual CGP and the type of sub-projects funded; it should generally be designed before a "Call-for-Proposals" is issued and the formal start of CGP sub-project implementation. Some key issues to be addressed in establishing an M&E system are summarized in Box 5. Box 5: Ensuring Adequacy and Usefulness of an M&E System at CGP Level A World Bank M&E system study found the following questions key to ensuring adequacy and quality of M&E plans: · Does the project document contain a section on M&E and data collection? · Is this M&E plan based on an analysis of project/program needs? · Does the (sub)-project agreement say who is responsible for collecting M&E information? · Does the agreement describe techniques (e.g., reports, surveys, beneficiary assessments, etc.) to be used in M&E? · Are the particulars of these techniques specified (e.g., what questions will be asked, of whom, what information will be sought, etc.)? · Are performance indicators specified by component/sub-component? · Are institutional arrangements for M&E adequate and does the implementing organization have evidence of the institutional capacity needed for M&E, also have they planned a budget for it? · Does the (sub)-project have defined, time-based milestones for implementation? · Would the output of the M&E system provide a basis for a decision to scale up or discontinue the sub-project? (b) Staffing an M&E System In order to satisfactorily perform its M&E responsibilities, the CGP Secretariat needs to have a staff member with the required experience and skills. He/she should be technically competent and well - trained in socio-economics and rural surveys. He/she should be able to provide training to the staff involved with sub-project implementation because most of the CGP sub-projects would not have specialized M&E staff. CGP sub-project implementation teams may, however, also be able to draw on M&E expertise from elsewhere in the institution where they are employed. The cost of implementing and staffing the M&E system should not be higher than 3% of the total cost of the CGP. (c) Staff Training Because of the special M&E requirements of World Bank-financed projects and CGPs, it is often necessary for CGP Secretariat staff responsible for M&E, to receive special training. Such training can either be provided through formal training courses at specialized institutions, or as "in-service" training through exposure to M&E systems for similar projects/programs within or outside the country. CGP sub- project implementation staff may also require training both in monitoring key progress indicators in their own sub-projects and in the collection and submission of data and information to the CGP Secretariat. 23 (d) Monitoring M&E Performance and Fine-tuning It is important for the CGP Secretariat to continuously monitor the performance and outputs of the M&E system. Questions that need to be raised are for example: · Do the findings of field visits concerning sub-project progress routinely confirm the M&E system findings? · Do the findings of the M&E system respond fully to the concerns of the Board? · Is sub-project implementation staff concerned about the time and effort required for collecting the required information and filling out forms? · Are the CGP staff and sub-project implementation staff adequately trained for a well-performing M&E system? · Is the "turn-around" time between data collection, the completion of analysis, and the communication of findings satisfactory and does it contribute to implementation quality? If the answers to these questions are not meeting expectations, adjustments need to be made to ensure better performance of the M&E system. In case it is necessary to further amend and improve the M&E system during project implementation, sometimes consultants may be called in to assist the CGP Board/Secretariat to modify or redesign an M&E system to better respond to the program needs. 24 CHAPTER VII THE MANAGEMENT INFORMATION SYSTEM IN A CGP (a) The Basics The administration of a CGP produces very large amounts of data and information. A complete, up-to-date and accurate database is required to record and process essential information such as on: numbers and performance of promoters/beneficiaries, contractors and intermediaries; technical data on CGP sub-projects (especially agreed milestones and indicators of overall progress); quality of contract implementation; and financial management, procurement and accounting. In addition to the large amount of data that is derived from the M&E system, each CGP-financed sub-project needs to have available basic information on its internal functioning i.e., concerning the use of human resources, physical assets and overall procurement, financial management and accounting. This information needs to be collected through agreed procedures and formats and processed in a timely manner to enhance decision-making aimed at improving the efficiency and effectiveness of a CGP and of the sub-projects it finances. Thus, a high quality and efficient Management Information System6 is needed to collect, store, manage and analyze the large volume of data and information generated in the context of implementing a CGP and to disseminate important conclusions. A full-fledged MIS requires information both on accounting and non-accounting data, on planning and outputs, including their quality and timeliness, on staff performance, service providers etc. An effective MIS facilitates proper and timely execution of routine tasks and improves organization and implementation of the over-all CGP work plan. The MIS comprises the procedures for collecting, handling, storing and analyzing data, and for drawing information and conclusions from the system. An MIS is normally computerized and can be facilitated by several existing MIS software programs. The flow of information needs to be organized and presented in such a way that it is easily accessible and understandable to grantees and decision makers. The development and use of an MIS is instrumental in shaping the structure of a CGP and in facilitating its improvement, and may ultimately determine its success and impact. To define the specifics of an MIS, the following questions should be answered: Users: Who will use the information, to take what decisions and who will provide the information? Data: What information is needed and what data therefore needs to be collected? How should data collection be organized to avoid duplication and inconsistencies, and to minimize data processing? Procedures: Who does what and when, so that information is collected, entered, analyzed and distributed timely to the right people? What should be the interaction between staff responsible for implementation and data, and between those staff themselves? Tools: What tools are needed to carry out the procedures? What is needed to get what is desired without drowning in technology (or providing long-term employment for consultants)? 6An MIS is sometimes defined more broadly as referring to all the information used for program management, including personnel records, training records and plans, sector analytical information, special studies, etc. However, as used in this manual, the MIS relates to M&E mostly concerning information on CGP (sub-project) implementation performance and impact. 25 (b) Essential Aspects of MIS Establishment and Operation Competitive Grant Programs are somewhat unique and they have a number of characteristics that an MIS must meet and be tailored toward, in order to fit them. The design of an MIS for a CGP must take into account: · The high number of contractors involved in sub-projects, often with limited management capacity; · The very large number of small contracts and disbursements involved with CGP sub-projects, with disbursements taking place in tranches; · The variety of procurement methods; · A demand-driven approach that makes rigid planning impossible; · Operations in different sectors that require a wide range of skills, monitoring approaches, service providers and contracts; · The frequent decentralization of management activities; · The sub-contracting of important activities, such as outreach, monitoring and evaluation, and supervision; and, · The multiple stakeholders each with distinct concerns and information about their sub-project. These guidelines are meant to be a practical tool to help those who are setting up an MIS for a CGP. It is important to recognize that in an MIS, flexibility is a must. When first putting an MIS in place for a CGP, it must be determined who will be using what information to make what type of decisions. Each sub-project in the CGP will require special attention to ensure that its requirements can be adequately addressed. Operational data enables the Secretariat to take decisions on disbursements and guide the staff's day-to-day activities. It is essential that the data, once processed can be used by the CGP Secretariat to define needs for action in three specific areas: (i) Problem resolution. Interventions in problem sub-projects that have been assigned "red flags" (Box 3), in which performance indicators and performance criteria reflect significant problems; (ii) Successes. Exceptional performance which might indicate opportunity to scale-up or replicate activities in other areas and to publicize achievements; and, (iii) Selection criteria. Experience can guide the choice of selection criteria for future calls- for-proposals for sub-projects, (based on the socio-economic profile of beneficiaries, geographic location and/or sector of proposed intervention), and information on activities already undertaken (sub-projects financed by sector and by location). An MIS will be required to provide at least two types of information on programsfinancial and strategic. This requires access to financial accounting and procurement information and concerning the impacts of CGP Secretariat activities, broken down by sub-projects and categories such as location, sector, and beneficiary group. (c) What Type of MIS Is Required? The MIS must provide information in a way that is concise and easy to understand. A computerized MIS as a matter of routine should generate standard computer reports. For example, reports on different types of sub-projects for each subject could easily be produced with information grouped by type: accounting and financial, technical, socio-economic, geographic, etc. This can be done by using varying levels of MIS software from "basic" to "complex." It is recommended that an MIS for use in a CGP be kept as simple as possible. This avoids the unnecessary collection of irrelevant data. 26 Guidelines on the process to follow for setting up an MIS, based on the desired features of an efficient system are: · The accounting system should be linked with the sub-project monitoring system; · The MIS tools should be organized "modularly"; · To the extent possible, the MIS should adapt to the decentralized organization and operation of the CGP; · Any new project components should be managed by the MIS in order to prevent the multiplication of management tools; · The MIS should facilitate the management of impact information; · MIS tools should facilitate decision making; and, · Information management should be secure. (d) Design Requirements The success of an MIS depends on the quality of its design. If the basic design is good, subsequent modifications will be relatively easy and development straightforward. The initial design of an MIS should start to take shape before the appraisal stage of a WB-financed project and should be completed by effectiveness. The detailed design should start when the procedures and indicators for the project are being prepared, since the MIS needs to be an integral part of the operating procedures. The designer/consultant for the MIS should work closely with the PMU/Secretariat to identify their information needs, and to ensure that the MIS becomes an effective management tool. For the sub- project monitoring module, the designer should focus on integrating the MIS into the sub-project "life cycle" (proposal, appraisal, selection, contracting, implementation, monitoring, completion, etc.), to provide the CGP Secretariat and sub-project staff with the appropriate tools for information management and decision-making, including up-to-date performance and outcome indicators. The design should be flexible enough to adapt to changes in procedures during the project. A list of critical data variables need to be identified as data input information. Once critical data variables have been entered into the MIS, filtering and aggregation can be implemented to generate the required reports. Various codes can be assigned to sub-projects to allow sorting of the sub-projects and analysis of the various components of the CGP portfolio. This might be by geographic area, administrative area (e.g. province), crop or product, type of project, etc. Critical data variables inputs can be found in the initial sub-project contract documents; some of these are: · Sub-project: objective -and/or goal-, title, duration (start & end dates), grant number, grant amount, other monies, including in kind; · Applicants (name of institution, address, phone/fax/E-mail); · Project Director (name, address, phone/fax/E-mail); · Bank identification name/address, ID code, Bank account #, account owner); · Collaborators; · Qualifications of sub-project leader and staff; and, · Sub-project summary description, justification for sub-project, main goal, objectives, expected outputs/deliverables, KPIs measured against activities, geographical location, technology used, and budget. Data variables that will evolve over a sub-project life span can be found in tranche release request, progress report, and completion report forms. The Tranche Release Form (See Table 4A) should include information on: · Achievement of indicators; 27 · Date of data collection; · Methodology of monitoring; · Financing utilized; · Actions to be taken to remedy problems; · Problems and successes within a specific tranche; and · Project grade (satisfactory, unsatisfactory, etc.) The attachment to the tranche release request form expands on information on the sub-project (see Annex 2) and provide data on: · Innovations introduced; · Constraints; · General status; · Administrative/management issues; · Collaboration · Publications; and, · Secretariat actions required. The Completion Report ( Annex 3) will include similar information as in the progress reports, but will also emphasize: · Quantitative results; · Impact assessment; · Quantifiable environmental and social impacts; · Beneficiary assessment; · Institutional impacts; and, · Potential future work. The above list of critical data variables can and should be adapted where necessary to fit individual CGP sub-projects. Box 6 provides a list of criteria submitted to a software design company to develop an MIS system for the Agricultural Research Council (ARC) of the Ministry of Agriculture and Forestry in Croatia. The design requirements can be used as a template and modified for usage in other CGPs. A comprehensive MIS data base may simplify reporting by linking different reports in the data base such that new reports can be generated drawing on data from the sub-project proposal and previous progress reports. 28 Box 6: Requirements for an MIS to Monitor CGP Implementation for the Agricultural Research Council (ARC), Ministry of Agriculture and Forestry, Croatia The ARC was established in 1997 to administer a CGP for applied and adaptive agricultural research in Croatia to assist in the upgrading of qualifications of Croatian researchers and to contribute to agricultural research policy development. Up to October 1998, ARC had approved about 137 research projects, developed by researchers at the faculties of agriculture, veterinary science, and food technology, and at other agricultural institutes. ARC funds the full costs of the projects, including a salary component additional to the salary received from the home institute, and operational costs (travel, labour, inputs). The fund also covers some major capital expenses, such as laboratory equipment, but cannot normally be used for purchasing vehicles or building new facilities. To monitor the implementation of the ARC-funded research projects, ARC established an MIS to support ARC Board members and project staff in both the strategic and operational management of the CGP. The MIS should help understand to what extent ARC is implementing priorities, assist in avoiding duplication, as well as facilitate monitoring progress in the funded sub-projects. The specific objectives of the MIS are to: (i) Describe the research port-folio of ARC projects and of ARC funding, by commodity, discipline, region, farming system, and executing institution; (ii) Register what type of research has been undertaken in the past, either with ARC, MOST (Ministry of Science and Technology), or other funds; (iii) Describe and evaluate the state of progress in the funded sub-projects (duration; time to completion; funds received and funds due; intermediate results); (iv) Document the final results and achievements of ARC-funded sub-projects; (v) Monitor disbursements by ARC to the approved sub-projects; (vi) Document overall expenses of ARC over time; (vii) Document ARC- financed contributions to research equipment; (viii) Track the approval rate of ARC projects, by commodity, discipline, region, farming system, and institution, over time and in total; and, (ix) Register the researchers and institutions involved with ARC projects. Conditions for implementation of the ARC management information system are: (i) Scientific information in the ARC database should be accessible on the World Wide Web, in order to facilitate communication with researchers and other interested persons; (ii) Financial and management information should be accessible only to ARC Board members and staff; (iii) Required information should be readily obtainable from the project proposal format in use by ARC; (iv) The database should be as compatible as possible with other scientific databases in use in Croatia, notably the MOST database; and, (v) The MIS should be accessible to people with low computational skills. (e) Development and Installation Once the required outputs of the design phase have been approved, they are the reference for the development phase, during which a developer will prepare the computerized tool. In choosing someone to develop the software, it is important to consider: the level of complexity of the design; the qualifications of the developer, including experience in similar assignments; and the sustainability of the system, that is, the need to maintain the system past the development stage, to make any required changes or additions and provide additional training when required. 29 (f) Cost Aspects The cost of an MIS will be determined by two main variables: the complexity of the CGP (size, components, number of donors, level of decentralization) and the complexity of the MIS. The costs cover all the activities necessary to design, develop and put the system in place and make it operational, including staff training. While the cost for a simple basic modular structure can be about US$20,000- 50,000, increasing the sophistication of the system can easily double costs. These costs do not include the purchase of equipment, or the costs of maintenance. Experience suggests that the more complex, completely integrated systems require heavy investments in time and money and can seldom be justified solely in the context of a CGP. The percentage of the costs associated with a specific phase can be estimated as follows: about 30 percent for design, 60 percent for development and implementation and 10 percent for training. However, they may be different depending on the type of consultants hired for design and development (international or local). They are also influenced by the complexity of the projects (which would increase the percentage for the design) and the complexity of the MIS (which would increase the percentage for development). Complex network aspects would also increase the percentage related to development. The simplest way to contain costs is to adapt an existing software package to fit the need of a CGP. Examples of such packages are the "inform-r" program designed by the International Service for National Agricultural Research (ISNAR) which is one of the CGIAR Institutes, "Promes" designed by GLOBALFACTS software manufacturer, and "Divana" a Croatian program designed by Microprom, another software manufacturer.7 Developing an MIS exclusively for a particular CGP can be extremely expensive and it is strongly advised that a software adaptation be used. (g) Monitoring MIS Performance and Organizing Maintenance/Upgrading How often should maintenance take place? Ideally, continuously, but not many CGPs have a volume of operations that warrants an in-house specialist. On the other hand, calling an MIS consultant only in emergency cases would be false economy, because often the problems cannot be rectified unless major (costly!) changes are introduced. It is recommended to go beyond the common practice of reviewing the MIS only at the CGP mid-term review, and to bring in a specialist at least once a year, especially during the early stages of a project. The specialist should coordinate with project information technology staff in the procurement unit and set up of all required hardware and software, including establishing all the networking and communication links required for the computerized MIS. A "User's Manual" should be produced for MIS practitioners including instructions both on procedures and technical aspects; it will also be necessary to coordinate with management the arrangements for the required training (number of sessions, participants, equipment, etc). It will need to be ensured that all users and operators have the computer skills necessary for operating the system (training courses in computer skills, if required, should be conducted prior to the MIS training sessions). When a paper-based system or other software is already in place, it will be important to develop a transition strategy to the newly developed system, minimizing interruptions to the ongoing program/sub- projects operation. Maintenance of the MIS will ensure its proper functioning, as well as updating to keep it relevant, efficient and effective. Given the dynamic nature of CGPs, systems tend to become obsolete even after a few years. Hence, it is important, as part of the design and development planning process, to include a maintenance strategy to regularly monitor MIS relevance and performance. Preferably, such an exercise is part of an overall information technology and information management strategy for the CGP. 7These are only examples and should not be considered in any way as an endorsement of the particular packages or firms. 30 A maintenance strategy may include a review of information requirements and needs for the CG program, including data gathering and reporting needs (is the information collected and produced useful? Is it used? Is there additional information that should be collected or produced?). A review of MIS efficiency is also important (is the software storing and producing information as expected? If not, why not? ). Also, does the Secretariat have requests or suggestions for improvements? If so can they be incorporated into the system. 31 CHAPTER VIII IMPACT ASSESSMENT OF CGP SUB-PROJECTS (a) Impact Assessment It is important to conduct an overall project impact assessment towards the end of project completion. However, it is also recommended that a sample of sub-projects should be evaluated at the end of their completion. The cost of these evaluations should be kept to a minimum level if possible. This chapter provides a brief guideline for conducting an impact assessment of CGP sub-projects. What do we mean by measurement of impacts? Impact measurement is the quantitative and qualitative assessment of the extent to which a project meets its objectives. A good logframe defines indicators in terms of a hierarchy from project outputs to project outcomes and project impacts; these terms are defined below: · Project outputs are the immediate results of the research or extension activity, such as an improved variety or the number of farmers trained in IPM. These outputs should be readily available from the sub-project reports and the MIS; · Project outcomes refer to the immediate effect of the output on users. This would typically be assessed by the extent of adoption of an improved variety or IPM practice, the gain in yield and profits per ha from adoption, and the reduction in pesticide use (for the above cases); and, · Project impacts assess changes in overall societal goals of improved incomes, poverty reduction and environmental preservation as a result of the project activities. Typically, these impacts would be measured by the following types of indicators: · Economic benefits in terms of the total economic gain due to the adoption of the practice, through higher incomes and/or cheaper food; · Impacts on poverty reduction though indicators of the number and type of direct beneficiaries in terms of farm size or income level as one measure; · Cost effectiveness, for example, cost per beneficiary in the case of technology transfer sub- projects; · Indicators of environmental benefits, such as reduction in pesticide use, or improvements in soil properties; and, · Indicators of institutional impacts at the local level- for example, formation and strengthening of producer associations and alliances. Assessment of the impacts of CGP sub-projects should focus on economic benefits in terms of improved productivity in relation to the cost of the sub-projects, with attention also to social, environmental and institutional benefits. Why do we measure impacts? There are two reasons to carry out impact assessments. First, the information provides important feedback to CGP management on the overall performance of its program and sub-project portfolio that can be used to adjust program strategies and priorities. Second, evidence of impacts is important in future dialogues with financing agencies in order to consolidate, institutionalize and possibly expand the CGP, as a critical and permanent component of the national AKIS. Basic concepts in impact assessment. Evaluation is best carried out by an independent group. This is commonly done in economic impact assessment, which may use two basic approaches: 32 · The "before and after" approach in which farmers' technology adoption and incomes are measured at the beginning of the sub-project, and then again after the completion of the sub-project; and/or · The "with and without" approach in which technology adoption and incomes of those farmers who participate in a project are compared to those who do not. The combination of these approaches provides the most rigorous impact assessment. Evaluation can also be carried out by the users themselves. Such an approach is usually called a beneficiary assessment (BA). Economic methods for impact evaluation. Methods for estimating economic benefits and costs of research and extension activities are well known. Annex 5 summarizes key references and software (many of which are available on the Web). The following parameters are needed to estimate benefits: · The change in unit cost of production (per ton) or value of an increase in quality due to the adoption of the technology; · The value of the production affected; and · The rate of adoption for a technology and its "ceiling". The flow of benefits and costs of a sub-project is often illustrated as in Figure 1. In the early years, costs of the research result in a negative flow of benefits. However, after the research is completed, the technology is gradually adopted, resulting in increasing the flows of benefits. Finally, farms stop using the technology because it becomes obsolete. These benefits are discounted and then compared with the net present value of costs to estimate a benefit-cost ratio or "internal rate of return" on the investment (the IRR).8 Figure 1. Flow of Net Benefits from Research 6 4 ($) 2 0 -2 Beneftis -4 -6 1 3 5 7 9 11 13 15 Year (b) A Practical Approach to Impact Assessment Economic impacts: A CGP typically supports a large number of diverse and geographically dispersed sub-projects in research and extension which makes it very costly to undertake rigorous impact assessment involving collecting base-line information and conducting formal surveys. A CGP must seek cost-effective short-cuts to evaluation. Finally, CGP sub-projects often fund marginal costs of testing and 8The internal rate of return is then computed by the value of the discount rate r, that makes the present value of net benefits equal to zero. That is t (B-C)/(1+r) = 0, where t is the year and r is the interest rate. 33 adapting technologies produced by several years of previous research. Evaluation must therefore look at the value added by the CGP sub-project as such, which in many cases is through faster adoption of a technology. Estimating the rate of return on research and extension investment requires several steps: · Defining the target group for the sub-project (this should be done in the sub-project proposal); · Defining the sub-project outputs in the form of new technologies or information and estimating the adoption of these outputs. For each technology generated and diffused by the sub-project, an estimate needs to be made of their adoption over time by final users (Figure 2); · Estimating the economic impact of sub-project outputs. This involves estimating the impact of sub-project outputs on yield, cost or quality; · Measuring sub-project costs. This step involves accounting for all sub-project research and extension costs associated with the technology in the selected time period; and, · Calculating net economic benefits to society from a given research and extension sub-project and estimating the IRR. To keep impact assessment within reasonable costs, only a sub-sample of sub-projects can be evaluated. While it is desirable to conduct formal surveys with randomly selected farmers using a questionnaire to estimate adoption or effects on yields, this is usually not cost-effective for a CGP, given the large number of dispersed sub-projects. It is therefore advisable to use a "rapid rural appraisal" approach for a selected sub-project in its area of influence. Figure 2. Adoption of Technologies Percent adopted Begin End of project ning of project Year Quantifying environment and social impacts: Evaluation of sub-projects should also be used to quantify, where appropriate, environmental and institutional benefits. Many sub-project reports may already provide estimates of reduction in pesticide use and changes in critical soil test parameters. These can be checked in the field and summarized to show changes in environmental indicators. Likewise, field work should be done to assess the profiles of adopters in terms of farm size, gender and ethnicity, to analyze the extent that the sub-project has reached socially and economically disadvantaged groups of the population, and contributed to poverty reduction. 34 Baseline information: A key question that must be defined at the beginning of the CGP and before starting each sub-project, is how much baseline information should be collected. Generally it is not cost-effective to collect comprehensive baseline information for each sub-project, given their small size and geographical dispersion. Good practice is to have each sub-project team collect the minimum information needed to track technology adoption and the extent of impact. This information is technology specific. For a varietal technology, this requires information on the estimated area under different varieties at the beginning of the sub-project. An IPM sub-project needs baseline information on the amount and types of pesticides used by farmers at the beginning of the sub-project. For some types of sub-projects, it may be possible to avoid collecting baseline information; for example, when an entirely new practice is being introduced for the first time. Box 7 describes the practices used in the earlier quoted CGP in Ecuador. Box 7: Ecuador--Program for Modernizing Agricultural and Livestock Services (PROMSA) The Use of "Reference Groups" for Impact Evaluation Each (sub)-project grantee must identify its target group of potential users of the outputs from the (sub)-project. The target group for applied and adaptive research often comprises farmers, but may be other researchers, agribusinesses, or others for different types of projects. Then, each sub-project identifies a group of 4-8 individuals from the target group or other interested stakeholders (other researchers, extension staff, agribusinesses, etc.), to assist in project implementation. This "Reference Group" participates in project planning, implementation and evaluation. The Group might meet several times a year to consult on project issues. The Group serves as a mechanism for participatory evaluation within the project; such evaluation might or might not be documented. The Group might also participate in more formal evaluations of the (sub)-project. Resource needs for impact evaluation: Resource needs for impact analysis will depend on the number and type of sub-projects. In Colombia (Box 8) CGP sub-project impact assessment required two field teams of two persons each working for up to four months, and associated travel and per diems, in order to evaluate 60 sub-projects. Box 8: Impact Evaluation in PRONATTA, Colombia PRONATTA in collaboration with a local and an international consultant has refined a cost- effective methodology for evaluation of the impacts of individual sub-projects. A sample of 60 sub- projects has been selected at random from over 300 completed sub-projects, stratified by region and by type of sub-project (research or extension). For each sub-project, an economic cost-benefit analysis is being carried out, based on data available in the PRONATTA MIS, opinions of experts in the sub- project theme and location, and field visits by a team consisting of two PRONATTA staff members and one contracted expert in the sub-project theme. The team is also assessing sub-project impacts in terms of a series of indicators of sustainability, competitiveness, social capital formation, and overall sub- project efficiency and effectiveness. Data collection and analytical methods were refined through a series of discussions and field tests of 12 sub-projects. (c) Beneficiary Assessment Beneficiary assessment has been successfully used in many programs to provide user feed-back on extension performance. A BA is carried out to inform management about ways of improving project performance from the point of view of the final users. More specific objectives include: determining the level of satisfaction of intended beneficiaries; understanding the degree and manner in which community members have participated in various phases of sub-project implementation; and learning how stakeholders feel performance of a sub-project could be improved. 35 Methodology: The core techniques are: · Conversational interviewing among representative groups of key stakeholders (intended beneficiaries, contractors, NGOs, government officials, etc.); · Focused interviews with intended beneficiaries; · Participant observations; and, · Institutional assessment. Samples should be representative of both numbers of farmers reached and numbers of sub-projects funded. Stratification should be by gender, ethnicity (where relevant), sub-project type and region of country. Interview Guide: The issues for the BA will be determined according to the nature of the research or extension activity. They will be addressed largely by interviewing beneficiaries, using a basic interview guide that may be modified for use with different stakeholder groups. This guide could include the following topics: · Exposure to the sub-project. Determining how people learned of its existence and what they know about it; · Participation. Degree and nature of involvement in decisions regarding the activity in the community and maintenance of what was created; · Partnerships. Collaboration with other entities -- local governments, NGOs, private sector; · Satisfaction with sub-project objectives, mode of operations, and results; and, · Recommendations for improvements in project operations. (d) Assessing Institutional Impacts Indicators for assessing institutional impacts depend on the overall objectives for using competitive funding mechanisms; these usually include one or more of the following: · To foster institutional pluralism by funding research and extension providers, other than National Agricultural Research Institutes (NARIs), such as universities, private firms, and NGOs; · To provide much needed operating cost funding to better utilize the substantial existing investment in scientists and infrastructure; · To foster partnerships among research and extension providers, by requiring that proposals be jointly prepared and executed; · To promote reform of NARIs by requiring them to compete for funding; · To upgrade the quality of the research and extension effort through rigorous screening of proposals and monitoring of implementation; · To promote a more demand-driven agenda by requiring that users and beneficiaries participate in proposal preparation and execution; · To quickly address new and important research and extension priorities; and, · To attract new sources of funding though establishment of technology funds set up on a competitive basis. Indicators of many of these impacts can be quantified and collated using available information from sub-project reports and the MIS. Examples include: · Number of different research and extension providers involved; · Percentage of financing provided by farmers and the private sector; · Number of farmer groups executing sub-projects; · Number of new alliances created by sub-projects; · Ratio of operating budget to salaries in public organizations executing sub-projects; · Shifts in research and extension resources toward identified new priorities. 36 ANNEX 1 DETAILED OUTLINE FOR CGP SUB- PROJECT PROPOSALS TO BE COMPLETED BEFORE GRANT AGREEMENT Date of Submission: ___________ Registration Number: ___________ Project title: ___________________________________________________________ Planned duration: __________________________________________________ Proposed starting date: ______________________________________________ Completion date: ___________________________________________________ Total project cost: $ ________________________________________________ Requested grant amount: $________________ Percent of total __________________ Amount of co-financing or in-kind support: $ _____________________________ Summary description of the project (maximum 1,000 words): Statement: This project application is submitted by ______________________, and has been made in full agreement with all participating institutions. The principal institution (institution of the Principal Officer) and the other participating institutions agree to provide timely and adequate staffing, buildings, logistic and administrative support, and any other support that the project requires. Accounts will be maintained, funds utilized, and project implementation supervised and monitored in accordance with the attached project documents and instructions from the "Agricultural Services Board." Responsibilities: The following person will be directly responsible for project implementation, devote the necessary time and effort to its effective and efficient implementation, and be the principal contact person for the Agricultural Services Board/Project Management Unit. Name: ______________________________________________________________ Organization: ________________________________________________________ Contact: ___________________________________________________________ Signature of the Principal Officer ______________________ Date: ____________ Commitment from participating organizations: The participating institutions commit to make available: · Funds, as indicated in the attached budget; · Qualified, full time staff (or equivalent part-time); · Timely operation and maintenance funds for vehicles; · Suitable office / laboratory / field facilities including their utilities for the duration of the project; · Other relevant support for an efficient and effective implementation of the project. Signatures: Name 1 ___________ Name 2 ___________ Name 3 ___________ Institution ___________ Institution ___________ Institution ___________ 1 2 3 Date ___________ Date ___________ Date ___________ Description of the Project: The description of the project must cover the following: · Resources available for the project, including facilities; · Objectives and methodology of the proposed project; · The relation of the project with other ongoing activities; · Key components of the project and delivery mechanisms; · Probability and methodology of adoption; 37 · Major risks to a successful completion and remedial actions; · Sustainability of the proposed activities; · Socio-economic impact. · Detailed activity budget, based on the activities identified in the project description, providing total expenditures, and CGP costs, source of non-CGP funding and annual phasing of the project procurement, activities and funding for a period up to three years); · Consolidated budget; · Completed templates for the time­bound schedule of tranche payments and tranche release request form. In addition to the narrative, the proposal will include where appropriate, the following annexes: · Cost / benefit analysis, based on the model suggested by the Secretariat; · Principal Officer's CV and information about their institution. Note: Project activities as described in the descriptive part should be represented in the following tables. 38 THE FOLLOWING FORMS ARE REQUIRED ALONG WITH THE DETAILED OUTLINE WHEN SUBMITTING A PROPOSAL FOR A SUB-PROJECT. ANNEX 1, TABLE 1: Template for the Time-bound Schedule of Tranche Payments. Tranches Date Milestones achieved; conditions for tranche release. Amount (US$) Tranche 1 Tranche 2 Tranche 3 Tranche 4 Tranche 5 Includes Final Report (10%) TOTAL 39 THE FOLLOWING FORMS ARE REQUIRED ALONG WITH THE DETAILED OUTLINE WHEN SUBMITTING A PROPOSAL FOR A SUB-PROJECT. ANNEX 1 TABLE 2: Tranche Release Request Form to be attached to the contract and filled out for each tranche. This form will be the basis for the monitoring the project. Project Title & No Start End No. Activities Date Date Quantifiable Indicators Financial Monitoring Procurement Monitoring Tranchne planned planned Key Performance Indicator funds allocated Main Items expected to be procured 1 2 3 4 5 6 This Form Is To Be Attached to the Contract 40 ANNEX 1, TABLE 3: Detailed Sub-project Activity Budget For Financial Monitoring Costs Source of Non-Competitive Grant Scheme Financing Phasing Activities Expenditure Competitive Beneficiary Total Grant Scheme Institution's Contribution Year 1 Year 2 Year 3 Contribution Contribution of Partners 1. Activity 1 Total activity 1 Staff costs a. Professional staff wages 1) b. Support staff wages 2) c. Technical assistance costs - National consultants 3) - Foreign consultants 3) Staff travel costs: a. Domestic b. International Operating cost: - Materials and services - Consumables - Fuel Capital cost 4) Training costs: a. Short time training fees5) b. Field trips and training for farmers - Lecturers fees 6) - Participant costs7) - Training materials *2. Activity 2 Idem Sub-total (Sum of activities) Overhead Communications, water, Expenses electricity and heating costs8) Unforeseen Maximum of 10 percent from Expenses sub-total TOTAL Provide this information for each of the activities. 1)Number of own staff x number of person- months x monthly wages; 2)Number of auxiliary staff x number of person-months x daily wages; 3)Number of consultants x person-hour x hour cost; 4)Detailed per element; 5)Number of lecturers of the implementing team (transport, lodging, per diem and cost of lectures); 6)Number of lecturers x number of person-hour x daily cost; 7)Number of persons x number of days x daily cost 8)Maximum 10 percent from sub-total NOTE: These elements have to be detailed for each activity 41 ANNEX 2 ANNEX 2, TABLE 1 MONITORING TEMPLATES TO BE SUBMITTED FOR TRANCHE RELEASE REQUEST AND TO PROVIDE PROGRESS REPORTING ON A SUB-PROJECT. Tranche release request form to be completed and submitted before tranche funding is approved. Project Title & No Tranche No No Activities Start Date End Date Quantifiable Financial Procurement **Indicators Indicators Monitoring Monitoring Verified by CGS Planned actual planned Actual Key Was allocated Category* Main Items Main Items Office Field Performance indicator Fu funds CGS expected to procured Indicator achieved. nds use be Yes, No, spen procured Partially* t 1 2 3 4 5 6 Status of project: Highly Satisfactory, Satisfactory, Marginally Satisfactory, Unsatisfactory, ALERT * If the indicators are partially completed or not completed at all, provide explanations on the attached form ** The column is for CGS use only! This form is for Monitoring purpose 42 ANNEX 2: TABLE 2: Template for the Tranche Release Request Attachment Explanation: Monitorable Reason for Action Taken/To be Taken to Rectify the Situation Indicator No partial or non-completion of indicators Explanation of Difference in Expenditure Allocated versus Spent Explanation of Difference in Proposed Procurement versus Actual Procurement On GENERAL COMMENTS ON PROBLEMS GENERAL COMMENTS ON SUCCESSES completion of specific tranche This was prepared by:______________________________ Date_____________________ Information was certified by_________________________ Date_____________________ Decision________________________________________________________________________________________ 43 ANNEX 2B An example of a completed template completed by a grantee in Georgia. ANNEX 2B, TABLE 1, Example of Monitoring Table completed for the First Tranche of a project on Biohumus and Earthworms in Georgia. Activities Start Date End Date Quantifiable Financial Procurement **Indicators 1 e Indicators Monitoring Monitoring Verified by Secretariat Planned Actual Planned Actual Key Performance Was Funds * Spe alloc Of Ca Main Items Main Items Office Field Tranch Indicator indicator Funds ndin te expected to procured achieved. ated spent gory be procured (Yes, No, No. Partially)* g* 1 Repair building for earthworm 1.03.02 10.03.02 Repair building Yes 1000 Grant: Goods Gas line Gas line, X population and fencing the to house worms 900 800 Fencing Fencing experimental plots and fence Works Concrete Concrete demonstration Other 200 labour Labour plots 100 2 Preparation of containers to 1.03.02 1.05.02 30 containers Yes 1100 Grant: Goods Materials Salary of 3 X house worm and generate built and located 1300 1500 for specialists biomass and locate them on on2 farms Works containers for 3 various farms 100 and labour months; Other to build field work 300 containers 3 Purchase the materials 5.5.02 05.6.02 Obtaining and Yes 1900 Grant: Goods Purchased Purchased X necessary to generate collecting worms 1400 1200 worms and worms and biohummas and purchase Trip manure manure manure for Other: 200 Trip to biohummas locate worms 4 Plant each of the 30 6.06.02 8.8.02 Harvest the No 2000 Grant: Vegetable X containers with different vegetables grown seed vegetables in purchase biohummas.show Labour results in higher Other yields Status of project: Highly Satisfactory, Satisfactory, Marginally Satisfactory, Unsatisfactory, ALERT If the indicators are partially completed or not completed at all, provide explanations on the attached form **Category of spending will come from one of 5 categories Goods Works Trips Training Operations 44 ANNEX 2B TABLE 2 : An Example of a Completed Attachment Sheet providing Explanation of Shortfalls in Sub-project KPIs. Monitorable Reason for partial or non-completion of Action Taken/To be Taken to Rectify the Situation Indicator No indicators 30 containers were not sown with different 3 containers will be used to grow 3 different types of vegetables (potato carrots, tomato, onions, vegetables in worm generated biohummas and 3 garlic etc). Due to time constraints. controls will be ground under the same conditions as controls. A greenhouse will be used to grow the vegetables in a three month period staring in December. Explanation of Expenditure Allocated versus Spent No explanation of funds in activity 4 Difference in Proposed Procurement versus Actual Procurement Review activity 4 On GENERAL COMMENTS ON PROBLEMS GENERAL COMMENTS ON SUCCESSES completion of specific tranche Time constraints prevented the plating of 30 The project will eventually meet all criteria and containers goals. This was prepared by: Sub-project leader Jamlet Badagadaze Date: 31/08/02 Information certified by: CGP Secretariat Rezo Asatiani Date:4/09/02 Decision: Implementation problems; tranche release should not proceed until further results seen. 45 ANNEX 2C ANNEX 2C TABLE 1: This Template is For The Secretariat's Use Only. Sub- Project Assessment and Comment Form ( it provides a tool for monitoring paperwork when reports arrive from the grantee) Project Number____________________ Project Name______________________ Project Director________________ Tranche Number General Comments on Project: Financial Aspects: Technical Aspects: Procurement Aspects: Legal Aspects: Overall Status: HIGHLY SATISFACTORY, SATISFACTORY, MARGINALLY SATISFACTORY, UNSATISFACTORY Date of Report Submission: Approval Date of the Report: Comments Regarding the Report: 46 ANNEX 3 FORMAT FOR A CGP SUB-PROJECT COMPLETION REPORT (Maximum length 15 pages, excluding annexes, if any) Purpose: The Sub-project Completion Report is the last activity under each CGP-financed sub-project. Its purpose is to link all findings derived from implementing the sub-project so that the overall achievements and impact of the entire project can be assessed. Project achievements should be discussed in terms of scientific accomplishments, contribution to human resource development, the relevance of findings to development, how the information/technology is being disseminated, what the present and expected future degree of adoption is and where relevant, the actual impact on productivity and farm incomes. The Final Sub-project Completion Report should not be a repetition of the regular progress reports and no regular progress report should be submitted when a completion report is due. The Sub-project Completion Report should include the following: Summary: The Summary should be typed double-spaced on a separate page. Use the same guidelines as for the regular progress report, but review and summarize the entire sub-project experience. The summary should clearly place project accomplishments in the overall context of agricultural development and/or environmental protection in the area served. Objectives and Rationale: Describe why the sub-project was conducted. What problems were addressed and why is it important to development? Briefly describe how the project fits into on-going research by other scientists, extension activities and reference pertinent literature. Describe the innovative aspects of the sub-project. Briefly explain how other institutions supported the sub-project. Methods and Results: Summarize how data were collected or how the sub-project was implemented. Present data through use of tables, charts, photographs. When appropriate, data should be statistically analyzed. This section should be presented as if for publication in a journal or extension publication. Project Activities/Outputs: List the important achievements (milestones, outputs and impacts) of the sub-project. Where appropriate provide a list of publications not previously submitted, patents (pending), and meetings attended, held or scheduled. Future Work: Will the project need further follow-up activities and/or lead to future work? If so, describe this. 47 ANNEX 4 M&E in the Nepal Hill Agricultural Research Project The DFID-financed Hill Agricultural Research Project (HARP) in Nepal, has the objective of "establishing a sustainable and effective hill agriculture research system". One component of the project is a Hill Research (HR) Program of competitive grants for research sub-projects. The HR Program was to demonstrate the value of a competitive funding program for research by producing high quality research outputs. The HR program provides a model of a well-managed competitive research fund. The HR program placed a high value on M&E to enable the Secretariat to know what happens with sub-projects and to demonstrate efficient and effective production of research outputs. The M&E system is based on use of logical frameworks for all sub-projects financed. These focus on activities that produce time-bound outputs leading to achievement of sub-project objectives and sub-project goals. Main elements of the M&E system are: Tri-mesterly reports that include: (i) assessment of likelihood of output delivery; (ii) activities against the project plan and milestones; (iii) problems encountered; (iv) financial expenditures against budget; (v) activities expected for the next trimester; and (vi), funds required for the next trimester. · Annual reports that are modified versions of tri-mesterly reports and summarize progress over the year. · Output-to-Purpose Reviews; these are progress review meetings held before the start of the final year of a sub-project. · Project Completion Reports that cover:(i) background to the sub-project; (ii) research implementation performance and log frame; (iii) status of delivery of expected outputs and results; (iv) prospects for adoption of new technologies and achievement of sub-project objectives; (v) data on key performance indicators; (vi) proposed follow-up actions; (vii) lessons learned; and, (viii) publications. The total length is about 10 pages. Routine reporting is mainly the responsibility of the sub-project grantee and is complemented and supplemented by field visits from the HR Secretariat staff to sub-project sites. The reporting and field visits allow the research teams to receive feedback from the Secretariat and are required for disbursement of additional tranches of funding. The original tri-mesterly reports succumbed to a natural inclination to focus on activities; they missed the more important issues of whether sub-project activities were likely to deliver on expected outputs. Change in format corrected this by emphasizing delivery of outputs. Performance assessment is a final aspect of the M&E system; it involves assessing how well the grantees have carried out the sub-projects, in order to help identify weaker grant recipients that should be excluded from future consideration or to help to improve their capabilities. The PCR is the basis for the performance assessment, which is carried out by technical specialists from the implementing agency. The PCR serves essentially as a "self- assessment" by the implementing agency. The performance assessment executed under the responsibility of the Secretariat, evaluates the PCR against a set of 28 indicators covering: project reporting, project justification, project implementation, delivery of project outputs, contribution of outputs to expected outcomes and benefits, and uptake of outputs. During implementation, the HR Program adopted the term "monitoring and support" for implementation monitoring during the life of the sub-project, while "evaluation" was used to refer to measurement of adoption (uptake) of research outputs after the sub-project was completed. Such evaluations after the end of sub-projects can be problematic but the HR Program made arrangements for contracting private organizations to undertake unbiased assessments of the achievements and impacts of a sample of sub-projects. This information on outcomes and impacts of the sub-projects is central to evaluation of the HARP as a whole. 48 ANNEX 5 Key References and Additional Readings Alex,G.1998. Assessing Agricultural Research. Towards Consensus on a Framework for Performance and Impact Assessment. Agricultural Research and Extension Group. Special Report No, 6 World Bank Washington, D.C. Alex,G., Byerlee D.2000. Monitoring and Evaluation for AKIS Project Framework and Options. World Bank, Washington, D.C. Horstkotte-Wesseler,G., M. Maredia, D.Byerlee, and G.Alex. 2000. Ex-Ante Economic Analysis in AKIS Projects: Methods and Guidelines for Good Practice. AKIS Thematic Group, Rural Development Family. World Bank, Washington, D.C. ISNAR.2000. Listing of Research Monitoring and Evaluation Publications. http://www.cgiar.org/isnar/publications/eval.htm. International Service for National Agricultural Research (ISNAR), The Hague. Lecuit,L., J.Elder, C.Hurtado, F.Rantua, K.Siblini, and M. Tovo. 1999. DeMIStifying MIS-Guidelines for Management Information Systems in Social Funds. World Bank, Washington, D.C. Misra, D. 1997. "Monitoring Extension Programmes and Resources." In Improving Agricultural Extension: A Reference Manual. Food and Agricultural Organization of the United Nations. Rome. Salmen, L.F. 2000. The Voice of the Farmer in Agricultural Extension. AKIS Discussion Paper, World Bank, Washington, D.C. World Bank. 1996. Lessons and Practices: Designing Project Monitoring and Evaluation. Operations Evaluation department. World Bank, Washington, D.C. Key Reference Sources on Impact Assessment Reference Books Alston, J., G. Norton, and P. Pardey. 1995. Presents the principles and practice of ex-post and ex-ante Science Under Scarcity, Cornell University economic evaluation methods and their use in research priority Press, Ithaca. setting. A wide range of approaches are reviewed, synthesized, and assessed using a unifying conceptual framework. Horton, D., P. Bellantyne, W. Peterson, B. A sourcebook that provides a synthesis of literature and experience Uribe, D. Gapasin, and K. Sheridan. 1993. on research monitoring and evaluation principles, processes, and Monitoring and Evaluating Agricultural methods. Research: A Sourcebook. CAB International, Wallingford. Manuals CIMMYT, 1993. The Adoption of An excellent manual for practitioners that explains step by step the Agricultural Technology: CIMMYT, Mexico.approaches to estimating technology adoption. Masters, W.A., B. Coulibaly, D. Sanogo, M. A guidebook intended to provide a concise summary of the tools Sidibé, and A. Williams. 1996. The needed to conduct persuasive impact studies, enabling researchers Economic Impact of Agricultural Research: to quantify the economic benefits and costs of their work. Three 49 A Practical Guide. Department of spreadsheet-based computer exercises help apply the methods Agricultural Economics, Purdue University, described in the manual. West Lafayette, IN. E-mail: Masters@AgEcon.Purdue.edu Janssen, W. and A. Kissi. 1997. Planning and A practical approach to combine natural resource management and Priority Setting for Regional Research. productivity concerns; introduces a methodology to choose Research Management Guidelines 4. regional priority research projects based on economic International Service for National considerations. Agricultural Research, The Hague. Available online at www.cgiar.org/isnar Collion, M.H. and A. Kissi. 1995. Guide to An approach to research program planning by objective, based on Program Planning and Priority Setting. a series of steps which include benefit-cost analysis as an approach Research Management Guidelines No. 2E. to priority setting. ISNAR, The Hague. Available online at www.cgiar.org/isnar Software Dream© - Dynamic Research Evaluation Www.cgiar.org/ifpri for Management This is a menu-driven computer program developed by ISNAR to facilitate application of the economic surplus model under a variety of market situations. The latest version of this program (June 1998) is available from IFPRI (contact: Stanley Wood, s.wood@cgnet.com). MODEXC ­ Model for Economic Analysis Http://www.ciat.cgiar.org/inslinks/modexc.htm This is a spreadsheet based economic analysis model developed by CIAT, available on the World Wide Web, that allows to calculate the NPV, IRR and B/C ratio of investments in agricultural research. The model can be used for economic analysis of technical change both ex-ante and ex-post under alternate scenarios of market situation. Web Sites ISNAR's information and discussion Http://www.cgiar.org/isnar/Fora/Priority/index.htm forum for agricultural research priority This web site provides the process, steps and methods of setting research priority setting. CIAT Impact Project Http://www.ciat.cgiar.org/inicio_in.htm, the CIAT web site features abstracts on the impact of agricultural research, data bases, trends, as well as a download version of MODEXC (see above). 50