Document of The World Bank Report No: ICR00003849 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-48910) ON A CREDIT IN THE AMOUNT OF SDR 10.3 MILLION (US$16.0 MILLION EQUIVALENT) TO THE REPUBLIC OF ARMENIA FOR A COMMUNITY AGRICULTURAL RESOURCES MANAGEMENT AND COMPETITIVENESS PROJECT April 28, 2017 Agriculture Global Practice South Caucasus Country Unit Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective September 30, 2016) Currency Unit = US$; SDR 474 AMD = 1 US$ 1 SDR = 0.71 US$ FISCAL YEAR July 1 June 30 ABBREVIATIONS AND ACRONYMS AI Artificial Insemination AIPP Avian Influenza Preparedness Project ANAU Armenian National Agrarian University AMD Armenian Dram APIU Agriculture Projects Implementation Unit CARD Center for Agriculture and Rural Development CARMAC Community Agricultural Resources Management and Competitiveness Project CPMLDC Community Pasture Management and Livestock Development Committee CPMLDP Community Pasture Management and Livestock Development Plan CPS Country Partnership Strategy CGP Competitive Grant Program CIS Commonwealth Independent States CV Community Veterinarian EMP Environment Management Plan EOP End of Project ERR Economic Rate of Return FM Financial Management GDP Gross Domestic Product GIZ Gesellschaft für Internationale Zusammenarbeit GoA Government of Armenia HI Heifer International IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report IDA International Development Association IFR Interim Financial Reports ISR Implementation Status and Results Report MASC Marz Agricultural Support Center METT Management Effectiveness Tracking Tool MoA Ministry of Agriculture MST Marz Support Team M&E Monitoring and Evaluation NGO Non-Governmental Organization NPK Nitrogen (N), Phosphorus (P) and Potassium (K) NPV Net Present Value OP/BP Operations Policy/Business/Policy OM Operational Manual O&M Operation and Maintenance PAD Project Appraisal Document PCR Project Completion Report PDO Project Development Objective POG Passing on the Gift PPF Project Preparation Fund PUA Pasture Users Association RASC Republican Agricultural Support Center RESCAD Rural Enterprise and Small-scale Commercial Agriculture Development SDR Special Drawing Rights SIL Specific Investment Loan SME DNC Small and Medium Enterprise Development National Center SMS Short Message Service TAP Technology Assessment Project TTL Task Team Leader UNDP United Nations Development Programme USAID United States Aid for International Development USD United States Dollar VSC Veterinary Service Center WB World Bank WWF World Wide Fund for Nature Country Director: Mercy Miyang Tembon Country Manager: Laura E. Bailey Senior Global Practice Director: Juergen Voegele Practice Manager: Julian Lampietti Project Team Leader: Arusyak Alaverdyan ICR Team Leader: Qun Li ARMENIA Community Agricultural Resources Management and Competitiveness Project CONTENTS 1. Project Context, Development Objectives and Design .............................................................. 1 2. Key Factors Affecting Implementation and Outcomes .............................................................. 3 3. Assessment of Outcomes .......................................................................................................... 10 4. Assessment of Risk to Development Outcome ......................................................................... 17 5. Assessment of Bank and Recipient Performance ..................................................................... 18 6. Lessons Learned........................................................................................................................ 21 Annex 1. Project Costs and Financing .......................................................................................... 23 Annex 2. Outputs by Component.................................................................................................. 24 Annex 3. Economic and Financial Analysis Introduction ............................................................ 36 Annex 4. Bank Lending and Implementation Support/Supervision Processes............................. 57 Annex 5. Beneficiary Survey Results ........................................................................................... 59 Annex 6. Stakeholder Workshop Report and Results................................................................... 64 Annex 7. Summary of Recipient 's PCR and/or Comments on Draft ICR ................................... 65 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................................... 72 Annex 9. List of Supporting Documents ...................................................................................... 73 MAP ARMENIA Community Agricultural Resources Management and Competitiveness Project A. Basic Information Community Agricultural Resource Country: Armenia Project Name: Management and Competitiveness Project Project ID: P120028 L/C/TF Number(s): IDA-48910 ICR Date: 03/20/2017 ICR Type: Core ICR REPUBLIC OF Lending Instrument: SIL Recipient: ARMENIA Original Total XDR 10.30M Disbursed Amount: XDR 10.20M Commitment: Revised Amount: XDR 10.30M Environmental Category: B Implementing Agencies: Agriculture Project Implementation Unit Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 02/22/2010 Effectiveness: 07/29/2011 07/26/2011 Appraisal: 11/29/2010 Restructuring(s): Approval: 03/22/2011 Mid-term Review: 07/28/2014 07/28/2014 Closing: 09/30/2016 09/30/2016 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Moderate Bank Performance: Satisfactory Recipient Performance: Satisfactory C.2 Detailed Ratings of Bank and Recipient Performance (by ICR) Bank Ratings Recipient Ratings Quality at Entry: Satisfactory Government: Satisfactory i Implementing Quality of Supervision: Satisfactory Satisfactory Agency/Agencies: Overall Bank Overall Recipient Satisfactory Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Quality at Entry Project at any time No None (QEA): (Yes/No): Problem Project at any Quality of No None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Major Sector/Sector Agriculture, Fishing and Forestry Other Agriculture, Fishing and Forestry 37 37 Animal production 40 40 Agricultural Extension, Research, and Other Support 7 7 Activities Public Administration Public administration - Agriculture, fishing and 6 6 forestry Industry, Trade and Services Agricultural markets, commercialization and agri- 10 10 business Major Theme/Theme/Sub Theme Environment and Natural Resource Management Renewable Natural Resources Asset Management 15 15 Biodiversity 15 15 Landscape Management 15 15 Urban and Rural Development Rural Development 47 47 Land Administration and Management 15 15 Rural Infrastructure and service delivery 47 47 ii Rural Markets 9 9 E. Bank Staff Positions At ICR At Approval Vice President: Cyril E. Muller Philippe H. Le Houerou Country Director: Mercy Miyang Tembon Asad Alam Practice Julian A. Lampietti Dina Umali-Deininger Manager/Manager: Project Team Leader: Arusyak Alaverdyan Doina Petrescu ICR Team Leader: Qun Li ICR Primary Author: Qun Li F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The project development objective is to improve productivity and sustainability of pasture/livestock livelihood systems in selected communities. Revised Project Development Objectives No changes were made to the project development objectives (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Increased livestock productivity measured by milk productivity (kg/year, Indicator 1 : for cattle) Value 100% 120% 137.4% quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments (incl. % 17% higher than the target value at completion achievement) Increased livestock productivity measured by milk productivity (kg/year, Indicator 2 : for sheep) Value 100% 110% 128.7% quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments 18.7% higher than the target value at completion iii (incl. % achievement) Increased livestock productivity measured by growth rates of animals Indicator 3 : (gram/day, for cattle) Value 100% 120% 127.4% quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments 7% higher than the target value at completion (incl. % achievement) Increased livestock productivity measured by growth rates of animals Indicator 4 : (gram/day, for sheep) Value 100% 105% 127.1% quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments 22% higher than the target value at completion (incl. % achievement) Increased efficiency of communal pasture management, as measured by Indicator 5 : increased communal budgetary revenues from lease of pastures Value quantitative or 100% 130% 157% Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments (incl. % 27% higher than the target value at completion achievement) Increased sales from livestock by livestock raising households Indicator 6 : (AMD/household) Value quantitative or 100% 120% 230.3% Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments (incl. % 110% higher than the target value at completion achievement) Increased Pasture Management Effectiveness (scoring system) Indicator 7 : Value 0.00 60 points 57 points quantitative or Qualitative) iv Date achieved 04/01/2011 09/30/2016 09/30/2016 Scores were significantly improved over the life of the project, although Comments the target was not fully achieved as questions deviated from realities in (incl. % achievement) the field. (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Number of pasture management plans developed and agreed by the Indicator 1 : communities Value 0.00 46 81 (quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments 176% of target value achieved at completion (incl. % achievement) Areas of pastures and grasslands leased (ha) Indicator 2 : Value 100% 140% 711% (quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments Over 500% of target value was achieved as a result of increased community (incl. % participation in the project. achievement) Number of farmers associations established Indicator 3 : Value 0.00 46 91 (quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments 198% of target value achieved at completion (incl. % achievement) Percentage of winter fodder requirements met Indicator 4 : Value 45 80 90 (quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments Exceeded 10% of target value v (incl. % achievement) Adoption rate by farmers in targeted communities (%) Indicator 5 : Value 70 90 92 (quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments 2% higher than the target value at completion (incl. % achievement) Improved outreach and performance as measured by increased share of Indicator 6 : revenue from contracts (%) Value 6 10 19 (quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments (incl. % 9% higher than targeted value achievement) No. of trained and certified community veterinarians providing services Indicator 7 : Value 0.00 48 67 (quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments (incl. % 140% of target value achieved at the project completion achievement) Percentage of grants completed with satisfactory rating Indicator 8 : Value 0.00 80 100 (quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 Comments (incl. % 20% higher than the target value achieved at completion achievement) No. non-recipients adopting similar technical innovations outside the grant Indicator 9: scheme Value 0.00 250 267 (quantitative or Qualitative) Date achieved 04/01/2011 09/30/2016 09/30/2016 vi 107% of the target value achieved at the project completion. 267 refers to Comments replication of similar projects through technology transfer activities. Total (incl. % achievement) number of indirect beneficiaries outside the grant scheme was 33475. Clients who have adopted an improved agriculture technology promoted Indicator 10: by the project Value 0.00 774 888 quantitative or Qualitative) Date achieved 08/13/2012 09/30/2016 09/30/2016 Comments 115% of target value achieved at completion (incl. % achievement) Clients who adopted an improved agriculture technology promoted by Indicator 11: project - female Value 0.00 220 257 quantitative or Qualitative) Date achieved 08/13/2012 09/30/2016 09/30/2016 Comments 117% of target value achieved at completion (incl. % achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 04/28/2011 Satisfactory Satisfactory 0.00 2 11/29/2011 Satisfactory Satisfactory 1.56 3 05/26/2012 Moderately Satisfactory Moderately Satisfactory 2.55 4 12/24/2012 Satisfactory Satisfactory 6.08 5 06/08/2013 Satisfactory Satisfactory 7.09 6 12/28/2013 Satisfactory Satisfactory 8.85 7 06/30/2014 Satisfactory Satisfactory 10.20 8 11/07/2014 Satisfactory Satisfactory 11.98 9 05/03/2015 Satisfactory Satisfactory 12.99 10 11/08/2015 Satisfactory Satisfactory 14.29 11 05/31/2016 Satisfactory Satisfactory 15.17 12 08/16/2016 Satisfactory Satisfactory 15.43 vii H. Restructuring (if any) Not Applicable I. Disbursement Profile viii 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Agriculture makes up about one fourth of the country’s Gross Domestic Product (GDP), and is the main source of employment and livelihoods for rural communities in Armenia, which make up almost 40 percent of total population. Armenia is a mountainous country with about 60 percent of the agricultural land being used as pasture for growing livestock which is essential for rural livelihoods. Despite having access to an extensive pasture lands for animal grazing, unorganized grazing practices led to land degradation and reduced pasture productivity. About 19 percent of pasture land in close vicinity to the livestock farmers’ villages was overgrazed while the rest was underutilized because of distance and access. This had led, on one hand, to degradation and erosion of nearby pastures, and on the other to underutilization of remote pastures, resulting in development of bushes, small trees and other inedible species. The cumulative impact of uncontrolled and unmanaged exploitation practiced over the last two decades led to clearly visible resource destruction around most mountainous villages in Armenia. 2. The degradation of pastures was found to be a major contributing factor to the contraction of Armenia’s livestock sector, undermining its role as one of the key sources of economic growth and rural livelihoods. This was attributed to the combination of: (i) the failure of pasture management to adapt to the new post-Soviet reality of small farm production that led to transfer of livestock ownership from defunct publicly-owned enterprises to family farms; (ii) the lack of effective public service (e.g. veterinary and extension) delivery in remote mountain locations to facilitate agriculture modernization; and (iii) the inability of small farms to participate in modern-day supply chains with consequent unwillingness of these farmers to pay for necessary inputs and support services for improved animal feeding schemes and more sustainable methods of pasture management. 3. The global financial and economic crisis of 2009 dealt a stark blow to Armenia’s economy in general and to small-scale farmers in particular. Driven by a 35 percent fall in remittances the economy contracted by 14.4 percent, the fiscal deficit reached 7.0 percent of GDP and poverty rose by nearly 3.0 percentage points. For the poorest rural people in mountainous communities - representing roughly one-third of all rural households in Armenia - where off-farm employment opportunities were scarce and subsistence agriculture was often combined with seasonal labor migration to the Russian Federation, the (under-performing) livestock sector became the main source of livelihood. Local communities were increasingly unable to provide basic services because of difficulties in collecting taxes from poor households. In response, the Government of Armenia (GoA) made a bold policy decision to decentralize pasture management and give the communities full legal rights to manage their own pasture areas, including collection and full ownership of pasture usage fees and rents. Due to severe fiscal constraints, the GoA was not able to provide the necessary investments, training and other support to enable the communities to manage these resources effectively. Project preparation was initiated against this backdrop 1 of an urgent need to link two broad objectives - strengthening economic livelihoods of poor villagers while reversing pasture degradation - despite the lack of necessary services and institutions and with the added need to change the mindset of rural households and communities towards stronger self-reliance. 1.2 Original Project Development Objectives and Key Indicators 4. According to the Financing Agreement (p. 5), the project development objective (PDO) was to improve productivity and sustainability of pasture/livestock livelihood systems in selected communities. 5. The key PDO indicators were: (i) increased livestock productivity as measured by milk productivity and increase in daily animal weight gain; (ii) increased efficiency of communal pasture management, as measured by increased communal budgetary revenues from lease of pastures; (iii) increased farm sales from livestock; and, (iv) increased Pasture management effectiveness. 1.3 Revised PDO 6. The PDO and Key Indicators remained unchanged. 1.4 Main Beneficiaries 7. The project originally targeted about “54 1 mountain village communities,” with a total population of around 78,000. The beneficiary population specifically included women (about 38,000 women) and youth. These were mainly small and medium scale farmers and herders located in six of the poorest Marzes (regions) of Armenia. 1.5 Original Components 8. Component 1: Community Pasture/Livestock Management System (Appraisal: US$15.36 million; Actual at EOP: US$17.88 million). This Component aimed to introduce efficient and sustainable community-managed pasture/fodder-based livestock production systems in selected mountainous communities, where livestock was the main source of livelihood and communities had expressed a strong interest in improving their pasture production. This required reversing the trend of destructive grazing, implementing more efficient pasture use, improving systems of fodder production and animal feeding, and raising the efficiency of animal production. Component 1 was comprised of: (i) Development of Community Pasture/Livestock Management Plans (US$2.30 million); and (ii) Community Fund for Implementation of Pasture/Livestock Management Plans (US$13.06 million). 1 Subsequently to project approval and before the actual project implementation, the Government of Armenia updated the project target to 55 communities. 2 9. Component 2: Strengthening Support Services (Appraisal: US$2.48 million; Actual at EOP: US$2.35 million). This Component aimed to increase livestock productivity and pasture health by improving the supporting services for farmers involved in livestock production. This was to be achieved by providing support to improve: (i) agricultural advisory services in livestock-related topics; and (ii) community animal health services. 10. Component 3: Competitive Grants Program (Appraisal: US$2.05 million; Actual at EOP: US$2.09 million). This Component aimed to increase sales from livestock and natural resources through support to village-level agri-businesses and farmer’s groups to develop new business opportunities, improve marketing, promote food safety practices, and introduce and demonstrate new technologies that could benefit communities focused on livestock production. 11. Component 4: Project Management and Monitoring and Evaluation (Appraisal: US$1.45 million; Actual at EOP: US$1.2 million). The project was managed by the same implementation agency - Agriculture Projects Implementation Unit (APIU) under the Ministry of Agriculture (MoA) that implemented the previous World Bank (WB) supported projects - Rural Enterprise and Small-scale Commercial Agriculture Development (RESCAD) Project and the Avian Influenza Preparedness (AIP) Project. This component was to finance (i) project management and training, including annual operational reviews and audits; and (ii) monitoring and evaluation. 1.6 Revised Components 12. The project components remained unchanged. 1.7 Other significant changes 13. There were no significant changes in the project context, objectives and design, although there were revisions made to sub-activities. The APIU succeeded in saving approximately US$600,000, which was productively routed into activities under Component 1 increasing the number of supported communities to 81 instead of the planned 54. All extended activities were agreed with the WB Project Task Team Leader (TTL) and the GoA. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 14. The project design included interventions aimed at the creation of Pasture User Associations (PUAs) on the basis of the Law on Consumer Cooperatives 2, ensuring access to reliable veterinary and agriculture extension services, new technologies, knowledge, 2 Because PUAs were established on the basis of the existing Law on Consumer Cooperatives, they were sometimes referred to as Pasture Users Consumer Cooperatives (PUCCs). 3 markets and market information for rural households in targeted locations. To encourage PUAs to adopt new animal husbandry practices enabling more sustainable use and management of pasture resources, the project introduced Community Pasture Management and Livestock Development Plans (CPMLDPs) – both a holistic and individually-tailored livestock development plans inclusive of all community members. Each PUA established the Community Pasture Management and Livestock Development Committee (CPMLDC) which assumed full responsibility for pasture management through implementation of the respective CPMLDPs. 15. The project design benefitted from institutional studies which identified governance arrangements that were broadly acceptable to small communities. It also took elements of successful experience from other CIS countries and China. Even during project preparation, the team sought to include the whole range of households in decision- making, and within them the women as well as the men. The team also had to convince Armenian policy makers that the PUAs would in fact increase community motivation and would constitute an improvement over the current situation in terms of stronger accountability to the livestock raising communities. 16. As the PUA concept was new to Armenia, the project supported development of adequate capacity of both the APIU and Marz Support Teams (MST) to supply extensive hands-on technical assistance to dozens of newly created PUAs. The APIU prepared a detailed Operational Manual (OM) for the Component 1, which included a clear description of ways to introduce community-based management arrangements to both save the resources and help accelerate the GoA decentralization reform process. 2.2 Implementation Key Factors Affecting Implementation and Outcome 17. The project was declared effective in July 2011, and closed as scheduled with full disbursement in September 2016. At project completion, all activities/components and output results identified at project appraisal were successfully completed or exceeded, and the overall project implementation was rated Satisfactory. 18. The Mid-Term Review was conducted in July 2014. As a result of savings generated through efficiency in project management, instead of the 54 communities planned at the beginning, 81 communities were engaged under the project. 19. To strengthen the financial sustainability of PUAs, the project design foresaw signing of a Lease Agreement between the village council and PUAs. According to this agreement, PUA would collect pasture use fees and allocate them to the following uses: (i) payment of the rental income to the community council as per Lease Agreement; (ii) investments in pasture and livestock production infrastructure; and (iii) expenditure to maintain earlier capital investments in pasture improvements. In 2012 the GoA adopted a new Law on Pastures which not only instituted the requirement of pasture use fees collection but also made the village councils directly responsible for collection. 4 20. The project benefited from close coordination and collaboration with a number of non-governmental organizations with positive impact on the pace of project implementation, stronger social cohesion within project communities and additional technical support. All PUAs were supported by Heifer International (HI) through a separate HI grant program, which financed 30 percent of the machinery costs for procurement of agricultural equipment and machinery planned under Component 1. HI’s assistance and contribution also included hand-holding technical support to PUAs. Similarly, under Component 2 the project collaborated with a local foundation named “Center for Agribusiness and Rural Development” (CARD) for more effective management and use of Veterinary Support Centers (VSCs). The project design foresaw that VSCs would be managed by cooperatives of local vet associations, however during the project implementation it was found that CARD was a more appropriate choice for taking responsibility in effective management of these centers. 21. By selecting communities that demonstrated a willingness to engage with the project and an interest in improving livestock farming, the APIU was able to accelerate implementation and realization of project benefits. Shortly after project approval, APIU drew up a long list of target communities with livestock farming as the main economic occupation and with strong interest in improving livestock/fodder production. The communities on this list had to conform to the selection criteria enumerated in the Component’s Operational Manual (e.g. number of livestock, pasture area and size of village population). In addition, during the selection process the APIU also paid careful attention to the level of interest, self-organization and the reputation of the community and community leaders demonstrated during the initial project launch workshops and social mobilization meetings as well as to the strength of community commitment to the project goals. For example, during the early stages of the project implementation, the APIU had to exclude 10 communities that failed to provide their share of project co-financing despite making a commitment to do so when joining the project. 2.3 Monitoring and Evaluation Design, Implementation and Utilization 22. Overall assessment: The quality of the project’s Monitoring and Evaluation (M&E) system was satisfactory. 23. M&E Design: A comprehensive M&E framework and database were designed and set up to keep track of implementation progress and performance of all project activities. M&E outcome and output result indicators were designed appropriately and this enabled project management to monitor and evaluate implementation status and progress, taking corrective measures towards the achievement of the PDO. These included, field supervision of the compliance of works, various surveys and data collection, progress reporting, final evaluations to document results and outputs of project implementation. 5 24. One of the M&E instruments applied by the project was the Management Effectiveness Tracking Tool (METT) 3 which was introduced during project implementation, and served as the primary source of information for the measurement of pasture management effectiveness (one of the PDO outcome indicators) and monitored and evaluated PUAs performance. Separate reporting systems were applied to the Competitive Grants Program (CGP) and the veterinary and extension components. A project website was established to include the M&E results for all project implementation outputs and outcomes. 25. M&E Implementation: Relevant project baseline data were collected at the beginning of project implementation, and the M&E framework (initiatives)/METT scoring system proposed under the project were effectively applied. All monitoring data/information was regularly entered into a database and compiled for further use. The APIU was able to regularly access field level data for the preparation of periodic project monitoring reports and impact studies. The main sources of data on pasture and livestock were reports by community mayors, and data collected directly by MSTs. The METT application was very useful for assessing performance of the PUAs, although some of the METT questions could have been better adapted to local field conditions. Respective lessons learned have been applied to the METT application in CARMAC 2 Project. 26. A series of beneficiary assessment and evaluation studies, many of which were gender disaggregated, were carried out in partnership with NGOs/research institutions and consultants to record and analyze the project implementation output and outcome results and impacts for all project components, during and at the end of project implementation. The good practices and lessons learned were identified and recorded to facilitate wide dissemination at community, regional and national levels (see details in Annex 5 and 6). 27. Utilization of the M&E results contributed to identifying key implementation issues, recommending adequate remedial actions, and enhancing the physical and financial implementation progress. The M&E data was evaluated and used to inform decision- making and for resource allocation, such as reallocating the cost saving from component 3 to expand the number of communities in the pasture management activities in component 1. In addition, the implementation performance of PUAs was closely monitored and 3 The Management Effectiveness Tracking Tool (METT) was originally developed by the WWF and the WB to track and monitor progress towards worldwide protected area management effectiveness (see: Reporting Progress in Protected Areas - A Site-Level Management Effectiveness Tracking Tool, WB/WWF Alliance for Forest Conservation and Sustainable Use, 2003). Following the principles and methodology this tool has been modified and adapted to track progress of management effectiveness of pasture and livestock management systems under the CARMAC project. The METT is based on the idea that many outcomes of good pasture management can only be observed and measured in a relatively long timeframe. However, good management principles likely leading to the desired outcomes could be well monitored and tracked starting from an early stage and on a regular basis within the project’s implementation period. Thereby good pasture and livestock management would follow a process that has six distinct stages, or elements including: a) understanding the context of existing management objectives and threats, b) progresses of appropriate planning, c) allocation of sufficient resources (inputs), d) application of management actions (processes), e) provision of products and services (outputs), and f) eventually results in terms of impacts or outcomes. 6 assessed by the METT scoring system (see details in section 3.3). Although the results showed a continuous improvement of management capacity and organizational coherence of PUAs in all aspects, any shortcomings in specific areas (e.g. if the PUA results showed “no grazing and livestock records are being kept”) were identified and corrected through immediate remedial actions (e.g. additional technical/capacity training provided by APIU experts). The effective use of the METT and M&E results contributed to and confirmed that the provisions of the CPMLDPs were fully implemented and PUAs adopted the planning process and continued developing their own rotational grazing regime following the principles of the original plans. The METT and M&E system has also been used very effectively in CARMAC 2 Project, combined with introduction of the Management Information System (MIS) system, which has further strengthened APIU’s M&E capacity. 2.4 Safeguard and Fiduciary Compliance Environmental Safeguards 28. The project triggered OP/BP 4.01 Environmental Assessment, and was classified as environmental Category B. OP 4.09 Pest Management was also triggered, because crop growing activities to be financed from the project could imply application of pesticides. A framework Environmental Management Plan (EMP) was developed to guide application of environmental safeguards under all components of the project as required. The EMP included detailed instructions on the development of investment-specific EMPs and Pest Management Plans, which might be required due to the nature of individual investments, and was disclosed and discussed with relevant stakeholders prior to tendering of civil works. 29. Environmental compliance under CARMAC project was satisfactory throughout the implementation period. Recommendations on the on-site management and final disposal of liquids and solid organic waste as well as of the construction water were provided case-by-case, with the purpose of achieving the best feasible outcome in the given circumstances. Close monitoring by APIU and supervision by the WB Task Team was carried out, which kept contractors and CGP beneficiaries compliant with health and safety rules and the prescribed waste management practices. 30. Civil works under Component 1 and 2 comprised some small-scale works (e.g. the animal stock watering points and small houses to provide premises for VSCs). Consultation meetings were held to discuss specific EMPs with the affected communities. All permits required for the construction, provision of utilities, and waste disposal are provided. All EMPs were disclosed on the APIU website (www.arspiu.com). APIU filed documentation on the technical supervision of works, including environmental monitoring and the required permits. 31. Overall, no measurable environmental damage was caused by the implementation of the project and it did not result in any negative impacts on the public health. Investments into the rehabilitation of degraded pastures are likely to yield positive environmental outcome in the medium term perspective through the decrease of soil erosion and the restoration of the natural habitat. 7 Social Safeguards 32. A grievance redress mechanism (including a village-based grievance focal point, along with a designated member of the APIU) was established and implemented, and community CPMLDCs were responsible for resolving complaints and redressing grievances. The system closely monitored and supported community-based participatory planning and decision-making to ensure equitable access to project benefits. The project also promoted social inclusion, especially for women and youth. Throughout the process of mobilizing PUAs, the MST placed a strong emphasis on the inclusion of female and youth household members and female headed households; 150 women and 220 youth worked at the PUA’s administrative level. 33. The project did not originally trigger OP 4.12 on Involuntary Resettlement. In the course of the project implementation, missions discovered that the construction of communal animal water points under Component 1 might require use of private lands. A protocol was established for consulting and negotiating with potentially affected private owners, who are also community members and beneficiaries of the project. Agreements were reached with them on willing buyer-willing seller basis. Because of these instances, compliance with OP 4.12 was monitored more intensively in the second half of the project. In all cases of private land use, the owners were fully informed about the activity, participation was voluntary, and there were no instances of land acquisition. 34. Overall the APIU provided regular social progress reports to the WB and ensured regular site visits. This substantially improved the understanding of communities of social safeguard policies. Based on post-evaluation of the social assessment and management of the project, the compliance with social safeguards policy was satisfactory. Fiduciary Compliance 35. Procurement. Procurement planning and procedures were in compliance with the WB policies. Project procurement was carried out in accordance with the WB guidelines and provisions stipulated in the project Financing Agreement . The APIU’s procurement staff were very responsive and provided all procurement support for (i) procurement of the civil works in all the project villages; (ii) supply of agriculture equipment/ machinery, tractors for the selected communities; (iii) construction of four VSCs (including the contracts for design and technical supervision); as well as for equipping of those centers. In addition, procurement during implementation was reviewed on a yearly basis and agreed plans of action after each review mission were followed, to correct omissions and findings accordingly. Thirteen contracts were reviewed and both Compliance & Performance Risk Ratings remained unchanged, i.e. “Moderate”. Financial Management and Disbursement 36. Financial Management: Financial Management (FM) arrangements at the APIU, including planning and budgeting, accounting, financial reporting, external audit, funds flow, internal controls, and organization and staffing arrangements were overall adequate and satisfactory to the WB throughout the project implementation period. A clear funds 8 flow mechanism with adequate reporting arrangements and controls were established, which were described in the project Financial Manual. Risks related to financial management were properly mitigated/resolved. The project’s ISR FM rating was Satisfactory during majority of the project duration. 37. IFRs and External Audits. Project interim un-audited financial reports, annual external audit and regular supervisions also helped to monitor compliance. The project quarterly interim un-audited financial reports (IFR) prepared by APIU were always received on time and found to be acceptable to the Bank. The auditors issued unmodified (clean) opinions on the Project’s annual financial statements, which were received by the due date. The Recipient complied with the public disclosure requirement for the audited financial statement. 38. Disbursement: All disbursement procedures were in compliance with the WB disbursement guidelines. WB funds were almost fully disbursed by the project closing date, with a small undisbursed balance (about 1 percent of the total SDR funds) due largely to exchange rate fluctuations. 39. Other. During a post review mission in 2014, the WB procurement team, identified an amount equivalent to US$145,205.05 paid under contracts signed with “Small and Medium Entrepreneurship Development National Center” Foundation (SME DNC) under the project as an ineligible expenditure. The issue did not relate to the FM arrangements in place at the APIU, but to the status of SME DNC (which receives state budget support), and to the fact the MoA had a representative in the Board of Trustees of SME DNC. As per the WB Procurement Guidelines this could constitute a conflict of interest. As a result of consultation with the GoA, the respective amount was remitted into the project’s Designated Account. 2.5 Post-completion Operation/Next Phase 40. The project achievements offer both a clear demonstration of the technical and institutional measures for vastly superior community-based pasture management and livestock development practices as well as a successful path for scaling up of these measures throughout Armenia. The project provided a significant opportunity at the current stage of the country’s agriculture sector transformation. Moreover, the project led to changes in national policy and legislation such as the new Law on Pastures of 2012 and the Law on Agricultural Cooperatives in 2015. 41. MoA and APIU gained valuable experience which was critical in scaling up the project activities under the follow-on project (CARMAC 2) which became effective in early 2015. This new project not only continues to support sustainable pasture management and livestock development at community level, but also addresses key constraints in increasing market access, supporting the commercialization process, and enhancing employment opportunities, including a new value chain component linking livestock and other agriculture producers with markets. The MoA is establishing a natural resources monitoring system to track the long-term impact of improved management 9 measures, including impacts on productivity, biodiversity and carbon sequestration. This would include using Information Technology (IT) and Geographic Information System (GIS) based monitoring and record keeping which is supported by the WB under the recently approved Agriculture Policy Monitoring and Evaluation Capacity Building Project (P158359). 42. The transition to post-completion operation of investments financed under the individual project components include: a) Collection of pasture use fee was mainstreamed nationwide by this 2012 Law on Pastures. Communities applied and collected pasture use fees in excess of estimated annual operation and maintenance (O&M) cost requirements for the community pasture management activities. b) Responsibility for management of VSCs was transferred to CARD, a well-run Armenian foundation sustainable with a successful track record in management of similar initiatives for the benefit of agricultural producers; and c) Individual impact assessment of each CGP beneficiary showed that businesses supported by Component 3 were generating financially sustainable and commercially attractive margins. 43. It is recommended that for the CARMAC 2 project APIU establish triennial updates of CPMLDPs based on professional inputs. The on-going project gives sufficient time and resources to embed this practice nationwide. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 44. Relevance of Objectives (High). The project’s objectives, as reflected in the PDO statement in the Financing Agreement, are comprised of three key outcomes: (1) improved livestock productivity; (2) more sustainable pasture management systems; and (3) improved livelihoods in selected communities. All three elements of the PDO remained highly relevant to the World Bank Country Partnership Strategy (CPS) for Armenia at the time of closing, the FY14-FY17 CPS. Under Engagement Area 1.2: Rural economy sustainably improved, it highlights that “the focus of the program is on boosting incomes of the poorest segments of the rural population through strengthening agricultural productivity…” (p. 25). It goes on to state: “Given the importance of the agriculture sector for employment and livelihoods in the rural areas, it will be a priority for the Bank's investment support…. The ongoing CARMAC Project [and] a follow-up CARMAC-2 project…are all instruments that focus on strengthening agricultural productivity and improving the welfare of the rural population” (p. 25). The FY14-FY17 CPS Results Framework also has a goal under Engagement Area 2 of “Improved management of land and pastures in the agricultural sector” (p. 38). Thus, all three of CARMAC’s PDO outcomes remained of significant importance in the CPS, just as they were at the time of project appraisal. 10 45. Relevance of the Design and Implementation (Substantial) . All project activities and components, and the overall program logic, were clearly designed to contribute to the achievement of the planned outputs and the expected PDO-level outcomes. This clear program logic was reflected in the robust Results Framework, which also included relevant indicators at each level in the results chain. A summary of the relevance of the design of project components to the achievement of the three key PDO-level outcomes is as follows: (i) Improvements in livestock productivity were designed to be supported by strengthening agricultural advisory services and community animal health services under Component 2, along with improved infrastructure, machinery, and stock breeding for increased fodder, pasture, and livestock production under Component 1. (ii) More sustainable pasture management systems were designed to be supported by developing community pasture/livestock management systems through the introduction and implementation of pasture/livestock management plans, and the establishment of CPMLDCs and PUAs for the effective institutional management of pasture and livestock resources under Component 1. (iii) Improved livelihoods were designed to be supported by increasing farmer incomes as a result of project activities to increase livestock productivity described under outcome (i), combined with some of the livestock product marketing activities financed by the Component 3 competitive grants. Diversification of incomes was designed to be supported by the development of new, non-livestock business opportunities that were also supported by the Component 3 competitive grants. 3.2 Achievement of Project Development Objectives Rating: Substantial 46. Achievement of the PDO is rated substantial. By closing, the project had achieved all three of the key PDO-level outcomes described in Section 3.1, through successful implementation of the relevant contributing project activities, as discussed below (reinforced by the information presented in the Datasheet and Annex 2): PDO 1: Improve Livestock Productivity (Substantial) 47. The expected PDO outcome on increased livestock productivity was measured by increased milk productivity and the daily animal weight gain. According to the project M&E results at project completion, milk production had increased from 1,428 kg/year at the project baseline to 1,964 kg/year in 2016 for cattle (17 percent higher than the PAD target), and from 66 kg to 85 kg/year for sheep (18.7 percent higher than the PAD target). The livestock/animal productivity (meat production) measured by growth rates (gram/day) of cattle and sheep increased from 320 to 408 gram/day for cattle (7 percent higher than the PAD target), and 81 to 103 gram/day for sheep (22 percent higher than the PAD target) respectively. 11 48. Per the project design, the increased livestock productivity was achieved mainly due to: (a) the introduction and implementation of efficient and sustainable community- managed pasture/fodder-based livestock production systems, which reversed the trend of destructive grazing, providing more efficient pasture use and quality fodder availability (the percentage of winter fodder requirements met was 112 percent of the PAD target). Other factors included improved systems of animal feed production, infrastructure for animal drinking water sites (100 percent of the PAD target), access roads to remote pasture areas (100 percent of the PAD target), and use of pasture management plans (150 percent of the PAD target); (b) under Component 2, the improved regional and local technical advisory systems, which increased adoption rates of new technologies by farmers in targeted communities by 92 percent (compared to the target rate of 90 percent). The new veterinary service centers (100 percent of target numbers), with additional trained and certified community veterinarians (139 percent of the target), improved herd parameters such as calving rates, mortality, milk quality, etc.; and (c) under Component 3, 69 small grants projects were completed (98 percent of target value), including support for the development of livestock breeding. All of these intermediate outcomes and outputs significantly contributed to increased livestock productivity in all project community areas, exceeding the PDO 1 targets. 49. There is strong evidence that the impressive gains in livestock productivity enjoyed by project beneficiaries are attributable to the CARMAC Project. As illustrated in Figure 2, Annex 3, the project worked with disadvantaged farmers who initially suffered from much lower levels of productivity than the national average. However, during project implementation, the CARMAC farmers experienced much higher levels of growth in cow milk yield than the national average, and also much faster than the without-project scenario. PDO 2: More Sustainable Pasture Management Systems (Substantial) 50. The PDO outcome for increased sustainability of communal pasture management was achieved through the improved institutional, technical and financial sustainability of PUAs at community level. This was measured in part by the PDO outcome indicator of increased communal budgetary revenues from the lease of pastures. The M&E data indicate that at project completion, the collection of pasture fees transferred to PUAs’ budget accounts was, on average, AMD 328,893 per community in 2012, and increased to AMD 561,395 per community in 2016. That exceeded the PAD outcome target by 27 percent. The operational strength and management capacity of the PUAs were also repeatedly assessed through the METT scoring system, which indicated a significant and continuous improvement of management capacity and organizational coherence of the PUAs in all aspects. 51. The specific institutional arrangement for community/cooperative based pasture management was widely established: 90 PUAs were established in 81 communities (150 percent of the target value); comprehensive CPMLDPs were developed and implemented in all those communities which covered 176,000 ha of pasture (18 percent of Armenia’s pasture area); and a sustainable regional and local technical advisory and veterinary service 12 system was developed and strengthened (for details see Annex 2, outputs for Component 2). 52. The technical interventions under these components increased pasture and fodder production in several ways, including: (i) improved access to distant pastures, thus allowing farmers to graze larger areas of pasture; (ii) increased area that could be used for hay production, as livestock could now be grazed on distant pastures; (iii) gradually increased pasture productivity as swards recovered from over-grazing; and (iv) allowed farmers to harvest more hay from larger areas, due to the provision of additional machinery. 53. The financial sustainability of the cooperative pasture management system was built up through the fee-based community services and diverse income resources (e.g. the agriculture machinery and extension services, pasture land lease, development of the value chain business opportunity, etc.). All service fees were calculated and collected to cover the full O&M costs and depreciation of the machinery. The institutional sustainability of the pasture management system was improved through the significant amount of technical, financial and institutional training on the effective implementation of the CPMLDC and PUA management programs, which were provided for community mayors, chairmen, accountants, farm households, and dispute resolution persons of PUAs (11,677 cooperative members trained, 100 percent of the target value). PDO 3: Improved Livelihoods (Substantial) 54. Livelihoods of rural residents were improved under the project in two ways: (i) by increasing farmers’ incomes from the sale of livestock products; and (ii) by providing grants to support activities to diversify income sources under Component 3. 55. Under CARMAC, livestock product revenues (including milk) increased from an average of AMD 532,147 per household in 2011 to AMD 1,226,226 in 2016. This represents an impressive increase of 112 percent over the PAD target. In addition to increased livestock productivity, it was achieved mainly due to the change in production type from subsistence to commercial for many farm households. One important factor was that winter fodder production increased from 45 to 90 percent of the requirement. 56. The competitive grants program under Component 3 funded proposals from village-level agri-business and farmer groups to introduce innovative income-generating activities. A total of 69 grants (100 percent of the target) were funded and fully completed. Of these, 24 were related to a broad array of livestock production and processing themes (including cattle, pigs, sheep, and poultry), while the remaining grants financed production and processing of higher value fruits and vegetables and other non-traditional commodities like rabbits, chinchillas, fish and honey. All projects helped create viable business opportunities, and provided significant income gains to the beneficiaries ranging from 5 to 60 percent, and increased employment opportunities from 2 to 20 people under each project. 13 57. Overall, a field survey indicated that 80 percent of project farmers had an increase in farm incomes (for details see Annex 3). According to the survey results, farm household income from livestock production had increased by about USD 1,000 on average. 3.3 Efficiency (Substantial) 58. The rate of return of project investments at closing was assessed to be higher than what was expected during preparation, and is also impressive in absolute terms, so project efficiency is rated Substantial. At the time of appraisal, a detailed economic rate of return (ERR) analysis was carried out, focused on the benefits and costs of the largest components of the project—Components 1 and 2—related to the expected increases in production of livestock products (primarily milk and meat), and the additional net income that they were expected to generate. The cost basis for these calculations was the total cost of investments under Components 1 and 2. Using fairly conservative assumptions, it was estimated that an ERR of 83.1 percent could be achieved (PAD, p. 13). 59. The ERR analysis was repeated at the time of project closing. A combination of project data and national statistics was used to estimate the difference between “with- project” and “without-project” scenarios. A major achievement of the project was to help physical livestock output grow at more than twice the national rate: 59 percent versus 24 percent. The total value of project villages’ incremental livestock output (based on the product of animal numbers, output per head, and constant 2011 prices) was used to calculate changes in beneficiaries’ incomes. While the project significantly out-performed the projections of the ex-ante model with respect to livestock numbers and liveweight gain, it fell well short of the highly optimistic projections for milk yield. Combined, the average annual benefit was AMD 7.2 billion (US$14.9 million) over 2015-16. At the same time, total expenditures on Components 1 and 2 over the life of the project was US$21.4 million. If the annual benefit of US$14.9 million can be sustained until 2025, the ERR from these two components will be 116 percent, which exceeds the 83.1 percent ERR projected at project appraisal (see Annex 3 for more details). 60. It is worth considering how the high level of economic return was reached, particularly given a drop in both cattle and sheep prices of about 50 percent over the life of the project. The key is in the lower volume of purchased inputs. In comparison to typical livestock projects, the driving factors here were very little purchased feed and no exotic breeding stock. Instead, the main input here—the mountain pastures—already existed but was poorly managed and used. 61. For the other, smaller component of the project—the Competitive Grants Program (CGP) under Component 3—a separate economic analysis was carried out at appraisal. This component was not as amenable to traditional ERR calculations as the first two components because it was demand-driven, so the actual investments and their benefits were impossible to know ex ante. Instead, the analysis at appraisal was based on the experience of an earlier project that supported a similar CGP in Armenia, the RESCAD Project. Ex-post economic analysis of that project’s investments yielded a rate of return of 104 percent. For CARMAC, the ex post rate of return of the CGP investments was assessed 14 on the basis of visits to all 69 sub-projects to collect data on the change in incomes of beneficiaries. The average rate of return calculated for these sub-projects was 17.8 percent, which is lower than expected, but still respectable. At the same time, it should be noted that the assessment visits were conducted only 12-18 months from the beginning of the investments, and experience shows that incomes are likely to grow further as implementation progresses. The weighted average ERR of all three investment components (weighted by the cost of each component) is 107 percent, which is quite high, and substantially higher than the weighted average of the ERRs expected at appraisal of 85 percent. 62. Effective organization of the CARMAC Project implementation contributed to cost-effective use of project funding. The project also benefited from timely co-financing of beneficiaries, as well as from productive partnership initiatives with HI, CARD, and commercial lenders. Project implementation arrangements made allowance for community-based procurement, which resulted in considerable savings used for expanding the number of beneficiary communities and enhancing the overall project impact. 3.4 Justification of Overall Outcome Rating Rating: Satisfactory 63. Overall, the Outcome Rating of the project is Satisfactory. All three key aspects of the PDO continue to be highly relevant to the World Bank-Armenia CPS at closing. The project design was substantially relevant, with a clear program logic and high-quality Results Framework that helped ensure that successful implementation of project activities was likely to lead to achievement of the intended outcomes. Achievement of the Project Development Objectives was substantial, with all three key outcomes being met or exceeded. Finally, efficiency is substantial, with a very high economic return from the preponderance of project investments under Components 1 and 2, a respectable ERR for the smaller Component 3, and a weighted average ERR that exceeds what was expected at appraisal. Taken together, an Outcome Rating of Satisfactory is justified. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 64. Although the project did not explicitly address poverty reduction, by targeting resources (mountain pastures) and assets (ruminant livestock) which traditionally are the preserve of the poorer members of Armenian society, it did have a significant poverty impact and probably should serve as a model in this regard. 65. Inclusion of social safeguards activities in the project contributed to progress in gender development at the community level by tailoring interventions to include women. These activities encouraged women to assume decision making roles in the administration of PUAs and CPMLDCs, causing a change in the mindset of community members with regards to women’s leadership roles. This approach was also practical, since men were often away doing seasonal or migratory work and women were present and able to assume more formal roles as decision-makers in the community. 15 66. The number of female beneficiaries in all project activities was tracked consistently by the APIU. Between 2011 and 2016 under Component 1 there were approximately 133,000 beneficiaries, of which 62,500 were women; under Component 2 there were 55,932 beneficiaries, of which 21,639 were women; and under Component 3 a minimum of two women formed part of every grant selection committee. The number of women and youth-run beneficiary enterprises was as follows: (i) the number of CGP Workshop participants for all seven rounds was 4,300, of which 472 were women and youth; (ii) among 69 winner subprojects, 16 were managed by women; (iii) 259 permanent jobs created, of which 53 were for women; and (iv) the total number of direct and indirect beneficiaries of the CGP was about 33,491, of which 5,429 were women. Overall, the total number of direct and indirect beneficiaries of CARMAC project was about 222,368, of which 89,568 were women (or approximately 40 percent). 67. Attention was also paid to youth inclusion in the project. Youth were provided additional consultations especially at CGP subprojects workshops in order to encourage them to apply for subprojects. Out of about 4,300 participants in CGP workshops, there were about 400 youth participants and five CGP subprojects were implemented by youth. The role of youth in the cooperative decision making processes, youth inclusion and empowerment activities were carried out through various capacity building activities, including project implementation meetings, project technical, financial and institutional training, etc. at the community level. As a result of all these activities, about 4,000 youth participated in PUAs (out of 11,685 members), of which about 220 youth worked at the PUA administrative level (out of about 589 members) and about 160 worked in CPMLDCs (out of about 503 members). (b) Institutional Change/Strengthening 68. Overall, project technical, financial and institutional training plans were fully implemented and strengthened all PUAs and project implementation agencies at the regional and community levels. 69. Training of Advisors: A program of in-service training was carried out by Republican Agricultural Support Center (RASC) annually. An average of around 228 technical advisory staff (100 percent of the target value) comprising both Marz Agricultural Support Centers (MASCs) staff and community advisors participated in annual training programs from 2012 to 2016. Topics included livestock production, animal health, pasture management, fruit and vegetable production, plant protection, bee-keeping and extension management aspects. Trainers primarily comprised specialists from the RASC, The Armenia National Agrarian University (ANAU) and Scientific Centers. Course evaluations indicated that the programs had effectively improved knowledge and skills of advisors, and had enabled advisors to provide training to more than 20,000 farmers annually. In 2012 and 2013, training in economic legislation and business planning was provided for a total of 297 specialists, which increased the capacity of the MASCs to help farmers to prepare business development plans (total about 2,330 plans prepared since 2012), and provide advice on legislative matters. These refresher training courses were essential to maintain 16 and improve advisors’ knowledge and skills and enhance their capacity to serve farmers and communities. 70. Training of Farmers: Additional training programs for farmers were carried out, particularly comprising courses in livestock topics. In 2012, such training courses were conducted for 878 farmers (including 168 women) in 35 communities under the project. From 2013 the farmers training programs were carried out using the Ministry’s core funding, with more than 20,000 farmers attending seminars/workshops annually, although not only specifically in livestock topics. Training was also provided for members of PUAs under the contract with Green Lane Company (an NGO). 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 71. A field survey and assessment of the impact of the local technical advisory system was conducted. Farmers clearly valued the service as evidenced by the very positive impact assessment results with 94 percent of farmers highly valuing the advisory services; 89 percent indicating that they had adopted technological changes as a result of advisory activities; 75 percent indicating that they had an increase in production; 80 percent indicating an increase in farm incomes; and 64 percent indicating their willingness to pay for improved services. Survey results also support the notion that steps should be undertaken to address to the development of advisory services through their consolidation and joint coordination, and increasing the outreach of paid advisory services. Reflecting on the importance and positive assessment of the advisory services, the MoA prepared a strategy for continuing development formally accepted by the GoA under Order No. 1516 dated December 2013. 72. In addition, all 69 small grant projects were evaluated. These projects helped create viable business opportunities, and provided significant income gains to the beneficiaries ranging from 5 to 60 percent, and increased employment opportunities from 2 to 20 people under each project. As result of the technology transfer activities, the evaluations indicated that elements of many grants had been emulated by other beneficiaries by the end of the grant period, with further adoption likely to take place over time. Examples of innovations included improved breed selection and animal husbandry; new dairy products and more developed contracting approaches with producers; improved storage, processing, packing and marketing of fruit and vegetable products; and introduction of various non-traditional poultry and crop products. 4. Assessment of Risk to Development Outcome Rating: Moderate 73. At project completion, the risk to the development outcomes is rated moderate. There continues to be government commitment to appropriate economic, financial and sector policies as evidenced in its development strategy: Armenia Development Strategy (ADS) 2014-2025. The ADS clearly articulates the long-term development vision for Armenia: creating jobs, building human capital, promoting sustainable agricultural development, increasing rural incomes, strengthening the social protection system and modernizing public administration and governance. 17 74. Legal framework and institutional capacity risks are moderate. The recent adoption of the Law on Agricultural Cooperation in Armenia provides better conditions for sustainable project livestock production and pasture management, creating good bases for cooperation and joining efforts of farmers on the management of the production and sales. The national regulatory framework specifically related to the scope of the CARMAC project consists of several GoA decrees that regulate the rational use of pastures. The project was implemented within the existing legal framework and primarily at the local/community level, where pasture management plans were part of signed lease contracts between community authorities and PUAs. Formal registration of cooperatives, training delivered for capacity building of their members and provision of machinery and equipment should be able to mitigate the risks. In addition, the ongoing Bank-financed follow-on project—CARMAC 2—continues contributing to the maintenance and enhancement of the gains achieved under the CARMAC Project. 75. Most informal farmers’ groups were transformed into agricultural cooperatives, which not only mitigated institutional risks but also created an excellent pre-condition for future development. Risks related to the technical expertise of CGP grantees had been mitigated via involvement of consultants in sub-project planning and operation. 76. The risk for acceptance and continuation of pasture management by communities is moderate. In almost all beneficiary communities. A substantial part of the selected communities’ pasture management was delegated to newly established PUAs. These entities were interested in proper management, which ensured collection of fees for Operation and Maintenance of the pasture and the machinery possessed by the PUAs. This system is currently operational in almost all communities with improved institutional, financial and technical capacity for all farmer cooperatives for future development. Therefore, risks were rated as moderate. 5. Assessment of Bank and Recipient Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory 77. The project identified the key livestock sector issues and aligned the project objectives with country priorities and WB CPS FY09-12. It also incorporated international lessons from good practices, strengthened the focus on the development of improved productivity and sustainability of pasture/livestock livelihood systems at community level, and promoted innovative design and flexibility to adapt to the country’s needs. The right issues were targeted with good technical solutions. The detailed project implementation plan and technical operational manual were well prepared, the project risks and mitigation measures were also adequately assessed, and the project M&E system (the result framework and key monitoring indicators) and the implementation agency were well 18 arranged. The rating of the WB performance in ensuring quality at entry is considered as Satisfactory. 78. The WB project team made a tangible contribution to the implementation of the Armenia agriculture/livestock development policies, through the introduction and promotion of a novel community-based pasture management approach which was innovative not only for Armenia, but also globally. This was done by applying the WB comprehensive global good practices related to agriculture and rural development, pastures and livestock management, and participatory community development from other projects, such as the Chinas Loess Plateau Watershed Rehabilitation Project, the Kyrgyzstan Agricultural Investments and Services Project and previous WB projects in Armenia, notably the RESCAD. These introduced successful participatory community development and competitive grants procedures to this project. (b) Quality of Supervision Rating: Satisfactory 79. The quality of supervision is rated Satisfactory. The WB project team acted in an efficient and proactive way and worked closely with the project counterparts. They played a crucial role in monitoring project procurement, disbursement, auditing and verifying that the APIU and community implementation agencies had conducted business in line with the WB environmental and social safeguards and fiduciary policies. In addition, the WB responded swiftly to the government’s requests for the reallocation of funds, which allowed APIU to likewise receive timely project counterpart contributions from the GoA and agree on any adjustments which sped up the physical and financial implementation progress, for completion of all project activities by the expected project closing date. 80. The subsequent WB missions strengthened implementation outcome through improved capacity building for APIU and all PUAs, including technical training, problem solving, and knowledge exchange workshops. As the concept of community-based pasture management was relatively new to Armenia, project supervision provided extensive support to APIU in this area. 81. The WB team actively encouraged coordination and harmonization with other development agencies (e.g. USAID, UNDP, GIZ, etc.) by APIU and beneficiary communities, and established the link between the APIU and its counterparts in other Europe and Central Asia countries to gain new knowledge and experiences from the countries where similar issues existed and innovative approaches were taken. 82. It is worth noting that, although the project was managed by three different TTLs during the project preparation and implementation, these changes did not adversely impact the pace of project implementation. The WB team deserves recognition for the timely preparation of a larger parallel CARMAC 2 project during the implementation of CARMAC project. 19 (c) Justification of Rating for Overall Bank Performance Rating: Satisfactory 83. Overall Bank performance relating to the design, implementation and outcome of the project is satisfactory. The project team played a decisive role in ensuring quality at entry, and in resolving problems and identifying opportunities during supervisions. Thus, the overall Bank performance rating can be considered Satisfactory. 5.2 Recipient Performance (a) Government Performance Rating: Satisfactory 84. The GoA embraced the concept of community-based pasture management and provided strong support to the project and its design and objectives from preparation through completion. In addition, the GoA arranged the smooth coordination between national and Marz-level government administrations/implementation agencies, which facilitated the efficient implementation of the project activities/components and the exchange of information and technical knowledge on community-based pasture management. The GoA also provided in kind contribution to support the involvement of national experts in project implementation. Due to the strong project ownership, CARMAC quickly became the flagship rural development initiative of the MoA in Armenia, and the GoA decided to proceed with preparation of CARMAC 2 project before the completion of the initial project. Both projects benefited from personal involvement and support of the senior management of the MoA during the initial stages of community mobilization. Based on the achieved results, the government performance rating can be considered satisfactory. (b) Implementing Agency or Agencies Performance Rating: Satisfactory 85. The project implementing agency managed the project effectively. The APIU organized a number of ad-hoc training programs, involving representatives from all project communities, and played a key role in transfer of essential skills to local communities such as the preparation of the strategic planning, micro-investment plans and livelihoods promotion activities, which provided a strong foundation for further investment and welfare improvement at community level, especially for the poorer and marginalized communities. 86. The APIU was highly effectively in engagement of communities and community leaders. Key APIU officers were assigned responsibilities for all project-supported activities in specific districts allowing them to develop deeper understanding of the community and sector problems and gain trust of communities and other local stakeholders. All APIU staff made frequent field visits to villages and participated in community meetings in order to answer any residual questions from community members and discuss solutions for any emerging problems. As a result, the problems and shortcomings identified during supervision were expeditiously resolved. 20 87. The performance of the implementing agency was consistently satisfactory. Financial management, procurement, reimbursement, compliance with WB procedures and policies on environment and social safeguards and other fiduciary requirements were met effectively. Despite its heavy workload, the APIU introduced a number of new and promising practices related to gender and citizens’ engagement. The APIU also applied excellent M&E systems which include regular collection of household level data and innovative approaches on collection of household livestock-related expenditure. The APIU also sanctioned a number of independent third-party project impact studies (see Annex 6 of this report). Based on the above, the implementing agency performance can be considered Satisfactory. (c) Justification of Rating for Overall Recipient Performance Rating: Satisfactory 88. Overall, the Recipient was committed to the project objectives and design, and continued to provide financial and institutional support, which secured the sustainability of the established community-based pasture management and livestock development system. The project implementing agency supported effective implementation of the project activities. The project approaches are being adopted more widely in the follow-up CARMAC 2 project. Based on the achieved results, the Recipient’s performance is rated satisfactory. 6. Lessons Learned 89. The main lesson was the effectiveness of the community management approach to managing one of the classic common property resources – pastures. While this approach to coastal fisheries has been proven over the past two decades, this project’s success is important for establishing this lesson for pastures. 90. A second lesson is that recovery of a local common property natural resource can be achieved in the lifetime of a single project, and that doing so can be used to substantially augment the incomes of project participants. 91. A third lesson is that the concurrent support for both pasture management and animal health produces more rapid improvements in livelihoods. Improving pasture management while at the same time making sure that a suite of veterinary services (including artificial insemination) was delivered through the VSCs quickly enabled year- round milk production to increase, and thus a more rapid improvement in livelihoods. 92. A fourth lesson is that it is important to build flexibility into the design of community co-financing requirements. The initial design required communities to provide 50 percent co-financing for agricultural equipment and machinery. During the implementation it became clear that this was too much for the communities. While the project was fortunate that HI stepped in to make up the gap between the design and what communities could afford, greater flexibility in the initial design – as was embedded into 21 the follow-on project (CARMAC 2) – would have enabled the project to move forward rapidly even without HI’s intervention. 93. Finally, an old lesson confirmed by the success of the project was the value of project simplicity. At appraisal, several additional components and activities were considered, but were explicitly rejected to avoid adding complexity. Such decisions are difficult, but typically improve the chances of project success. Comments on Issues Raised by Recipient /Implementing Agencies/Partners (a) Recipient /implementing agencies 94. See Annex 7 for Summary of Recipient’s Comments on draft ICR. (b) Other partners and stakeholders (e.g. NGOs/private sector/civil society) 95. Almost all PUAs established within the CARMAC project were supported by Heifer International with 30 percent of co-financing for procurement of agricultural machinery and equipment planned in Component 1. HI’s assistance had a considerable impact not only in terms of financial support to PUAs, but also due to the working principles widely applied by HI and well-accepted by the farmers' owned organizations. Due to this working principle, called “Passing on the Gift” (POG), the amount of HI's contribution was "earned" by the PUAs from its commercial activities (e.g. provision of services to the farmers) and "paid back" to the same PUA for further development of its material and technical base. Thus, due to applying HI’s POG working principle the PUAs were able to collect within a very short time (from one to two agricultural seasons) more than AMD 200 million that were directed back to the PUAs for new machinery procurement, and therefore increasing the overall volume of support to the PUAs. 96. This working principle was so valuable that it was incorporated in the design of the CARMAC 2 project. To be eligible for HI funding beneficiaries only needed to submit an application and have their own 20 percent share deposited in PUAs account before receiving HI support. HI funding was based on POG approach which in this case was the condition of purchasing machinery and equipment for new farmers using 20 percent net revenues set aside from the provision of machinery services. 22 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions) Total Baseline Cost Component 1: Community Pasture/Livestock Management 15.36 17.88 116.43% System Component 2: Strengthening 2.48 2.35 94.90% Support Services Component 3: Competitive 2.05 2.09 102.00% Grants Program Component 4: Project 1.45 1.20 82.47% Management, M&E Price Contingencies Total Project Costs 21.34 23.52 110.21% Front-end fee PPF 0.00 0.00 .00 Front-end fee IBRD 0.00 0.00 .00 Total Financing Required 21.34 23.52 110.21% (b) Financing Appraisal Actual/Latest Type of Estimate Estimate Percentage of Source of Funds Cofinancing (USD (USD Appraisal millions) millions) Recipient 5.33 8.27 155.16 International Development 16.00 15.33 95.814 Association (IDA) 4 The percentage of PAD was 99.03 percent financed in SDR. The difference was largely due to the changes in the exchange rate. 23 Annex 2. Outputs by Component Component 1: Community Pasture/Livestock Management System Expected Actual Percent Target Completion of Component Results (Output) (Number) (Number) Target 1 Number of communities 54 81 150 2 Total population of communities 78,000 222,3685 285 4 Total pasture area 230,0006 178,000 77.4 5 Head of livestock (head) 120,000 120,000 100 6 Developed CPMLDPs 55 81 147 7 Established CPMLDCs 55 81 147 8 Established PUAs 55 81 147 9 Constructed water pipelines (km) 199 199.9 100 10 Constructed stock-watering points 243 243 100 11 Improved roads (sq. m) 75,900 75,900 100 12 Constructed barns 23 23 100 13 Constructed shepherd houses 22 22 100 14 Provided agricultural machinery 736 736 100 15 Trained PUAs members (person) 11,677 11,677 100 16 of which women (person) 3,107 3,107 100 Pasture areas provided with stock- 17 121,500 121,500 100 watering (ha) Degraded pasture areas rehabilitated 18 342 342 100 (ha) Component 2 – Strengthening Advisory Services Subcomponent 2.1: Agriculture Advisory Services (a): Technology Assessment Projects 5 A total population of around 78,000 of which about 38.000 women was expected at project appraisal. Due to the increased number of project communities - 81 instead of 54 previously envisaged at project appraisal, the incremental number of beneficiaries of the “Pasture Management and Livestock Development” Component was 133,000 (62,500 women). So the overall number of CARMAC project beneficiaries at completion finally was about 222,368 (89,568 women, about 30 percent). 6 At project appraisal, the total improved pasture areas were roughly estimated at the national average size by community (4,500 ha per community). At the project completion, the actual pasture areas ware calculated based on the specific communities involved in the project, and much smaller than estimated areas at PAD due to the actual small mountain villages selected in the poor remote areas. 24 Expected Actual Percent Technology Assessment/Advisory Target Completion of Projects (TAPs) Topic (Number of (Number of Target TAPs) TAPs) 1 Plant protection 40 40 100.0 2 Vegetable growing 24 23 95.8* 3 Veterinary and fodder production 23 23 100.0 4 Fertilization, Legume cultivation 13 12 92.37 5 Potato cultivation 9 9 100.0 6 Bee health management 2 2 100.0 7 Drip irrigation 1 1 100.0 8 Other 16 16 100.0 Total 150 148 98.7 Subcomponent 2.1 Agriculture Advisory Services (b): Agricultural Machinery procured for Marz Agricultural Support Centers (MASCs) Expected Actual Percent Agricultural Machinery Target Completion of (number) (number) Target 1 Wheel tractor 9 9 100.0 2 Baler 8 8 100.0 3 Plow 6 6 100.0 4 Cultivator 5 5 100.0 5 Double-wheel tractor (motor-block) 4 4 100.0 6 Sprayer (attachable) 4 4 100.0 7 Sprayer (assembled) 4 4 100.0 8 Drill 2 2 100.0 9 Grass(hay)-cutter 2 2 100.0 10 Rake 2 2 100.0 11 Automotive grass(hay)-cutter 1 1 100.0 12 Carrot harvester 1 1 100.0 13 Trailer 1 1 100.0 Total 49 49 100.0 Subcomponent 2.1 Agriculture Advisory Services (c): Equipment Provided to the Republican Agricultural Support Centre (RASC) 7 One project from each two topics was stopped because of weather conditions. 25 Expected Actual Percent Equipment Target Completion of (set) (set) Target 1 Computer 12 12 100.0 2 UPS 11 11 100.0 3 Laser printer 3 3 100.0 4 Camera, Digital recorder 3 3 100.0 5 Digital printing system 1 1 100.0 6 Video conference equipment 1 1 100.0 7 I-Pad 1 1 100.0 8 TV set 1 1 100.0 Total 33 33 100.0 Subcomponent 2.1 Agriculture Advisory Services (d): Trainings for MASCs Specialists Year Expected Actual Percent Target Completion of (number of (number of Target person/woman) person/woman) 1 2011-2012 278/45 278/45 100.0 2 2013 277/45 277/45 100.0 3 2014 253/48 253/48 100.0 4 2015-2016 330/60 330/60 100.0 Total 1138/198 1138/198 100.0 Sub-component 2.2: Community Animal Health Services Outputs Expected Actual Percent Target Completion of (number) (number) Target 1 5 veterinary service centers 5 5 100% 2 Special vehicles for veterinary 0 4 services centers 3 Trained veterinarians 48 67 139% 4 Equipped veterinarians 48 67 139% Total 101 144 126% Component 3: Competitive Grants Program Expected Actual Percent Sector (field) Target Completion of Target 26 (number of (number of project) project) 1. Livestock farming, including: 1.1 Cattle breeding 1 1 100 1.2 Pig breeding 3 3 100 1.3 Sheep breeding 2 2 100 1.4 Poultry breeding 6 6 100 1.5 Rabbit breeding 2 2 100 1.6 Fish breeding 1 1 100 1.7 Chinchilla breeding 1 1 100 Fruits and vegetable 100 2. 3 3 production/growing 3. High value berry production 8 8 100 4. Industrial crops and seed production 5 5 100 5. Plant protection 1 1 100 Fruit and vegetable processing, dried 100 6. 17 17 fruit production Milk processing and dairy 100 7. 8 8 production 8. Vegetable oil production 4 4 100 9. Honey production and processing 1 1 100 10. Other projects 6 6 100 Total 69 69 100 1. Overall, at project completion, all project investment activities/components had been fully implemented and completed in general, and all implementation outputs by component and the intermediate outcome targets expected at the project appraisal had been achieved or exceeded (as presented in the Outputs by Component Table above and the Data Sheet Table), which had contributed to the project PDO achievement successfully. Detailed output results achieved by component as presented are described below: 2. Component 1: Community Pasture/Livestock Management System. The implementation of the component one was fully completed at the project completion, and all expected output and outcome results were highly satisfactorily achieved as follows:  CPMLDPs were developed and implemented in 81 beneficiary communities (147 percent of target value), with the involvement of 133,000 livestock farmers (31,000 families - 100 percent of the target value). The project post-evaluation confirmed that the provisions of the CPMLDPs were fully implemented and the management rules and regulations were largely followed. Some of the early PUAs exceeded the original rotational grazing planning framework of three years. The PUAs adopted the planning process and continued developing their own rotational grazing regime following the principles of the original plans. 27  CPMLDCs and PUAs consumer ccooperatives were established for the effective management of the pasture and livestock resources in 81 communities (147 percent of the target value), with a significant amount of training provided for community mayors, chairmen, accountants, and dispute resolving persons of PUAs (11,677 cooperative members trained - 100 percent of the target value). The operational strength and management capacity of the PUAs was repeatedly assessed by the METT which showed a continuing improvement of the management capacity and organizational coherence of the PUAs in all management aspects. The average score increased from 18 (first assessment) to 57 in 2016 (95 percent of the target value).  Rational and sustainable use of communal pastures: This was achieved through the completed construction/improvement of community infrastructures for pasture/livestock development; including 200 km water pipelines, 243 stock-watering points, 75,900 square m pasture access roads, 23 barns, 22 shepherd houses (100 percent of the target values). The procurement of 736 sets of agricultural machinery, mainly used for the production of additional winter fodder (100 percent of target value) in 81 communities, allowed the use of an additional 121,500 ha of previously unused or underused valuable pasture lands, and 342 ha of degraded pasture areas rehabilitated (100 percent of the target value). This reduced pressure on heavily grazed nearby areas and further degradation for a total 176,000 ha of land.  Efficiency of pasture management: Proceeds from the lease of pastures, which were transferred to PUAs resulted in budget increase from AMD 328,893 in 2012 to AMD 561,395 in 2016 (121 percent of the target value), based on an increased lease of pasture and grass land from 123,714 ha to 176,000 ha (50 percent of the target value).  Institutional and financial sustainability of PUAs : In addition to the increased pasture fee indicated above, the adequate fee arrangement and collection system for agricultural machinery services and pasture management was established. The project post-assessment on the arrangement of the fee calculation and collection for agricultural machinery services indicated that the APIU and PUA had introduced and applied an electronic system for the fee calculation and collections. The agricultural machinery service charge for various machineries/equipment had been calculated based on the full O&M and depreciation costs, which provided sufficient financial support to newly established PUAs for their future development and implementation of CPMLDPs, through a self-financially sufficient management system. All PUAs had used computers for record keeping of the pasture use arrangements, machinery use and fee collections.  Efficiency of livestock farming increased livestock feeding : Production of winter fodder increased from 45 percent to 90 percent of the actual requirement (112 percent of the target value), which led to increased milk production both for cattle and sheep by 37.4 percent and 28.7 percent respectively (compared with expected project targets of 20 percent and 10 percent). Similarly, the growth rates of animals for cattle and 28 sheep increased by 27.4 percent and 27.1 percent respectively against the end project targets of 20 percent and 5 percent.  Farm income from livestock sales: Increased by almost 130 percent at project completion (from AMD 532,147 to AMD 1,226,226 in 2016), which significantly exceeded the appraisal expectations of 20 percent. The main reason for such unexpected increase as compared to a more moderate productivity increase is seen in the change of many households from subsistence production to commercial market production.  Technical measures for rehabilitation and protection of pasture resources : This was demonstrated and tested in selected small pilot areas (three to max eight hectares), including (i) increase of productive areas by stone collection, shrub cutting and extirpation; (ii) improvement of topsoil aeration through rake works; (iii)) improvement of vegetative cover and pasture productivity with enhanced flora composition through direct seeding of grass mixtures with leguminous and cereal plants (about 18-20 kg seed mixture per ha); and (iv) improvement of soil NPK nutrients through mineral fertilization following norms and account taken of current soil conditions, which contributed directly to the PDO achievement on the improved productivity of the pasture/livestock production. 3. Component 2: Strengthening Support Services: Component 2 included two sub- components as follows: a) Subcomponent 2.1 Agriculture Advisory Services. The outputs achieved under this sub-component strengthened the capacity of the existing network consisting of MASCs and the RASC to deliver services in livestock and related topics through: (i) improved effectiveness and outreach through training and provision of essential equipment; (ii) funding on-farm technology assessment projects (TAPs); (iii) livestock training and demonstrations for farmers; and (iv) improved information systems using modern information and communication technologies including a pilot Short Message Service (SMS) messaging system. This sub-component has been completed satisfactorily. All main output results expected at project appraisal had been fully achieved as follows:  TAP program was completed in 2014 with a total of 148 TAPs (99 percent of target value8) and 94 fact sheets published; 49 sets of agriculture machinery (100 percent of the target value) were provided to MASCs; 33 sets of essential communication and office equipment were provided (100 percent of the target value) to the RASC including some notebook computers for advisory staff. Training programs were organized for farmers and in-service refresher training provided for 1,138 MASC and community advisors and specialists (100 percent of the target value). A pilot SMS messaging system was tested. The MoA continued to provide budget funds for core 8 One project from each two topics was stopped due to the weather conditions. 29 advisory activities as defined in annual work-plans and budgets prepared by the RASC and all MASCs, and maintained the budget at AMD 396 million for 2016. In addition, the MASCs and RASC self-financed 14 percent of their costs in 2015 through provision of various services and contracts. As the Joint Stock Companies, MASCs and RASC retained these earnings from their operations and activities. Most activities funded under the project will be continued as part of the core program funded by the MoA.  Training of Advisors: A program of in-service training organized by the RASC has been carried out annually. An average of around 228 technical advisory staff (including about 40 women, 100 percent of the target value) comprising both MASC staff and community advisors participated in annual training program (from 2012 to 2016). Topics included various livestock production, animal health, pasture management, fruit and vegetable production, plant protection, bee-keeping and extension management aspects. Trainers primarily comprised specialists from the RASC, ANAU and Scientific Centers. Course evaluations indicated that the program has effectively improved knowledge and skills of advisors, and has enabled advisors to provide training to more than 20,000 farmers annually. In 2012 and 2013, training in economic legislation including business planning was also provided for a total of 297 specialists. This training increased the capacity of the MASCs to help farm to prepare business plans and to provide advice on legislative matters, with a total of 2,330 plans prepared since 2012 including 1,223 in 2015. These refresher training courses are essential to maintain and improve advisors’ knowledge and skills and to enhance their capacity to serve farmers and communities.  Training of Farmers: Following on the training of MASCs' advisors, additional training programs for farmers were carried out, particularly comprising courses in livestock topics. In 2012, such training courses were conducted for 878 farmers (including 168 women) in 35 communities under the project. From 2013 the farmer training program was carried out using the MoA core funding, with more than 20,000 farmers attending seminars/workshops annually, although not specifically in livestock topics only. However, training was provided for members of PUAs under the contract with Green Lane NGO.  Provision of Essential Equipment: Assets including agricultural machinery, portable computers (notebooks), and communication and office equipment were provided to the MASCs and RASC to allow increased potential for self-financing and to increase outreach. In 2015, the agricultural machinery and equipment generated AMD 11.7 million income for the MASCs, while an additional AMD 5.9 million was generated by the RASC from the use of the equipment provided. Additional video equipment and computers for television production purposes were also provided to the RASC at a cost of around US$50,000 at the end of 2015 (see the detailed procured agricultural machinery and communication and office equipment presented in the table above).  Improved Information Systems: To improve outreach, community advisory rooms were established under the previous now closed RESCAD Project and 47 portable computers provided under the project to increase access to information. MASC advisors visited each community on a scheduled basis, and thus farmers were able to 30 access information regularly. A pilot SMS messaging system was tested in one Marz under the project, and nearly all communities have internet access and weather information is readily available. Each MASC and RASC also has developed a good quality website providing access to information, and distributes newspapers and other printed material to interested farmers, including fact sheets from the TAP.  Advisory Service Outputs: The level of activity of the public advisory system was impressive. More than 70,000 farmers were served in 2015, including 22,895 permanent clients. Activities in 2015 included 146 demonstrations, 186 field trials, 1,370 seminars or trainings, 107 radio and TV programs, 1,223 business plans and 899 publications. Typically, around 20-30 percent of direct participants in the programs were women and almost 40 percent indirect beneficiaries.  Advisory System Strategy: A strategy for sustainable development of the advisory system was prepared by the MoA and formally accepted by the GoA in December 2013. The strategy acknowledges the important role of the RASC and MASCs and recognizes that budget support will continue to be needed. The strategy also proposes actions to further strengthen the system, such as improving the logistical base, increasing the number of advisors, creating demonstration fields adjacent to the MASCs, continuing in-service training, enhancing marketing and business support, and expanding collaboration with local self-government bodies. This strategy was endorsed by the WB as described in the project technical note on strengthening agricultural extension for the future WB and national project implementation.  Scientific Center of Agriculture: Around AMD 71 million (approximately US$150,000) was provided for the purchase of field equipment for the Scientific Center of Agriculture, which is the agency primarily responsible for breeding and selection of wheat, barley and grain legume varieties; maintenance of breeding material including lines provided by international centers such as The International Maize and Wheat Improvement Center NGO and International Center for Agricultural and Rural Development (ICARD); and production of super-elite and elite seed for sale to producers. They initially had outdated and inefficient farm machinery and relied on rented machinery, with poor seed bed preparation, harvest losses, and difficulty in keeping breeding lines and varieties separate. Equipment provided consisted of a 4WD tractor and cultivation and planting equipment, fertilizer spreader, sprayer, baler and a grain conveyor delivered at the end of December 2015. This equipment has enabled the Center to repay about AMD 8 million from the sale of high-generation wheat seed so far, with the expectation that the rest will be covered through additional crop sales from rotational crops and savings from machinery renting. In the future, the higher yields and increased net profits from operations are expected to enable increased expenditures for scientific and breeding activities.  Impact Assessment: Impact assessments of the advisory services were carried out in 2012 and 2014 by an independent consultancy using the same methodology as previous surveys in 2004 and 2008. The final results were very positive: 71 percent of farmers used the advisory services, 97 percent of respondent farmers knew when their advisors visited, 95 percent of farmers stated that MASCs’ advisors visited at least once a month, 85 percent indicated an income increase, 92 percent had introduced 31 technological innovations, 97 percent indicated productivity growth, and 63 percent indicated willingness to pay. These indicators show a steady increase overtime, and provide solid justification for the continuing budgetary support from the MoA. It should also be noted that agricultural productivity, as measured by the Food and Agriculture Organization crop and livestock production indices, has shown a parallel increase of more than 16 percent during the last ten years. While these trends cannot be attributed entirely to the advisory system, there is little doubt that increased farmer knowledge and skills have contributed. b) Sub-Component 2.2: Community Animal Health Services . The output results achieved under this sub-component improved and provided adequate animal health services at the community level in the six Marzes targeted areas by: (i) mobilization of Community Veterinarians (CV), including training and certification; (ii) provision of veterinary equipment and supplies for participating veterinarians; and (iii) establishment of VSCs in the project beneficiary Marzes. The main output targets expected in the PAD were fully achieved and exceeded as follows:  VSCs: five VSCs (125 percent of the PAD target value) had been established in the Khndzoresk community of the Syunik Marz, the Artashavan community of the Aragatsotn Marz, Chambarak community of the Gegharkunik Marz, Berd community of the Tavush Marz and Panik community of the Shirak Marz. The centers were equipped with necessary tools and equipment. Four mobile veterinary vehicles were procured for the completed VSCs, and 67 community veterinarians were selected and trained (139 percent of the target value), and the required veterinary equipment procured and distributed (139 percent of the target value), of which 15 veterinarians also received artificial insemination equipment, and all which were operational at the project completion.  Mobilization and Training of Community Veterinarians : The project has created a network of veterinarians to deliver services in the project Marzes (8 CVs per Marz). The CARD was contracted to train and select CVs and carried out the training program from February to December 2013. The training actually included 100 CVs for participation in the program with 67 fully trained and certified -- more than the originally foreseen 48 CVs. Training modules included artificial insemination (AI), diagnosis and treatment of diseases including reproductive diseases, dehorning and hoof trimming, pharmacology, blood sampling, dairy cattle management, hygiene, and private veterinary business management. A veterinary equipment package was then provided to the 67 trained CVs with 15 CVs also receiving AI equipment.  Veterinary Service Centers: Five VSCs under the CARMAC project were constructed, staffed and equipped, and are operational. Each VSC has a training room, office space, and a dispensary for sale of a range of items including veterinary supplies and medicines, feeds and supplements, some agricultural chemicals and fertilizers, bee- keeping and cheese-making inputs and other supplies. Equipment such as small milking machines, mowers, and sprayers are also available. The project provided items such as furniture, computers, training equipment, refrigerator, scales and some 32 artificial insemination equipment, while CARD is providing semen for artificial insemination, and veterinary supplies and medicines.  Mobile Veterinary Vehicles: Mobile veterinary vehicles were procured for the four completed VSCs. These are small vans (1,600 cc) with 3.5 t capacity fitted with shelving, cabinets and with space for basic veterinary and artificial insemination equipment. They will enable transport of medicines and semen and allow greater outreach of the veterinary services provided through the VSCs.  VSC Management: The VSC management consists of a trained veterinarian, assistant and book-keeper. There is also a network of around 8 to 20 CVs associated with each VSC. In addition to sales of equipment and supplies, the VSCs acts as a focal point for training and consultancy activities for the CVs and farmers in the surrounding communities. In most cases, there are linkages with the MASC and MASC advisors in the communities.  Impact Assessment: A field survey was conducted to assess VSC performance including the amount and categories of sales; number of beneficiaries; trainings and consultancies; and details of artificial insemination and other activities, especially results of better animal health care and farmer satisfaction with the services. According to the post-evaluation, all centers are proving high efficacy of the works carried out by the centers, including the wide range of veterinary services provided, and there is high contentment and satisfaction of the beneficiaries (farms) using the VSC services. The VSCs contributed to the provision of quality veterinary services to the farmers and increased availability of veterinary medications and tools. The evaluation results showed that during the first semester of 2016 the turnover of the centers from the sold goods and provided services amounted to around AMD 30 million with 6,900 visitors from 71 communities, of which 73 percent of the visitors got veterinary consultation, 43 percent for veterinary services, and 25 percent for artificial insemination. 4. Component 3: Competitive Grants Program: This component funded proposals from village-level agri-business and farmer groups to introduce innovative technologies and income-generating activities that could benefit communities focused on livestock production. The maximum grant amount was US$20,000, plus a beneficiary contribution of at least 30 percent of the grant. The expected output targets as agreed at the project appraisal were fully achieved.  A total of 69 grants (100 percent of the target value) were funded through the CGP over seven rounds, and fully completed. During the project, 7 competitive rounds were launched and a total of 224 proposals received, from which 69 were selected and implemented. All subprojects were evaluated, the average project size was US$37,000 (with grant amounts ranging from US$21,000 to US$51,000) and the average grant amount was US$16,000 (with grant amounts ranging from US$10,000 to US$17,000).  Grant Characteristics: 24 of the 69 grants were related to various livestock production and processing topics, while the remaining grants primarily were related to production and processing of higher value fruit and vegetables and other non-traditional 33 commodities. Since many Armenian farmers are mixed livestock and crop producers, the CGP was able to introduce and test improved livestock technologies as well as a range of possible alternative income-generating opportunities for all communities. Grants were awarded to applicants from all 10 Marzes as planned.  Technology transfer: A major focus of the CGP was to demonstrate innovative technologies and improved business practices to other farmers and stakeholders. Accordingly, the costs of technology transfer were included as an integral part of all grants. At project completion, technology transfer activities had been carried out and completed for all 69 grant projects. In total, some 210 technical seminars and 108 field visit days or open days were carried out; around 11,600 leaflets and 4,700 brochures printed and distributed; 105 TV spots broadcast and 90 different posters and 15 newspaper articles published. Service providers contracted through each grant were responsible for the technology transfer activities, and included a range of ANAU faculty staff, researchers, advisory staff, NGOs and private consultants.  Beneficiary Contribution: The beneficiary contribution, including both in-kind and cash, was estimated to be US$1.792 million (57 percent of the total grant projects cost), a very significant investment by the beneficiaries. The cash contribution was US$535,865 (almost 30 percent of the total beneficiary contribution). Direct beneficiaries included 28 non-formal farmers’ groups (including 3 women’s groups), five Cooperatives, 13 Limited Liability Companies and 23 registered individual entrepreneurs. There were 334 direct beneficiaries and an estimated number of more than 33,475 indirect beneficiaries, including purchasers of products produced by the grant winners.  Impact Assessment/Outcomes: All 69 grant projects were evaluated. All projects helped create viable business opportunities, and provided significant income gains to the beneficiaries ranging from 5 to 60 percent, and increased employment opportunities from 2 to 20 people under each project. As result of the technology transfer activities, the evaluations indicated that elements of many grants were emulated by other beneficiaries by the end of the grant period, with further adoption likely to take place over time. Examples of innovations included improved breed selection and animal husbandry; new dairy products and more developed contracting approaches with producers; improved storage, processing, packing and marketing of fruit and vegetable products; and introduction of various non-traditional poultry and crops. 5. Component 4: Project Management and Monitoring and Evaluation : The project was managed by the same APIU that implemented the RESCAD and the AIP projects. This component financed: a) Project management and training, including annual operational reviews and audits All activities under this component were fully implemented and completed, and expected output results achieved. The approach to management, monitoring and evaluation varied somewhat between project components and sub-components according to the nature of the activities involved. 34 b) Overall M&E System: M&E system was developed and implemented at the early stages of project implementation, including annual reporting on implementation of activities and the performance of livestock enterprises (milk yields, daily live weight gain, livestock numbers, increased farm sales from livestock and increase in incomes, and winter-fodder requirements met) and an electronic system/format was developed for better recording of pasture fees and agriculture service fee due and received. c) M&E of the component 1: For M&E purposes the APIU also used an METT (described earlier) to measure progress and outputs in pasture management for each project community. Following the principles and methodology, this tool has been modified and adopted to track progress of management effectiveness of pasture and livestock management systems under the CARMAC project. Monitoring of the training included keeping records of the trainings held and the number of people that attended, surveying participants’ pre- and post-course knowledge and their ratings of the training received. Records were kept of the equipment provided and its impact on the beneficiaries d) M&E of the component 2: Monitoring of the TAP was based on visits to a random selection of the funded projects. For the animal health sub-component, monitoring concentrated on records of trainings given, equipment provided and progress made in the construction of veterinary centers e) M&E of the component 3: Detailed records were kept of each round of the CGP implemented under Component 3, and 18 of the 69 grants were selected for detailed monitoring and evaluation, resulting in an overall performance score of 1-10. 35 Annex 3. Economic and Financial Analysis Introduction 1. The ultimate objective of the CARMAC project was to increase the net income of livestock producers. Other measures and targets, such as increasing pasture productivity and livestock productivity, are intermediate goals along the road to the ultimate objective of raising incomes. 2. The project’s structure and achievements are described below, while the table on the following page sets out the project elements and their main anticipated effects on costs, benefits and intermediate indicators. 3. The main task of this economic impact evaluation is to quantify the costs and benefits, and compare them over time to calculate the project’s Economic Rate of Return (ERR). A secondary task is to analyze the intermediate impacts to help understand how the project achieved or failed to achieve its stated objectives, and from this to draw conclusions that may be used to improve the effectiveness of CARMAC 2 and similar projects. Measuring benefits 4. The ultimate benefit to be measured is the net income of project communities, but it may be assumed that the project had no significant impact on non-agricultural income or expenditure, so this may be reduced to measuring the impact on net farm income. Most activities under Components 1 & 2 may be assumed to affect only livestock income and expenditure, so the measure may be further narrowed to that of impact on net livestock income, together with separate treatment of the impact of Component 1d (machinery provision) on costs and revenues from cash- crop production. 5. The project may be assumed to have had very little impact on the fixed costs of livestock production, since the land belongs to the beneficiary communities, almost all the labor is provided by the beneficiary farmers and herders, and the project-related capital expenditures are treated in the cost side of the equation. Thus, the measure of net livestock income may be reduced to that of livestock gross margins. 6. Unfortunately, project monitoring did not record livestock-related variable costs such as feed, purchased forage and veterinary costs. Field visits by the consultant and Marz Support Teams (MSTs) indicated that neither farmers nor community leaders kept records of these expenditures, and that asking beneficiaries now to try and recall their expenditure over the previous five years would be highly unreliable. This deficiency has been addressed for CARMAC 2 project by the introduction of record keeping on a representative sample of farms, but as far as CARMAC project is concerned, the best available measure from which to estimate project impact is changes in the value of livestock output. 7. Using the value of livestock output as a proxy for livestock gross margins effectively assumes that the project had no net impact on livestock variable costs. Any increase in livestock numbers would tend to increase feed costs, whilst increased pasture productivity would tend to reduce them. The veterinary component may both have increased the consumption of veterinary services and reduced their price. It is thus hard to say whether the project increased or decreased 36 livestock variable costs, so the assumption of no net impact is the most reasonable that can be made. 8. Component 3 (competitive grants program) had very different impacts on a different group of beneficiaries, and so is evaluated separately. 37 Table 1: Project elements and their main anticipated effects Component Main activities Costs Intermediate impact Benefits 1. Community a) Management:  Work by APIU & MSTs  Increased pasture production, Pasture/Livestock  Establishment of Pasture Users’  Local contracts to assess harvested through cutting & Management Associations pasture and develop grazing System  Development of Pasture Management Plans  Increased livestock output through higher Management Plans livestock numbers &/or higher output per head  Implementation of rotational  Feed and other costs decreased, static or grazing increased by less than output value b) Access:  Funding of design, works  Increased area of pasture utilized  Construction of access roads and & equipment  Increased productivity of rested watering points for remote pastures nearby pastures c) Machinery:  Grants for machinery  Increased area of crops grown &  Lower production costs compared to  Provision of agricultural purchase forage harvested, &/or lower using private machinery contractors machinery for forage and crop machinery fees  Increased sales/decreased purchases of production  Higher crop & forage cash & forage crops productivity due to more timely  Increased livestock output where extra operations feed & forage used on-farm – Captured above 2. Strengthening a) Advisory:  Contribution to advisory  Better management of forage,  Higher output &/or lower costs – Support Services  Strengthening state advisory staff salaries crops & livestock Captured by measures for Component 1 services  Purchase of equipment  Running training courses and  Funding of training & technology demonstrations demonstrations  Supporting Pasture Users’ Associations to buy advice c) Veterinary:  Provision of supplies &  Healthier livestock  Higher output from healthier livestock –  Establishment of veterinary service equipment  Higher genetic potential of Captured by measures for Component 1 centers  Training progeny from AI  Vaccination campaigns  Establishment of AI services 3. Competitive  Grants for crop and livestock  Grants to successful  Improvements in quantity,  Increased profitability of grant Grants Program production applicants quality, marketing &/or cost beneficiaries – Measured under separate  Grants for agro-processing reduction evaluation of Component 3 4. Project  Establishing Agricultural Policy  Funding of staff,  Allowed the above activities to Management and Implementation Unit (APIU) equipment & travel take place effectively and at Monitoring and  Forming Marz Support Teams  External contracts reasonable cost Benefits captured by the measures for Evaluation (MSTs) Costs may be spread across Components 1-3  External monitoring and Components 1-3 environmental assessment 38 Analyzing impact 9. Changes in the value of livestock output over the course of the project are due to four main factors: a. Project expansion – Progressive expansion of the project from 55 communities in 2011 to 81 communities by the end of 2014. b. External factors – The changes in livestock numbers and productivity that would have happened without the project due to weather, to farmers’ responses to market conditions, and to developments in the livestock sector that were taking place anyway. c. Project impact on livestock output – The impact of the project on livestock numbers and productivity. d. Market fluctuations – The prices obtained for milk and meat, which change continuously. 10. The goal of this economic evaluation is to measure the third factor – Project impact on livestock output – and to convert it into a monetary value. External factors 11. The hardest part of any evaluation is creating the counterfactual or “without- project” scenario so as to estimate how the beneficiary units would have performed each year if they had not been in the project. The three main approaches commonly used for this are:  Before-and-after comparison , which implies a without-project scenario of the beneficiary units continuing exactly as they were in the baseline year;  Control group, where the without-project scenario is that the beneficiary units would have performed the same as the control group in each project year;  Difference-in-differences, where the without-project scenario starts from the baseline and then assumes that the beneficiary units would change each year in the same way as the control group. 12. No control groups were identified for this project, so this evaluation applies a modified difference-in-differences approach, whereby the without-project scenario starts from the baseline year and assumes thereafter that the beneficiary units would change each year in line with the national flock or herd. This is done by applying chain indices calculated from national statistics, after adjustment to remove the contribution of the beneficiary communities to those same statistics; this methodology is set out in Appendix 3. Project expansion 13. In order to remove the effect of the expanding project base, annual values are estimated for each beneficiary community from 2011 until they joined the project and began providing annual data. This was done by taking the first year in which data were available, and then working backwards to 2011 and applying the same chain indices as 39 used to create the without-project scenario. The 81 project communities fall into three groups:  55 communities joined the project in 2011 and began providing data immediately (though in some cases it took a further year to two to formalize establishment of their Pasture Users’ Association), so no estimation was required;  12 communities joined and started providing data from 2013; estimates have been made for their performance in 2011-12 (labeled in the graphs as “pre-project”);  14 communities joined and started providing data from 2014; pre-project estimates have been made for 2011-13. 14. Thus, the model contains data for all 81 communities right from 2011 and so the estimated impact is not distorted by project expansion. For the years before the 2013 and 2014 communities joined the project, their extrapolated with-project values are identical to their estimated without-project values and so have no effect on the calculated impact. One of the most important steps for communities joining the project was the formation of their Pasture Users’ Association (PUAs) and its registration as a formal cooperative. In most cases, communities began providing regular data from the year in which they registered their cooperative but sometimes the data flow began a year earlier or later. This analysis treats the start year as being the first year in which data were collected, which in practice is when communities began to work regularly with and be influenced by the project and its MSTs. Market fluctuations 15. The project did not include a marketing component or any measures designed to improve product quality, so it may be assumed that beneficiary communities would have received the same prices even if they had not been in the project. The economic model therefore uses actual prices reported by the communities for the with-project scenario, and a weighted average of these prices for the without-project scenario. 16. For prices that were not regularly recorded by project communities (i.e. prices of sheep’s milk and live sheep), national estimates have been made for each year and used for both the with- and without-project scenarios. Project impact 17. The impact of the project is calculated as the difference between the with-project and without-project scenarios. Estimates of impact are made in this way for livestock numbers, productivity, physical output and output value. Implementation over time 18. The project began with 55 communities recruited in 2011-12. Once investment plans were developed for these communities it became apparent that not all project funds would be consumed, so a second group of 28 communities was recruited in 2013-14, of 40 which two later dropped out. This gives a total of 55 + 28 - 2 = 81 communities now available for evaluation. 19. The following chart shows, for each of these communities, when its PUA was registered as a cooperative, when the civil works for watering points and access roads were completed, and when the first and last items of equipment were handed over: Figure 1: Timeline of CARMAC cooperative formation 20. It can be seen that works were still underway and new equipment being provided through to the middle of 2015, sometimes less than a year before the final project monitoring data were collected on 1st August 2016. Thus for many of the communities the project benefits will still be developing, with a typical timeline as follows: Year 1  Selection and recruitment of project communities  Training seminars  Formation of Pasture Users’ Associations and registration as Consumer Cooperatives  Detailed assessment including first “METT” score 9  Preparation of Pasture Management Plan  No significant benefits in Year 1 9 The project applied a “Management Effectiveness Tracking Tool” (METT) to record changes in pasture management. METT assessments were typically made in the 1st, 2nd and 4th years of a community’s involvement in the project. 41 Year 2  Implementation of Pasture Management Plan on existing pasture area  First use of Artificial Insemination (AI) 10  Tendering, contracting and works to install access roads and water points, typically completing in the autumn, close to the end of the grazing season  Procurement of agricultural machinery, typically arriving in the autumn in time for ploughing and seeding of winter cereals  Farmers may retain more female calves and lambs for future breeding, in anticipation of improved feed supply  Limited benefits in Year 2 Year 3  First calves born to AI  Grazing extended to new areas, thanks to access roads and watering points  Grazing pressure on nearby pastures reduced, allowing the start of pasture regeneration  Increased areas available for hay production, using new machinery  First harvest of crops using new machinery  Larger number of heifers and gimmers11 mated  Livestock go into the winter with an increased supply of feed and fodder  Significant benefits start to arise from Year 3 Year 4  Increased cow herd and ewe flock to produce offspring and milk  Birth-weights increased due to better winter nutrition  Higher milk yield of cows and ewes in early lactation due to higher fodder availability  Increased grass availability for grazing and hay-making, largely similar to Year 3 with possible further increase due to pasture regeneration  Crop benefits from new machinery as in Year 3  Benefits grow Year 5  First calves born to AI themselves give birth and start producing milk  Continuation of benefits as in Year 4  Benefits continue to grow, though more slowly than in Years 3 and 4 10 In practice, the project gave less emphasis to Artificial Insemination than originally planned. The timeline shows the fastest possible stream of benefits from this activity; these benefits would slip back if the start was delayed, and would not apply to communities where AI was not implemented. 11 Female sheep (about to be) mated for the first time. 42 Subsequent years  Pasture may continue to improve for several years if managed well  Cattle herd progressively influenced by improved genes from AI  Herds and flocks will gradually expand, until the point where grazing pressure again threatens pasture productivity and so the PUA has to start rationing grazing rights  Maintenance costs for roads, watering points and machinery will tend to rise over time  Benefits will gradually approach a plateau According to this logic, the first significant project benefits would arise in 2013-14 for the first 55 communities, and in 2015-16 for the second group of 26 communities. Benefits will be expected to reach their maximum level over the period 2017-20. Impact of Components 1 & 2: Pasture, livestock & fodder 21. Component 1 acted to increase the profitability of community pasture/livestock systems through improvements in pasture management and access, and the provision of machinery for crop and fodder production. Component 2 strengthened advisory and veterinary services to help pastoral communities and other farmers. Pasture Impact on pasture and fodder production 22. Interventions under these components may be expected to have increased pasture and fodder production in several ways:  improving access to distant pastures and so allowing farmers to graze larger areas of pasture;  increasing the area that could be shut up for hay production, as livestock could now be grazed on distant pastures;  gradually increasing pasture productivity as swards recovered from over-grazing; and  allowing farmers to harvest more hay from larger areas, due to the provision of additional machinery. 23. The areas of pasture improved under the project are recorded in round terms (e.g. “500 ha,” “1000 h” or “1500 ha”) because the actual area grazed depended on how far the herders moved from the access roads and watering points, but detailed estimates were made of fodder (i.e. hay) production in each community. With almost 20,000 households affected by the end of the project, farmers may have varied their response from year to year according to the amount of fodder and grazing available. Sales and purchases of feed and fodder were not recorded in CARMAC project (but will be in CARMAC 2 project), so the remainder of this analysis looks at how increased grazing and fodder production has affected livestock numbers, productivity, output and revenue. 43 Cattle Cattle numbers 24. Project communities provided annual estimates of their total numbers of cattle and of milking cows. The number of cows in beneficiary communities grew almost exactly in line with national trends, indicating that most farmers did not particularly try to increase their herds in response to the greater availability of grazing and fodder. However, total cattle increased more slowly than cow numbers, with the difference being due to “Other cattle”. This shows that in beneficiary communities the number of fattening calves and replacement heifers grew less than national trends and slower than the growth in cows, leading to a change in herd composition. This effect was most noticeable in the later years of the project when liveweight prices were falling sharply whilst milk prices held firm. 25. The data provided by project communities are not very specific on how and when livestock numbers were recorded, but it seems that some farmers may have responded to the difficult market conditions by selling their surplus calves earlier and focusing more on milk production. It is not clear whether this was a specific feature of the project, driven in some way by the increasing availability of fodder resources, or was a general feature of mountain communities. Milk yield 26. Estimates of milk yield were provided annually by communities, based on discussion with all their members: Figure 2: Milk yield per cow 44 27. The red double line shows the average milk yield from national statistics, rising by 11 percent over the project period, from 2,040 to 2,270 liters/cow. Yields in beneficiary communities (solid green line) were initially much lower, just 1,500 liters/cow, due to the small breeds and low level of feeding. The tan line shows the average yield of non- beneficiary communities, including lowland herds with higher-yielding cows, which started at around 2,140 liters/cow. 28. The dotted green line shows how milk yield in the beneficiary communities would have evolved if they had followed national non-project trends: a modest increase of 125 liters/cow. However, the reality was that beneficiary communities performed much better than this, increasing yields by more than 500 liters/cow (one-third) over the lifetime of the project. Comparing this with the previous graphs indicates that farmers generally used the increased grazing and fodder availability to feed their animals better and increase yields, rather than to expand their herds. Sale weight 29. Male calves, and female calves’ surplus to replacement requirements, are normally reared for 9-10 months and sold in the autumn to traders and butchers who visit the village. Cull cows are marketed in the same way, and a proportion of the calves are slaughtered on the farm to meet household requirements. Cattle are sold live and weighed before sale to establish their price. Community leaders provided the project with average weights for the animals sold during the year, but not with precise information on the numbers sold. It is therefore assumed that the number of animals sold is similar to the number of “Other cattle,” calculated as total cattle minus milking cows. 30. This number will be an overestimate in so far as it includes replacement heifers that actually remained on the farm, but an underestimate in omitting cull cow sales; however, in the long run these two will balance each other out as each heifer eventually becomes a cull cow and so the “Other cattle” number may be quite a good proxy for cattle sales. 45 Figure 3: Average liveweight of sold cattle 31. Here the difference in starting points is even more pronounced – liveweight gain in project communities was just one-third that in the rest of the country – but the project impact was also striking: while non-project herds experienced a 22 percent fall in liveweight gain, project communities enjoyed a 26 percent increase. Comparing actual performance of project herds with the predicted without-project scenario, average liveweights of sold cattle increased by 32 kg/head rather than falling by 26 kg/head. Sheep 32. Whilst the total number of sheep and goats in project communities is fairly similar to the total number of cattle (both stood at around half a million in 2011), sheep are kept by fewer households and in larger flocks, with greater variation between households and between communities. Overall, the methodology and findings for sheep are generally similar to those for cattle, apart from an issue of definitions and the results for liveweight gain. Livestock output 33. This section multiplies production by prices to calculate the total value of livestock output under the with-project and without-project scenarios. 34. Project communities provided annual data on average prices achieved for cow’s milk and cattle sold for slaughter, but did not provide comparable information for sheep. Therefore prices for slaughter sheep and sheep’s milk were collected from a sample of communities in 2016 and extrapolated backwards for earlier years by assuming constant price ratios for cows’ milk:sheep’s milk and cattle:sheep. 46 Output value at 2011 prices 35. The following chart shows estimates of total output value at 2011 prices and therefore reflects only changes in production, without market effects. 36. Values for the four elements of livestock output were calculated as follows:  Value of cows’ milk = Number of cows  Milk yield/cow  Price of cows’ milk  Value of sheep’ milk = Number of sheep  Milk yield/cow  Price of sheep’s milk (this assumes that the value for “Sheep” reflects only breeding ewes, and so may be an overestimate; price of sheep’s milk is taken from 2016 and worked backwards in proportion to cows’ milk prices)  Value of cattle sales = Number of other cattle  Liveweight/head  Price of sold cattle  Value of sheep sales = Number of sheep  Liveweight/head  Price of sold sheep (assumes an average of one lamb sold per ewe, taking “Sheep” as representing ewes; lambing percentage would typically be higher than 100 percent so this may be an underestimate; price of sheep liveweight is also taken from 2016 and worked backwards in proportion to cattle prices) Figure 4: Value of livestock output at 2011 prices 37. If prices had remained constant, all elements of livestock output would have risen steadily throughout the project, increasing the annual livestock income of beneficiary communities by almost 60 percent, from AMD 27.4 billion to AMD 43.5 billion. The average breakdown of livestock income over the whole project period was as follows: 47 Table 2: Breakdown of livestock income Product Cattle Sheep Total Milk 27% 3% 30% Meat 43% 27% 70% Total 70% 30% 100% 38. In terms of products, 30 percent of income came from milk and 70 percent from meat, and in terms of species, 70 percent came from cattle and 30 percent from sheep. Overall, sheep’s milk accounted for just 3 percent of total livestock income whilst sales of cattle generated 43 percent. 39. The purple line gives the total output under the without-project scenario, showing what sales would have been if livestock numbers and productivity had simply tracked the national non-project flocks and herds. Comparing actual output with this line shows that cumulative benefits over the project at constant prices would have been AMD 26.7 billion, equivalent to US$55 million at the current rate of exchange. Price development 40. That is what would have happened if prices had remained constant; however, agricultural prices are never constant: Figure 5: Output prices 41. Milk prices (the lower two lines and the left-hand axis) rose by 30 percent over the project, with a dip in 2015 that was recovered the following year. Meat prices, on the other hand, fell by 33 percent from 2011 to 2015, and then plummeted by 42 percent in the final 48 year of the project. Prices for 2016 were taken from an informal survey by the project’s MSTs rather than from community records, so the final year’s drop may not be entirely reliable, but in any case it is clear that farmers experienced a marked decline in livestock prices. 42. If the trends of rising milk prices and falling liveweight prices are reliable and continue in future, then this may encourage farmers to adjust their production systems more in favor of milk production. The increasing availability of artificial insemination, which the project supported, provides the possibility for farmers to introduce better dairy genes, though they will need to further improve feeding in order to get full advantage from this. With sheep, farmers might in theory be able to wean lambs earlier and take more of the milk for sale, but with milk representing only 10 percent of the total output value from sheep flocks, this is unlikely to be applied widely. Output at actual prices 43. The final chart shows actual output value at the prices prevailing in each year of the project: Figure 6: Value of livestock output at actual prices 44. This shows that in reality the overall value of livestock output fluctuated considerably, dipping in 2012, growing steadily for the next three years (driven by milk in 2013 and 2014 and by meat in 2015), and then falling sharply in 2016. Annual output value increased by 11 percent over the whole period, from AMD 27.4 billion to AMD 49 30.3 billion. The purple line again shows the without-project scenario and shows that if these communities had followed national trends, falling meat prices would have outweighed productivity gains and rising milk prices, so that farmers would have seen their total livestock incomes fall. The best measure of project impact is to compare the actual results of project communities against the without-project scenario that would otherwise have applied over this difficult period: this shows an aggregate benefit of AMD 21.9 billion (US$45million) over the five-year life of the project, with the majority of this benefit arising in the last two years. Whilst this is US$10million lower than it would have been if markets had remained stable, it brought a major income boost to project communities and allowed their livestock incomes to rise when they would otherwise have fallen. Set against project expenditure of around US$18 million for these two components, it represents a very good return on investment. Costs 45. The APIU provided the following table of total project expenditure, including direct project spending, beneficiary contributions and contributions from the GoA: Table 3: Final project expenditure Project Components 2011 2012 2013 2014 2015 2016 Total Component 1. Community Pasture/Livestock Management $ 843,917 $ 5,124,502 $ 3,602,558 $ 4,356,359 $ 2,676,718 $ 1,279,315 $ 17,883,369 System 1.1 Pasture/Livestock Management $ 348,428 $ 368,730 $ 238,134 $ 313,626 $ 229,155 $ 139,733 $ 1,637,805 Planning & Support System 1.2 Community Funds $ 495,489 $ 4,755,772 $ 3,364,424 $ 4,042,733 $ 2,447,563 $ 1,139,583 $ 16,245,564 Component 2. Strengthening $ 100,234 $ 788,561 $ 331,980 $ 364,453 $ 358,191 $ 405,680 $ 2,349,098 Institutions and Support Services 2.1 Agricultural Advisory Services $ 88,913 $ 680,200 $ 113,603 $ 87,153 $ 226,133 $ 16,118 $ 1,212,120 2.2 Community Grant Health $ 11,321 $ 108,361 $ 218,378 $ 277,299 $ 132,058 $ 389,561 $ 1,136,978 Services Component 3. Competitive Grant $ 144,225 $ 337,772 $ 349,191 $ 464,253 $ 476,459 $ 319,329 $ 2,091,229 Program Component 4. Project $ 174,272 $ 261,966 $ 235,642 $ 236,134 $ 170,008 $ 117,876 $ 1,195,897 Management, M&E 4.1 Project Management $ 174,272 $ 233,977 $ 188,119 $ 172,191 $ 143,763 $ 109,978 $ 1,022,299 4.2 Monitoring and Evaluation, $0 $ 27,989 $ 47,523 $ 63,943 $ 26,245 $ 7,898 $ 173,598 Audit Total Project Expenditures $1,123,960 $6,651,487 $4,519,371 $5,421,199 $3,681,375 $2,122,200 $23,519,593 Excluding Component 3 $1,118,422 $6,175,029 $4,170,180 $4,956,946 $3,204,917 $1,802,871 $21,428,365 Economic Rate of Return and progress versus targets 46. This section presents an economic analysis 12 based on the stream of costs and benefits over time, in line with the WB ICR guidelines (Operations Policy and Country Services, August 2006; last updated May 2011). It also presents a comparison of various physical indicators with targets set down before the project began. 12 The original ex ante analysis did not include a separate Financial Rate of Return, and hence there was no value to update. 50 Estimation of Economic Rate of Return 47. This section sets out the calculation of the project’s ERR over the 15-year period 2011-2025, in line with the 15-year period used for the ex-ante analysis. Costs are taken from Table 3. In order to estimate how the stream of benefits may develop in future, the following assumptions have been made:  The investments in access roads, watering points, veterinary services and pasture management systems should be durable and their benefits continue for the full 15- year period of the ERR calculation, at the average level calculated for 2015-26. It might be argued that benefits would continue to grow as pasture recovery continues, or alternatively that benefits night decline as infrastructure deteriorated and commitment to the new pasture-management principles declined without the continuing encouragement of the MSTs. This should be investigated by follow-up monitoring but for the now the assumption of constant benefits seems most reasonable.  Benefits of machinery should continue for its working life, assumed to range from around 10 years for motorized items such as tractors and combines, to 20 years for items such as ploughs that have few moving parts. This roughly corresponds with the ERR calculation period, so these benefits too are assumed to continue unchanged until 2025. 48. These assumptions result in an Economic Rate of Return of 116 percent, which compares favorably with the values of 82.2 percent in the ex-ante spreadsheet and 83.1 percent in the PAD. Sensitivity analysis 49. Two of the most critical assumptions in the ERR model are that: 1. In the absence of the project, livestock numbers and productivity would have changed from year to year in line with the national herd and flock; 2. The average annual benefits obtained over 2015-16 will continue at that level until at least 2025. 50. There is no immediately available alternative to the first assumption, though if statistics could be obtained for non-project communities in similar mountain areas, this might be a better comparison. 51. The significance of the second assumption can be tested against two alternatives: a) Optimistic: Annual benefits grow at 20 percent per year for the next 4 years (2017-20) as investments made late in the project come to fruition, and then remain at that level for 2021-25. This is consistent with the timeline presented above. b) Pessimistic: Annual benefits will continue at the 2015-16 level for the next 4 years, and then decline at 20 percent per year as commitment to the PUAs and 51 the principles of improved pasture management wane without continued project support. Table 4: Assumptions Scenario ERR Basic 116.3 % Optimistic 119.5 % Pessimistic 116.2 % 52. Neither assumption has a big impact on the calculated ERR, since they take effect relatively late in the 15-year period of analysis. The sensitivity analysis in the PAD allowed for 1, 2 or 3 years of delay in accruing benefits, which would reduce the ERR from 83.1 percent to 77 percent, 58 percent and 47 percent respectively. As shown above, the project had a “staggered start” over the period 2011-2014 so benefits were indeed delayed, but so were significant proportions of total project expenditure. The most appropriate benchmark against which to compare project performance probably lies between the “1-year delay” and “2-years delay” scenarios, with a projected ERR of 58-77 percent. Comparison with pre-project forecasts 53. The ex-ante economic model produced forecasts for a wide range of physical and financial variables, and a number of key figures were included in the PAD in Annex 3 (Results Framework and Monitoring) or Annex 9 (Economic and Financial Analysis). This section compares actual outcomes against these forecasts. Physical indicators versus forecasts 54. The following table and graph compare progress in physical indicators over the first five years of the project with that predicted in the ex-ante economic model and with the quantitative targets set out in Annex 3 of the PAD: Table 5: Progress in Physical Indicators Indicator & units Forecast PAD Actual Units Chang Indicator Yr 0 Yr 5 Change 2011 2015 Change forecast (actual) e Number, cattle Number 1,102 1,241 13% - 1,030 1,309 27% Number, sheep Number 1,784 2,096 17% - 1,307 1,662 27% Milk yield, cattle Litres/head/year 1,500 2,500 67% 20% 1,494 1,953 31% Milk yield, sheep Litres/head/year 35 60 71% 10% 60 85 42% Sale weight/LWG, kg/head (kg 180 203 13% 20% 119 149 25% steers gain/head/year) Sale weight/LWG, kg/head (kg 15 15 0% 5% 30 38 27% lambs gain/head/year) Source: Sheet “Comparison” 52 Figure 7: Actual versus targets Source: Sheet “Chart – Comparison” 55. The project significantly out-performed the projections of the ex-ante model with respect to livestock numbers and liveweight gain, but fell well short of its highly optimistic projections for milk yield. It exceeded the PAD target for both yields and liveweight gain. Key ratios versus forecasts 56. The following table compares project outcomes against a number of key ratios set out in the PAD13: Table 6: Comparison with PAD PAD Outcome Indicator Denominator Value Denominator Value NPV @ 12% - $ 58.5-59.8 m - $ 62.9 m per farm household in participating communities 19,300 $ 3,031-4,423 19,943 $ 3,152 per ha of pasture land in participating communities 153,333 $ 382-421 171,799 $ 366 per animal unit in participating communities 46,000 $ 1,272-1,470 214,414 $ 293 per livestock unit in participating communities - 106,918 $ 588 ERR - 83.1% - 116.3% B/C ratio (NPV-with/NPV-without) - 1.60 - 1.15 13 Comparison is slightly complicated by the fact that the numbers in paragraph 2 of PAD Annex 9 differ from those in the accompanying table, and none ties up with the figures calculated by dividing either of the NPV values by the quoted denominators, which are different again from the values in the ex-ante model. To be fair, such minor discrepancies are common in documents subject to multiple revisions and based on complex spreadsheets; this Economic Impact Evaluation is probably no exception. 53 B/C ratio (NPV-net benefit/NPV-costs) 3.69 3.80 57. The overall Net Present Value (NPV) came out around 5 percent higher than the forecasts. The differences in the calculated ratios are largely due to the fact that the implemented project was larger than forecast:  The number of farm households (using “livestock households” from the CARMAC project monitoring data) was very similar to the projection, so the NPV per household fell within the forecast range;  The area of pasture was 12 percent higher than expected, causing the NPV per hectare to fall slightly below projections;  The number of animal units was much higher than in the ex-ante model which, unusually, treated one adult sheep as being equivalent to an adult cow; the table above also shows a value using more conventional livestock units 14. Depending which of these measures is used, the livestock fund of the project communities was 2.3-4.6 times the size in the ex-ante model, and so NPV per animal unit is commensurately lower. 58. The PAD also quoted a rather unconventional Benefit/Cost (B/C) Ratio, calculated as the NPV of the with-project forecast (after subtracting project costs) against the NPV of the without-project forecast. Since the actual project was considerably larger than forecast (with-project NPV of US$477million versus US$157million), this B/C ratio is somewhat lower. 59. Usually the Benefit/Cost Ratio is calculated as the ratio of the NPV of project net benefits (with-project minus without-project) against the NPV of project costs; applying this measure shows the project performing slightly better than the forecast, in line with the slightly higher overall NPV. 60. The ERR exceeded the forecast by 40 percent, compared to a difference of only 5 percent in the NPV. This is due to the project benefits generally arising about one year earlier than forecast (apart from a dip in 2014): 14 Cows = 1.00; Other cattle = 0.65; Ewes = 0.15; Other sheep & goats = 0.10. Adapted from Goss (2012), Code of Good Agricultural Practice for Montenegro; final draft before preparation for print by the Montenegro Ministry of Agriculture and Rural Development. 54 Figure 8: Annual net benefits Impact of Component 3: Competitive Grants Scheme 61. The APIU contracted an independent evaluator to assess the CGP, based on site visits and calculation of actual costs and benefits. Separate reports were prepared for each of the seven CGP rounds, covering all 69 projects. 62. The independent consultant’s spreadsheets estimated benefits as the increase in income since the projects commenced, which can be expressed as a percentage of the investment cost to give a value for “Return on Investment”. However, the assessment visits typically took place 12-18 months after individual sub-project start and so in many cases the full impact will not yet have become apparent, and the spreadsheets do not record how long had elapsed between completing the investment and undertaking the evaluation visit. With these two limitations it is only possible to calculate a provisional ERR for this component, and so it is excluded from the overall ERR calculations presented above. 63. The simple measure of Return on Investment was in all cases positive and gives an overall value of 17.8 percent by the time the assessment visits were carried out, though with considerable variation between projects. It is important to note that most of these business would hope to see their return grow in subsequent years as the new ventures reach maturity. 55 Chain index methodology The two underlying assumptions are: 1. Until they joined the project, beneficiary herds will have experienced the same percentage change each year as the national non-project herd. 2. If beneficiary herds had not been in the project, they would have experienced the same percentage change each year as the national non-project herd. Upper-case letters represent original data; lower case represents variables to be calculated. Ty = total national herd in year y, from statistics Py = project herd in year y, from project records Ey = new entrants to project herd in year y, from project records (part of P y) ry = pre-project herd (future beneficiaries that have not yet joined the project) in year y, to be estimated by = beneficiary herd (project + pre-project) in year y: by = Py + ry ny = national non-project herd in year y: ny = T y - b y iy = index of change in non-project herd from year y-1 to year y: iy = ny/ny-1 iy = (Ty – (Py – Ey)) / (Ty-1 – Py-1) ry = 0 in final project year when no further entrants are possible, otherwise: ry = (ry+1 + Ey+1) / iy+1 cy = counterfactual, i.e. estimated size of beneficiary herd in year y without the project cy = by in first project year, otherwise: cy = cy-1  iy Counterfactual estimates may be made in this way for all additive variables (livestock numbers, total milk production, total liveweight gain). Counterfactual estimates of ratio variables (milk yield per head, liveweight gain per head) must be calculated for each year from the counterfactual estimates of quantities and livestock numbers for that year. my = project impact in year y: my = b y - c y Project impact may be estimated in this way for both additive. 56 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Alexander Astvatsatryan Consultant GGO03 ECSAR - Brian G. Bedard Sr. Livestock Specialist HIS Josef Ernstberger HQ Consultant ST GFA03 Artavazd Hakobyan Sr. Agriculture Economist GFADR Darejan Kapanadze Sr. Environmental Specialist GEN03 Doina Petrescu Program Leader AFCC1 Arman Vatyan Sr. Financial Management Specialist GGODR Arusyak Alaverdyan Sr. Agricultural Specialist GFA03 David Lugg Consultant GFA03 Garik Sergeyan Sr. Financial Management Specialist GG021 Armine Aydinyan Procurement Specialist GGO03 Sarah G. Michael Program Leader ECCSC Ara Karapetyan Consultant GFA03 Kosuke Anan Sr. Social Development Specialist GSU02 Lusine Grigoryan Financial Management Specialist GGO21 Marina B. Sahakyan Temporary ECCAR Sophia Georgieva Social Development Specialist GSU03 Deepal Fernando Consultant GGO03 Caroline Plante Livestock Specialist GFA04 Dina Umali-Deininger Practice Manager GFA07 Benoit Blarel Practice Manager GEN05 Jeren Kabayeva Agricultural Specialist GFA03 Pierre Olivier Colleye Sr. Agriculture Economist GFA13 Valencia Copeland Program Assistant GFA03 Asad Alam Country Director MNC03 Ahmed A.R. Eiweda Lead Urban Specialist GSU08 Supervision/ICR Arusyak Alaverdyan Sr. Agricultural Specialist GFA03 Bekzod Shamsiev Sr. Agriculture Economist GFA03 Darejan Kapanadze Sr. Environmental Specialist GEN03 Sarah G. Michael Program Leader ECCSC Garik Sergeyan Sr. Financial Management Specialist GG021 David Lugg Consultant Josef Ernstberger HQ Consultant ST GFA03 Doina Petrescu Program Leader AFCC1 Alexander Astvatsatryan Consultant GGO03 ECSAR - Brian G. Bedard Sr. Livestock Specialist HIS Artavazd Hakobyan Sr. Agriculture Economist GFADR 57 Arman Vatyan Sr. Financial Management Specialist GGODR Ghada Youness Sr. Counsel LEGLE Nicolas Ahouissoussi Sr. Agriculture Economist GFA01 Martin Lenihan Sr. Social Development Specialist GSU02 David Lugg Consultant GFA03 Armine Aydinyan Procurement Specialist GGO03 Ara Karapetyan Consultant GFA03 Kosuke Anan Sr. Social Development Specialist GSU02 Lusine Grigoryan Financial Management Specialist GGO21 Marina B. Sahakyan Temporary ECCAR Sophia Georgieva Social Development Specialist GSU03 Deepal Fernando Consultant GGO03 Caroline Plante Livestock Specialist GFA04 Qun Li Sr. Agriculture Economist GFA05 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY 10 24.18 94.62 FY 11 28.20 116.71 Total: 52.38 211.33 Supervision/ICR FY 11 9.61 31.27 FY 12 30.10 165.93 FY 13 23.56 113.35 FY 14 20.14 75.86 FY 15 17.21 95.55 FY 16 17.04 80.21 FY 17 12.00 70.48 Total: 172.04 632.65 58 Annex 5. Beneficiary Survey Results 1. In addition to regular progress reports and the Project Completion Report (PCR) by the client, the project also commissioned a number of evaluation studies, surveys and collected a wide range of data on project activities and impacts. The main studies are summarized below. Overall Project Impact Assessment. 2. Economic Impact Evaluation of the CARMAC project. End-of-project assessment. The main task of this economic impact evaluation was to quantify the costs and benefits, and compare them over time to calculate the project’s Economic Rate of Return (ERR). A secondary task was to analyze the intermediate impacts to help understand how the project achieved or failed to achieve its stated objectives, and from this to draw conclusions that may be used to improve the effectiveness of CARMAC 2 and similar projects. The approach of this evaluation was to use a combination of project data and national statistics to estimate how key variables in beneficiary communities would have changed without the project. Impact was calculated as the difference between these “with-project” and “without-project” scenarios. Assessments of Component 1 – Pasture management 3. Impact Assessment of the “Community Pasture & Livestock Management System”. This Impact Assessment looked at the overall impact of the “Community Pasture & Livestock Management System,” i.e. Component 1 and some aspects of Component 2, based on a regionally stratified random sample of 400 beneficiary households. The study found considerable variation amongst respondents, with around one-third reporting increased livestock numbers and half reporting a decrease; overall the sample reported a net decrease in livestock, in contrast to the modest gain shown by the 81 project communities in aggregate. Households that did expand their flocks or herds gave “Pasture improvement and increased equipped-ness with agricultural machinery” as the main reason – one example of the ability of a detailed sample survey to investigate causation and perceptions in a way that is not possible with macro-level impact assessments. 4. Productivity per head grew markedly, with milk yields up 18-21 percent and meat yield up 6 percent for lambs and 23 percent for calves. The sampling methodology used in the survey appears sound and the sample is reasonably large, so it is unlikely that the differences were just a result of chance selection. The reports’ authors speculated that farmers might have deliberately under-reported livestock numbers and output, perhaps through fear of taxation or some other kind of charge, which would explain the discrepancy across several different indicators. 5. The survey used the expenditure method to estimate total household income over the period 2011-16. As a result of activities implemented under the project in the target communities, the average income per livestock household had increased by about 59 US$1,000 (AMD 464,914) from Year 2011 to 2016. The calculations made by the APIU that are based on the comparative analyses of 2011-2016 livestock number, milk yield and weight gain. The details see the table below: CARMAC Impact Assessment on Livestock Household Income (AMD) Beneficiary households’ income in 2011 and 2016 for the Component 1 Indicators 2011 2016 Change Total income of 400 surveyed households 1,746,878,996 1,932,844,413  11% Average annual income per beneficiary household 4,367,197 4,832,111 Per capita annual income of beneficiary households 802,056 913,443  14% Per capita monthly income of beneficiary households 66,838 76,120 6. The survey also used the expenditure method to estimate the household income and the share coming from livestock, which fell from 37 percent to 30 percent over the period 2011-16. This comprised an 11 percent fall in total livestock earnings (although it is still higher than the baseline income) and a 23 percent increase in non-livestock earnings. One very important conclusion from these data is that even in remote mountain communities, households gain around two-thirds of their income from sources other than livestock production (e.g. financed by component 2 and 3 to diversify the income sources); future livelihood projects might want to investigate what these sources are and consider how they might be increased, before automatically focusing on livestock. 7. Summary/final report of impact studies on changes in vegetation and environmental indices in pasturelands of the RA 63 communities. The CARMAC project carried out two different treatments for environmental effects on pasture:  Main CARMAC project: Development and implementation of Pasture Management Plans including: a. Rotational grazing, including temporary exclusion of grazing from over- grazed areas; b. Increasing usage of remote pastures, through provision of access roads and stock watering points, to even out grazing pressure between nearby pastures (previously over-grazed) and remote pastures (previously under-grazed); c. Gradual increase in livestock numbers over the duration of the project. d. Small-scale plots under a parallel project funded by the Global Environmental Facility (GEF): Renovation of seriously degraded pastures through: e. Removal of shrubs and stones; f. Reseeding, including harrowing, seed drilling and fertilizing; g. Total exclusion of grazing until the pastures recovered. h. It should be noted that the GEF plots were deliberately selected from the most degraded areas and thus significantly different from the CARMAC project plots before any treatments began. 60 8. This study reported the results of measurements of vegetation cover, composition, dry matter yield and estimated carbon fixation, both before and after each of the two treatments. Measurements on the CARMAC project pastures showed significant increases in vegetation cover, together with increases in peak dry matter that were not statistically significant; greater improvements were found on the GEF plots. Carbon fixation in above- ground biomass was calculated as a constant percentage of dry matter and hence showed the same responses. 9. This work does not give a direct indication of the impact of improved pasture management on greenhouse gas emissions, as it deals only with above-ground vegetation at the peak of the season before that vegetation is harvested by cutting or grazing and continues through the carbon cycle within the livestock sub-system. However, it does provide some detailed and valuable information that may be used as inputs for more comprehensive system modeling, including modeling of the impacts of improved pasture management and restoration measures. The wide range of measurements taken in different strata (“nearby,” “middle,” “remote” and “mountain” pastures) constitute a significant contribution to understanding of this complex farming system. Assessments of Component 2.1 – Advisory services 10. Impact Assessments of Marz Agricultural Support Centers. A series of survey- based assessments were carried out. They found that farmers were generally very positive about the support they got from the Marz advisors, and most considered that this support had helped to protect farm incomes during difficult market conditions. The most common subject of advice was crop production, with livestock husbandry in second place and a very variable demand for business advice. 11. The surveys collected slightly more detailed household income data than that of the impact assessment discussed in the previous section: 85 percent of surveyed households had some income from agriculture, 65 percent had income from employment and 57 percent received pensions or student benefits. As a share of total household income, sales of agricultural products and services was the most important, at 38 percent, followed by employment income at 35 percent. These two sets of figures show that most farms are dependent on several different incomes sources, not just agriculture. Over the surveyed period, total household income rose whilst farm income grew in some Marzes and shrank in others, declining as a share of total income. 12. Technology Assessment Projects. A field survey and assessment on the impact of the local technical advisory system had been conducted. As part of its advisory sub- component, the CARMAC project supported 148 Technology Assessment Projects (TAPs) – trials of new technologies with a strong demonstration element, usually carried out with support from regional Marz Agricultural Support Centers (MASCs). The majority of the TAPs focused on crop production, with 20 on livestock-related issues, and the most common elements were the introduction of crop varieties, cultivation techniques and methods of crop protection that are already in use elsewhere but not yet common in Armenia; the focus was thus more on technology transfer and demonstration than on original research – a focus that is quite appropriate to the immediate needs of most farmers. 61 13. The number of projects by theme and year is summarized in the PCR. A textual database presents the main elements and outcomes in each year, with the large majority of outcomes appearing quite positive. Some entries give quantitative or economic assessments, though without statistical analysis. An earlier review by the consultant under the auspices of an EU Budgetary Support Disbursement Review found that the TAPs were one of the most highly regarded elements of the advisory program. 14. The survey and assessment results showed that farmers clearly value the service as evidenced by the very positive impact assessment results, including 94 percent of farmers highly valuing the advisory services; 89 percent indicating that they had adopted technological change as a result of advisory activities; 75 percent indicating that they had an increase in production and 80 percent an increase in farm incomes; and 64 percent indicating their willingness to pay for services. Reflecting the importance and positive assessment of the advisory services, the Ministry prepared a strategy for continuing development formally accepted by the GoA under Order No. 1516, December 2013. Assessments of Component 2.2 – Veterinary services 15. Evaluation of Veterinary Service Centers established under the CARMAC project. The project supported establishment of five Veterinary Service Centers (VSCs), of which three were operational at the time of this evaluation (September 2016), one had just been completed and one was still under preparation. The evaluation focused on the three operational centers, together with one created earlier by the CARD Foundation. The main findings were:  The centers provide a varied range of goods and services, typically including agricultural inputs and machinery as well as veterinary services, advice and artificial insemination.  Centers typically have a customer base of 600-700 farmers, with around 400 visitors per month, most usually to purchase supplies and ask for advice.  The centers also offer training in various subjects, but farmers are less aware of or interested in this aspect.  Centers themselves are rather small businesses, typically with just a couple of vets, and their success depends crucially on the approach and business abilities of the principal.  The centers cooperate well with other (usually state-employed) community veterinarians, supplying them with inputs and calling on their services when demand exceeds the capacity of the center staff. 16. Surveys of customers, veterinarians and community leaders showed a high degree of satisfaction with the service centers, but the report conveys the impression that they are still struggling to find their niche in a farming community that tends to aim for cost minimization rather than profit maximization, and is not able or accustomed to spend heavily on inputs and advice. It also observed that two other projects have provided veterinary services and artificial insemination, respectively, free of charge, which 62 compounds the difficulties of establishing a sustainable business model. The PCR notes that the veterinary sub-component met most of its quantitative targets (numbers of centers established, veterinarians trained, communities served, etc.) and achieved a high degree of beneficiary satisfaction. Assessments of Component 3 – Competitive Grants Program 17. Assessment of Competitive Grants Program projects. This assessment covered all 69 CGP projects from 7 rounds of tendering. The methodology used field visits and structured discussions with principal beneficiary and farmer group members, plus development of standardized spreadsheets to assess the project costs and impact on net income. Assessment also looked at the level of technology transfer (a specific element of the program), at socio-economic impact and at any potential adverse environmental impacts. Results from this study, which was completed in May 2016, are used in section to assess the economic impact of Component 3. 18. All projects had helped create viable business opportunities, and provided significant income gains to the beneficiaries ranging from 5 to 60 percent, and had increased employment opportunities from 2 to 20 people under each project. As result of the technology transfer activities, the evaluations indicated that elements of many grants had been emulated by other beneficiaries by the end of the grant period, with further adoption likely to take place over time. Examples of innovations included improved breed selection and animal husbandry; new dairy products and more developed contracting approaches with producers; improved storage, processing, packing and marketing of fruit and vegetable products; and introduction of various non-traditional poultry and crop products. 63 Annex 6. Stakeholder Workshop Report and Results Consultation with stakeholders 1. The project conducted formal consultation with stakeholders on multiple occasions and in several different ways including stakeholder surveys, focus groups and workshops.  The Impact Assessment of the “Community Pasture & Livestock Management System” included both a stratified survey of beneficiaries and a qualitative survey of the mayors of 21 project communities;  The Evaluation of Veterinary Service Centers established under the CARMAC project also consulted both direct beneficiaries and community leaders;  The Impact Assessments of Marz Agricultural Support Centers consulted three different groups: randomly selected farmers, lead farmers, and community leaders. 2. All three of these studies involved both closed and open questions, within which respondents could present their views and suggestions for improvement. Workshops with the Ministry of Agriculture and industry stakeholders included discussion of both extension and opportunities for further development of the dairy sector:  The discussions on extension focused on ways to improve the efficiency and effectiveness of the extension service through restructuring, development of common resources and greatly increased use of activities that can reach large numbers of farmers, such as demonstration plots and use of mass media.  The Round Table with ministry staff and dairy processors looked at the challenge of improving milk hygiene so as to protect consumers, widen processing options and increase international competitiveness in dairy products. It explored options for public-private cooperation through an Armenian “Milk Quality Initiative,” based on models employed successfully in Serbia and the northern part of Cyprus, and proposed next steps for its implementation. 64 Annex 7. Summary of Recipient 's PCR and/or Comments on Draft ICR15 Introduction 1. The Government of Armenia (GoA) received a credit from the International Development Association (IDA) to implement the Community Agricultural Resource Management and Competitiveness (CARMAC) Project. The total project cost is US$21.33 million, of which US$16.0 million is financed by an IDA credit. The project consists of four components: I. Community Pasture/Livestock Management System, II. Strengthening Agricultural Support Services, III. Competitive Grants Program, and IV. Project Management and Monitoring and Evaluation. 2. The Project Development Objective (PDO) is to improve productivity and sustainability of pasture/livestock livelihood systems in selected communities. The GoA is represented by the Republic of Armenia (RA) Ministry of Agriculture (MoA) and the Agricultural Projects Implementation Unit (APIU) State Agency. This Project Completion Report (PCR) has been prepared by the APIU on behalf of the RA MoA. Relevance 3. Relevance of the CARMAC project has been assessed from the following aspects: a. Relevance of the project objective to the strategy and priorities of Armenia, b. Relevance of the project to the development priorities of Armenia, c. Relevance of the implemented components and activities to the project objective, d. Relevance of allocated resources. From the viewpoint of all aspects the CARMAC relevance was assessed to be high. Project Implementation Output and Outcome Achievements Component 1: Implementation Results 15 The Recipient’s Project Completion Report was prepared by APIU and approved by the MoA, and submitted to the WB before the project completion in September, 2016. Therefore, the final achieved output and outcome results by component listed in the above tables and document were not the final figures at the project completion. However, all final completed project output and outcome results had been collected by APIU during the WB ICR mission and already reflected in the final WB ICR report. 65 4. Implementation of the Component 1 consisted of the following major activities: preparation and introduction of the OM, selection of the target communities and establishment of cooperatives, organization of the transfer of pasture areas’ management responsibility to cooperatives, preparation of the Community Pasture Management and Livestock Development Plans (CPMLDPs), funding eligible investments, support the provision of trainings to PUAs, support in management of cooperatives, support to fodder production, application of protective and natural resource rehabilitation measures and mitigating grazing pressure, following the environmental and social safeguards. 5. Currently it is still early to evaluate the long-term impact the project interventions under Component 1, especially in terms of the impact of rehabilitation of degraded pastures, their environmental impact, the mechanism of managing pastures based on sustainable pastureland use approaches set in the pasture management plans beyond the project lifecycle, etc. However, the impact of other outputs can be estimated. Set up of stock watering system (construction of water lines and watering points) on 121,500 ha pasturelands, increased livestock productivity speaks for themselves. Emphasis should be put on the positive effect the participatory approach adopted by the project has had throughout the planning, development and implementation of CPMLDPs. 6. Implementation of this Component involved some building of human capital of target local population. More than 11,500 pasture users participated in trainings, workshops and group discussions with a view to make them more efficient in natural resource management as well as enhance/improve their business oriented thinking. The trainings have been essential learning station for pasture users, where they acquired knowledge and proficiency in dimensions much needed for self-organization and self- reliance in further arrangement for managing the community pastures through the already operational PUAs. Thus, the project interventions had significant positive social impact on local population and on biodiversity. Main outcomes of the CARMAC project Component 1 implementation are compiled in the table below: Table 1 - Achievements of the Component 1 (as of August 31, 2016) Feature Outcome 1. . Number of communities 81 = 55+26 2. Number of population addressed 133,000 3. Area covered, hectare 176,000 4. Length of waterlines, km 199.9 5. Number of water points 243 6. Milk yield increase for cattle, percent 137 7. Weight gain for cattle, percent 127 8. Increase in community budget revenues, percent 199 9. Winter fodder requirements met, percent 198 10. Increase in profit made from sales of livestock products, percent 268 66 Overall, an enabling environment for sustainability has been established both by the CARMAC project interventions and partly it is up to communities to portray self- organization, which is successfully and remarkably exercised by certain communities. The latter could be appropriate examples of replication by lagging communities. Component 2 Implementation Results Sub-component 2.1 Results 7. Implementation of the sub-component 2.1 consisted of the following major activities: preparation of the OM, implementation of technology assessment projects (TAP) program, testing the advanced communication technologies, provision of equipment for Marz Agricultural Support Centers (MASCs), supporting the provision of trainings to MASCs by the Republican Agriculture Support Center (RASC), funding the provision of trainings to farmers, overall contribution to the reform of the agricultural advisory system and cost recovery of the MASCs and the RASC. 8. Implementation of the sub-component 2.1 largely achieved the targets assigned to its implementation. In particular: a. Adoption rate (of advanced agricultural practices and technologies) by farmers in targeted communities was targeted to be 90 percent, and that target was achieved as of the end of the 2014. b. Improved outreach and performance as measured by increased share of revenue from contracts was targeted to be respectively on 10 percent for MASCs and 14 percent for the RASC. Both targets have been achieved and surpassed. c. Number of implemented TAPs was planned to be 150 and 148 were successfully completed. d. Provision of the agricultural machinery and equipment was surpassed; additional support was allocated. e. 100 percent of all farmers that received trainings from the MASCs rated the knowledge they gained to be either very useful or useful. f. The GoA made formal strategic decision to continue the development of the extension services system in Armenia. Implementation of the sub-component 2.1 achieved and even surpassed its targets. The results are assessed as Satisfactory. Sub-component 2.2 results 9. Implementation of the sub-component 2.1 consisted of the following major activities: mobilization of community veterinarians, establishment of VSCs, provision of mobile veterinary clinics, coordination of smooth operations of the VSCs. Implementation 67 of the sub-component 2.2 largely achieved the targets assigned to its implementation. In particular: a. The number of trained and certified community veterinarians providing services was initially assigned to be 48. The implementation resulted in training of 100 veterinarians, of which 67 were certified and provided with all necessary veterinary equipment, 15 of whom also received artificial insemination equipment. b. Establishment of five VSCs within the scope of the project. Over 70 communities are already served by the first three VSCs and this number will grow along with operation of newly established two VSCs. c. Provision of three mobile veterinary clinics at three of the permanent VSCs. d. Beneficiaries (high) satisfaction from the various aspects of the VSCs’ operation varies in the range of 81 percent and above. Implementation of the sub-component 2.2 achieved its targets. The results are assessed as Satisfactory. 3.3 Component 3 Implementation Results 10. Implementation of the Component 3 consisted of the following major activities: preparation of the OM, implementation of preparatory works (including awareness raising and information dissemination measures), formation of necessary structures (such as CGP Secretariat, evaluation committee, database of service providers, etc.), implementation of 7 rounds of tendering, selection of grantees and their funding. Achievements of the CGP implementation were the following: Table 2 - Achievements of the CGP implementation as of the August 31, 2016 Number of Roun Grant amount Applicants cash Direct Indirect completed d disbursed, $ contribution, $ beneficiaries beneficiaries16 projects I 8 150,076 72,052 66 4,850 II 9 166,477 78,487 33 10,440 III 7 134,909 61,325 21 3,270 IV 6 116,875 43,657 24 4,850 V 7 139,850 57,377 19 1,155 VI 16 314,667 145,926 100 5,500 VII 16 311,542 118,213 71 3,410 T 69 1,334,396 577,037 334 33,475 16 Farmers and other beneficiaries who did not directly benefit from the projects (were not among the owners, their families, and employees of grantees), but obtained additional knowledge, were given opportunity to cooperate, etc. 68 11. Implementation of the Component 3 largely achieved the targets assigned to its implementation. In particular: a. Percentage of grants completed with satisfactory rating was planned to be 80 percent. Results of the external/independent evaluation of completed projects are substantially higher – 100 percent. b. The CGP provided grant funding to 69 beneficiaries against the assigned number of 70. c. 247 technology transfer seminars have been conducted by the grantees with about 11,138 participants. Implementation of the Component 3 achieved its targets. The results are assessed as Satisfactory. Results Framework and PDO Achievement 12. Monitoring information on achievement of the project results has been regularly collected through the whole process of the implementation. The detailed expected targets agreed at project appraisal had been achieved as follows: Table 3 - PDO level indicators PCR as of End Baseline as 01.09.2016 Target as Indicator Name (Unit) of of 01.04.2011 30.09.2016 Increased livestock productivity measured by milk 100% 137% 1. 120 % productivity, percent and kg/year, (for cattle), 1,428 1,953 Increased livestock productivity measured by milk 100% 129% 2. 110% productivity, percent and kg/year (for sheep) 66 85 Increased livestock productivity measured by growth 100% 127% 3. 120% rates of animals, percent and gram/day (for cattle) 320 408 Increased livestock productivity measured by growth 100% 127% 4. 105% rates of animals, percent and gram/day, for sheep) 81 103 Increased efficiency of communal pasture management, as measured by increased communal 100% 151% 5. 130% budgetary revenues from lease of pastures (average 328,893 495,474 per community), percent Increased sales from livestock by livestock raising 100% 226% 6. 120% households, percent and AMD/HH 532,147 1,204,567 Increased Pasture Management Effectiveness (scoring 7. 0 57 60 system), Number 69 0 Clients who have adopted an improved agricultural 8. As of 1,048 774 technology promoted by the project, Number 13.08.2012 0 Clients who adopted an improved agricultural 9. As of 324 220 technology promoted by project – female, Number 13.08.2012 Table 4 - Intermediate Results Indicators End Baseline as PCR as of Target as Indicator Name (Unit) of 01.09.2016 of 01.04.2011 30.09.2016 Component 1: Community Pasture/Livestock Management System – Indicators Number of pasture management plans developed and 1. 0 81 55 agreed by the communities, Number 100% 711% 140% 2. Areas of pastures and grasslands leased, ha 24,742 176,000 34,640 Number of farmers’ associations established, 3. 0 91 46 Number Percentage of winter fodder requirements met, 4. 45% 89%* 80% percent, * Component 2: Strengthening Support Services – Indicators Adoption rate by farmers in targeted communities, 5. 70% 92% 90% percent MASCs - Improved outreach and performance as measured by 6% 14% 10% 6. 18% increased share of revenue from contracts, percent RASC - 14% 10% 66 (1 Number of trained and certified community 7. 0 person 48 veterinarians providing services, Number died) Component 3: Competitive Grants Program – Indicators Percentage of grants completed with satisfactory 8. 0% 100% 80% rating, percent Number non-recipients adopting similar technical 9. 0 215 250 innovations outside the grant scheme, Number 13. Overall, the project met and exceeded all implementation outcome targets, as such the project can be considered as successful in reaching all of its PDOs indicators. In 70 addition, the project objectives and design were and continue to be highly relevant to the country’s priorities and needs in agriculture livestock development and management. Thus, a Satisfactory rating is proposed for the overall outcome. Government email dated April 26, 2017: After analysis of the document, we do not have any comments or suggestion. However, we have informed that the Foundation, previously Project Implementation Unit, which has been mandated organization of the Ministry to implement the project to explore the documents in details and provide their feedback. Armen Harutyunyan | Deputy Minister Ministry of Agriculture of the Republic of Armenia Government Building 3 | Yerevan | 0010 | Armenia T (direct) +374 (11) 58 02 56 | F +374 (10) 54 51 49 E aharutyunyan@minagro.am | W www.minagro.am 71 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders N.A. 72 Annex 9. List of Supporting Documents 1. Recipient’s Project Completion Report, September 2016 2. Economic Impact Evaluation of the CARMAC project 3. Assessment of Competitive Grant Scheme projects 4. Impact Assessment of the Community Pasture & Livestock Management System 5. Evaluation of Veterinary Service Centers 6. Impact Assessments of Marz Agricultural Support Centers 7. Project Appraisal Document – May 18, 2011 8. Project ISRs 9. Supervision Missions’ Aide Memoires 73 MAP 74