Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) Report Number: ICRR0021981 1. Project Data Project ID Project Name P145162 Pasture Management Improvement Project Country Practice Area(Lead) Kyrgyz Republic Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-55210,IDA-H9760 31-Mar-2019 10,913,213.94 Bank Approval Date Closing Date (Actual) 15-Jul-2014 28-Jun-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 15,000,000.00 0.00 Revised Commitment 12,208,160.00 0.00 Actual 10,913,269.82 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hassan Wally J. W. van Holst Christopher David Nelson IEGSD (Unit 4) Pellekaan 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) as stated in the Project Appraisal Document (PAD, para 16) was identical to the one stated in the Financing Agreement (FA, p. 5) and aimed to: "improve community-based pasture and livestock management in the project area." Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) For the purpose of this review the PDO will be parsed into it two elements, namely:  To improve community-based pasture in the project area which will be referred to in Section 4 of this review as Objective 1  To improve community-based livestock management in the project area which will be referred to in Section 4 of this review as Objective 2 b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The PDO was supported by the following three components: 1. Community Based Pasture Management (appraisal cost: US$11.44 million, actual cost: US$8.7 million). This component would support improved pasture governance and technical capacity for pasture management by: (i) improving public awareness of the pasture related legislation; (ii) demarcating internal pasture boundaries, resulting in more accurate land tax charges and resolution of land use disputes; (iii) building more inclusive Pasture User Union (PUU) governance and increasing women’s participation in decision making, resulting in more equitable access to pastures; (iv) improving the technical competency of PUUs, rural advisory service providers and state agencies (Pasture Department, State Agency for Environmental Protection and Forestry and the Livestock and Pasture Research Institute) resulting in improved community pasture management planning and implementation; (v) demonstrating to pasture users of the benefits of aligning stocking rates with pasture carrying capacity, resulting in reduced pasture degradation and higher pasture productivity; (vi) demonstrating to pasture users the benefits pasture improvement, winter fodder and feed improvement, pasture infrastructure improvement and advisory services, leading to greater willingness to pay for PUUs services; (vii) funding community based investments in pasture, feeding and livestock improvement; and finally by (viii) introducing community based pasture management into forestry enterprise pastures. This component included three sub-components: 1.1. Community Based Pasture Management and Investment. 1.2. Strengthening State Pasture Institutions. 1.3 Forestry Enterprise Pasture Management and Investment 2. Community Based Animal Health and Husbandry Services (appraisal cost: US$1.85 million, actual cost: US$0.8 million). This component would support the following activities: (i) building the capacity of each PUU’s Animal Health and Husbandry (AHH) Group - a group of up to seven PUU members appointed by the PUU pasture committee to plan and oversee implementation of PUU animal health and husbandry activities - resulting in improved AHH plans; (ii) equipping and training private veterinarians and facilitating contracts between veterinarians and PUUs, resulting in stronger implementation of AHH plans; Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) (iii) demonstrating to pasture users the benefits of veterinary services and improved animal health, resulting in greater willingness to pay for veterinary services; (iv) supporting the establishment of rayon associations of private veterinarians to provide some centralized input supply services and represent their interests to government; (v) supporting the development of training material for veterinarians in cooperation with the Veterinary Chamber and the Kyrgyz National Agrarian University; and (vi) building the capacity of the Veterinary Chamber to support the assessment and professional development of private veterinarians, resulting in higher private veterinary standards. This component included two sub-components: 2.1. Animal Health and Husbandry Services Planning and Investment. 2.2. Private Veterinarian Development. 3. Project Management (appraisal cost: US$1.71 million, actual cost: US$1.40 million). This component would finance project management activities of (i) Ministry of Agriculture and Melioration Agricultural Projects Implementation Unit (APIU) and (ii) the Community Development and Investment Agency (ARIS), including budgeting, work planning, financial management, procurement and monitoring and evaluation including a baseline survey, to be repeated at mid-term and project completion, as well as reporting on implementation progress and on the impact of the PDO and intermediate outcome indicators. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The total project cost was expected to be US$15.0 million, including total base costs of US$14.6 million, and price contingencies of US$0.4 million. This amount was revised down to US$12.21 million as part of the February 2019 restructuring. The actual total cost was US$10.91 million. Financing. The project was financed through an IDA grant for an amount of US$6.75 million and an IDA credit for an amount of US$8.25 million with standard IDA terms and a final maturity of 38 years including a grace period of 6 years. The actual amounts disbursed according to the ICR (p. 7) were US$5.48 million for the IDA credit (71%) and the grant was fully disbursed (100%). The difference in the IDA credit amount was due to a cancellation of US$2.8 million, which was made at the Government's request due the cancellation of sub-component 1.3 (Forestry Enterprise Pasture Management and Investment) since its activities were taken over by the newly-approved Integrated Forest Ecosystem Management Project (P151102). Also, there was a ten percent appreciation of US$ to SDR that happened over the course of project implementation. According to the ICR (footnote #8): "The SDR/US$ exchange rate was 1.38 at project closing, compared to 1.53 at project appraisal, which reduced the available amount of US$." Borrower Contribution. The project was fully funded through the above-mentioned financing instruments with no borrower contribution. Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) Dates. The project was approved on July 15, 2014 and became effective 13 months later on August 27, 2015. The Mid-Term Review (MTR) was conducted on August 23, 2017, which was in line the specified date in the PAD- start of the third year of the project. The project was expected to close on March 31, 2019. The actual closing date was three months later on June 28, 2019. According to the ICR (para 15) the extension was needed to "capture the spring season of 2019, and to complete several activities, notably pasture, feeding and livestock improvement micro-projects" and "to partially compensate for start-up delays." The project was restructured once (level 2) on February 26, 2019, when the amount disbursed was US$9.06 million, in order to extend the loan closing date by three months, modify components and associated cost, a partial cancellation of the IDA credit in the amount of US$2.8 million (Restructuring Paper, para 7), reallocation between disbursement categories and accommodate changes in the implementation schedule. 3. Relevance of Objectives Rationale Context at Appraisal. Agriculture is a key priority for the Kyrgyz Government because of its contribution to poverty reduction and rural employment, promoting rural-urban income equality and food security. Real GDP growth in agriculture, driven by irrigated agriculture (1.3 million hectares) and pasture-based livestock production (9 million hectares), averaged 2.27% per annum between 2003 and 2011 before declining to 1.2% per annum in 2012. Livestock production makes up around half of agricultural GDP and is dominated by household and small farm production. Livestock are important for rural household food security, provision of regular income and acting as a social safety net because they can be sold in times of hardship and economic distress. There is great potential to increase livestock productivity through addressing fundamental animal health and nutrition constraints. The project would support PUUs to contract private veterinarians to implement the animal health and husbandry plans, which is an important approach to raise livestock productivity and protect human health. At appraisal, objectives were in line with the Government priorities for the livestock sector. The Government's Strategy for Development of Veterinary Services (2008) called for supporting the development of private veterinary services by contracting private veterinarians for the public veterinary service, including vaccination and by establishing a Veterinary Chamber to register and in the future regulate private veterinarians. Objectives were also in line with the Bank's Country Partnership Strategy (CPS, 14-17). The CPS featured support for improved governance as a single pillar. The CPS aimed to make the state more accountable to its citizens and, at the same time, to strengthen citizens’ voice in the activities of the state. This was to be achieved in three dimensions of the relationship between the state and its citizens, respectively: (i) as provider and consumer of essential public services, (ii) as enabler and developer of private business, and (iii) as steward and user of natural resources and physical infrastructure, which are key public goods. In this respect, the project featured a community-based approach through which improved stewardship of natural resources would be achieved as envisaged under the CPS. At project completion, objectives continued to be in line with the Government priorities for the livestock sector. This is emphasized through three important developments: first, a draft law to amend the Law on Pastures has been cleared by the Government and is under discussion by Parliament; second, a draft Resolution of the Government of the Kyrgyz Republic on measures to implement the Kyrgyz Republic Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) Pasture Law has also been cleared and is expecting approval; and third, a Program on Development of Pasture Farming and Livestock Breeding for 2019–2023 has been finalized with inputs from several ministries. Objectives were also in line with the Bank's Country Partnership Framework (CPF, FY19- FY22). While the focus of the CPF shifted from governance improvement to diversified, export-oriented, inclusive, and sustainable growth, addressing governance challenges through citizen participation and measuring satisfaction continued to be a CPF priority (CPF, p. 12). Objectives were also in line with the third focus area of the CPF (Enhance Economic Opportunities and Resilience, p. 24) where the project supported sustainable growth in the livestock sector. The statement of the PDO in the financing agreement to "improve community-based pasture and livestock management in the project area" is vague but the meaning of “improve” is elaborated by the PDO indicators (ICR, para 9). The statement also lacked any connection to the Bank's higher-level objectives, namely, reducing poverty and enhancing shared prosperity. Based on the above-mentioned information, Relevance of Objectives is rated Substantial rather than High. This rating reflects the vague ambition of the PDO to “improve community-based pasture and livestock management in the project area” which is nevertheless elaborated by the precise PDO indicators, a decrease in relevance at completion, and the lack of connection of the PDO to the Bank’s higher level objectives of poverty reduction and enhancing shared prosperity. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve community-based pasture in the project area. Rationale Theory of Change (ToC). There are a number of critical elements that provide the basis for the theory of change for this objective. The project supports the preparation and implementation of pasture management plans including training and legal support to pasture communities which also involves pasture boundary demarcation and pasture inventory. The project would also provide sub-grants to improve pasture infrastructure such as rural roads, water points, among others. In addition, the project would provide equipment and training to state institutions responsible for advisory services for pasture management Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) to strengthen their capacity. Community-based pasture management is expected to improve through the enhanced implementation of the community pasture management plans. Also, with better pasture management and planning, the area of utilizable pastures is expected to increase. As a long-term outcome, the project would increase pasture productivity and improve the livelihoods of the rural population. Overall, the ToC included logical assumptions and the stated activities were clearly linked to the PDO. However, a critical assumption that was overlooked relates to timely implementation of pasture-related activities to achieve the envisaged increase in utilizable pasture area. (a) To Improve Community Based Pasture Management in the Project Area Outputs The information below is based on the ICR (Annex 1) unless referenced otherwise.  A single format of the community pasture management and use plan (CPMUP) has been developed for the Pasture User Union (PUU).  PUU assessment tool based on implementation of CPMUP has been developed.  140 community pasture management plans were supported from 2017 to 2019 compared to a target of 112 (target exceeded).  Legal awareness and Regulatory Reform: the draft Law “On Amendments to Certain Legislative Acts of the Kyrgyz Republic (to the Land Code of the Kyrgyz Republic, the Kyrgyz Republic Law “On Pastures”) has been developed and cleared with Ministry of Finance, the Ministry of Economy and the State Tax Inspectorate of the Kyrgyz Republic, State Agency for Local Self-Government and Inter- Ethnic Relations (SALSGIER) of the Kyrgyz Republic, and the Ministry of Justice; the draft Resolution of the Government of the Kyrgyz Republic “On Amendments to the Resolution of the Government of the Kyrgyz Republic” “On measures to implement the Kyrgyz Republic Pasture Law” dated 19 June 2009 #386 was developed and cleared with relevant ministries, and the draft Program on Development of Pasture Farming and Livestock Breeding for 2019–2023 has been developed and finalized according to the comments and suggestions of the ministries.  Social mobilization: the project supported a total of 17 social activities such as focus groups and other types of meetings for sharing information on issues such as Pasture User Union and Pasture Committee reports and Community Pasture Management and Land Development Plans, with 110,306 participants of which 27,343 (24.8%) were female (no target provided).  Methodological Guidelines on demarcation of the pasture external boundaries between PUUs and establishment of the pasture internal boundaries between farms were developed and approved.  The external pasture boundaries of all Pasture Committees were digitized and installed in Google Earth Pro. Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162)  Information from Hydrometeorological Center on climate change and weather forecast in Chui and Talas was available and sent to pasture users.  The Kyrgyz Livestock and Pasture Research Institute (KLPRI) was supported in developing “Caragana control plan”, a control plan for a persistent weed in Kyrgyz, which included: 1) pest control approaches; 2) use of herbicides and their management; 3) the Bank's policy regarding pesticides, regulatory legal framework and institutional capacity; 4) monitoring and evaluation, cost estimates.  406 Community micro-projects (target: 280) such roads, bridges and water-points were implemented to improve access to pastures. Of these 76 micro-projects were in pasture infrastructure improvement, and 330 in procurement of machinery and equipment (target exceeded).  The project supported an extensive public awareness and communication campaign.  Direct project beneficiaries, including female beneficiaries reached 197,268, of which 49.6% female (target: 190,000 of which 10% female beneficiaries, both targets exceeded (ICR, para 25). Outcome To achieve the project's objective it supported the formulation of Community Pasture Management and Use Plans (CPMUPs). These were comprehensive plans with analysis of pasture conditions, pasture improvement activities including: machinery planning and maintenance, grazing plans and grazing timetable, and budget planning. According to the ICR (para 20) there were no such plans prior to the project intervention. Notable achievements that resulted from the implementation of the CPMUPs included: an increase in usable pasture area by 32 thousand hectares (3.1% of the total pasture area) in two oblasts, reducing the number of PUUs where pastures were overgrazed from 140 (all PUUs in the project area) to 20 with total elimination of cases of extreme overgrazing, improvement in pasture conditions, which was confirmed through beneficiary surveys, project data, and Bank team during supervision missions (ICR, para 21). That said, the ICR (para 21) explained that a longer implementation period was needed to ensure a representative quantitative assessment of changes in pasture conditions, since project activities covered two years only (2017 and 2018). The outcome was assessed through three PDO indicators: first, the number of CPMUPs implemented satisfactorily (scored 75% or higher) based on the PUU performance criteria which came to 113 at project closure, slightly exceeding the final target of 112; (PUU performance reflects an assessment of two dimensions: 1. The technical condition represented by: pasture conditions; grazing pastures and riparian areas; and feed, shelters and watering site management, and 2. The institutional assessment reflected in pasture usage fee collection and pasture load. The technical assessment score is averaged with two institutional scores: pasture usage fee collection and grazing pressure to give an overall mark for PUU performance. Satisfactory level is determined to be 75% or higher, ICR Appendix B). The second PDO indicator measured the percentage increase in utilizable pasture area, which reached 3.1% of the total pasture area or about 39% of the final target set at 8%. The third PDO indicator third, was the beneficiary satisfaction ratio from implementation of CPMUPs which according to a random survey of the population in the project area reached 59% at project closure exceeding the final target set at 50%. The same survey showed that that the number of satisfied beneficiaries was 75% if respondents were direct pasture users. The project exceeded two of its three PDO indicators and fell short on one as noted above. The ICR (para 23) provided a logical explanation to the under achievement of the second outcome indicator (percentage increase in utilizable pasture area) due to "significantly reduced project actual implementation period due to Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) start-up and effectiveness delays." On the positive side, the ICR (para 22) reported a 50% increase in the fee collection for pasture use in 2018 compared to 2015, which indicated that the new pasture governance system in the project areas had the support and approval of the pasture users. It is also plausible to assume that the utilizable pasture area would continue to increase post completion given that 59% of the PUUs had the necessary machinery required for rehabilitation and improving access to pasture areas (ICR, para 23), and hence could be expected to achieve the target for the increase in “utilizable” pasture area. Based on the above-mentioned assessment, the evidence in the ICR points to the success of the project in improving community-based pasture management in the project area, despite falling short on meeting the target on increment in utilizable pasture area before the project completion. Therefore, the efficacy of this outcome for Objective 1 is rated Substantial. Rating Substantial OBJECTIVE 2 Objective (b) to improve livestock management in the project area. Rationale Theory of Change (ToC). In this case change is facilitated by a number of critical elements including: plans for animal health and husbandry which are supported by local governments, sub-grants are of sufficient size to cover investment needs, and better livestock management is expected to lead to increased productivity and higher incomes in accordance with assumptions made for economic analysis. To achieve the stated objective, the project would support the formulation and implementation of Animal Health and Husbandry Plans (AHHPs) in project areas. The project would also provide sub-grants for veterinary services and animal husbandry. In addition, the project would also provide training and equipment to private veterinarians, and support the establishment of rayon (district) associations of private veterinarians. Consultancy support would be provided to the Veterinary Chamber; and each PUU’s Animal Health and Husbandry Group. As a result of the above-mentioned activities in the short-term, improved veterinary services and training are expected to result in improved animal health and husbandry practices. As a long-term outcome, the project is expected to increase productivity in the livestock sector, and improve livelihoods of the rural population. Overall, the ToC included logical assumptions and the stated activities were clearly linked to the PDO. Outputs The information below is based on the ICR (Annex 1) unless referenced otherwise. Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162)  The draft Laws “On Animal Protection”, “On Control of Epizooty” and the proposed amendments to the Law “On Veterinary Medicine” were developed and cleared with relevant ministries;  140 PUUs (target: 112) were supported in preparing, implementing and monitoring plans for animal health and animal husbandry (target exceeded).  Training modules on animal health were developed on the following topics: “Foot and mouth disease”, “Common horse diseases”, “Methods of sampling and delivery of samples of pathological materials to the veterinary laboratory”, “Nodular dermatitis of cattle”  The project supported trainings on 19 different topics related to animal health and feeding. These trainings were attended by a total of 5,609 participants of which 6% were females.  69 community sub-projects were implemented to improve livestock management  743 private veterinarians were trained (target of 420 was exceeded, ICR, para 30).  7,885 client days (target: 14,500) of training were provided, of which 854 days were for women (target: 1,450). Targets were not achieved due to shortened implementation time and the cancellation of forestry activities (ICR, para 30). Outcome To improve livestock management, the project supported the formulation of Animal Health and Husbandry Plans (AHHPs) in project areas. AHHPs featured basic veterinary services including training of private veterinarians and training of PUU members on good animal husbandry practices. According to the ICR (para 27) the project activities improved livestock management through increasing veterinary practices by 100%. These practices provided comprehensive animal health and vaccinations against various diseases including: Foot and Mouth (FMD) and Brucellosis as well as antiparasitic and anthelminthic treatments. Also, the implementation of AHHPs resulted in better coordination of veterinarian activities such as animal identification, vaccination, and animal disease control (ICR, para 27). However, the ICR did not include information on the proportion of animals that were vaccinated in the project area. The project's achievements were assessed through two PDO indicators, first, the number of AHHPs implemented satisfactorily (Satisfactory level is determined to be 75% or higher) based on PUU performance criteria reached 120 at project closure exceeding the final target of 112. PUU performance was assessed based on four criteria : i) livestock living conditions; ii) animal nutrition; iii) feed storage; and iv) animal health. Each criterion was rated on a four-point scale which were averaged to give an overall mark), and second, the proportion of pasture users satisfied with the quality of services provided by private veterinarians reached 96% according to the final evaluation survey compared to a final target of 50%. The ICR (para 28) also highlighted that better livestock management was reflected in the "lack of noticeable cases of animal diseases during project implementation and improved productivity of animals as confirmed by the economic assessment (see section 5 for more details)." Based on the above-mentioned assessment, the ICR has provided evidence that the project activities contributed to improving livestock management in project areas. The project also met its outcome targets. Therefore, efficacy of this outcome for Objective 2 is rated Substantial. Rating Substantial Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale As discussed above, the project activities contributed to the improvement of community-based pasture and livestock management. Despite the shortened implementation period, the project met, or is expected to meet its objectives, and activities were valued by beneficiaries as reflected by the high satisfaction levels reported in the beneficiary survey. Based on the Substantial achievement of both elements of the PDO, the overall efficacy is rated Substantial despite some shortcomings. Overall Efficacy Rating Substantial 5. Efficiency Economic and Financial Efficiency ex ante  The economic and financial analysis (EFA) at appraisal estimated that the project would generate an economic internal rate of return (EIRR) of 52.4% and net incremental benefits with an economic net present value (ENPV) of US$32.13 million. The project was estimated to generate a financial internal rate of return (FIRR) of 49.2% and net incremental benefits with a financial net present value (FNPV) of US$29.22 million.  Expected quantifiable benefits included: (a) Improved incomes for livestock owners as a result of higher livestock productivity due to improvements in animal nutrition and health. (b) Improved animal nutrition from pastures would be generated from better alignment of stocking rates with pasture carry capacity, resting of degraded pastures and improved access to under-utilized pastures. (c) Improved animal health and reduced mortality would result from better animal disease control and improved nutrition.  Assumptions. For component 1, it was assumed that the utilized pasture area in the with-project scenario would increase by 8.0% as a result of investments in pasture infrastructure. Pasture fees are assumed to be 80% higher in the with-project scenario compared to the without-project scenario, to ensure that PUU revenue would be sufficient to cover increased costs of infrastructure maintenance and PUU services. For component 2, it was assumed that as a result of PUU animal health and husbandry planning and improved veterinary services, in the with-project scenario, livestock mortality rates are 2.5% lower, lambing rates are 2.5% higher, and live-weight gain in sheep and cattle as well as milk yields were 2.5% higher, compared to the without-project scenario. Veterinary costs were assumed to double in the with- project scenario compared to the without-project scenario to deliver better services.  Sensitivity analysis. A 10% increase in project costs, a 10% decrease in project benefits, a 10% reduction in the increased area of utilizable pasture resulting from the project or a 10% reduction in the Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) key impact on livestock productivity resulted in EIRRs of between 47% and 49%. Project benefits would have to decline to 24% of their assumed value for the project to generate an EIRR of only 10%. ex post The economic analysis in the ICR of the actual results of this project is based on two partial analyses. The first is an analysis of the incremental benefits and costs for part of the whole project. According to the ICR “incremental” refers to the difference between benefits and costs “with and without” the project. The second analysis is of the incremental benefits and costs observed for project activities in a non-random sample of one “typical” pasture committee (PC). For the project as a whole data were used from the two regions (Chui and Talas) covered by the project. The data from these two regions were used to estimate annual incremental increases in livestock production based on the actual increase in utilized improved pasture area (3% per annum) which (accounting for a lower stocking rate on the improved pasture area) led a 2% per annum increase in livestock production which, in conjunction with unstated livestock sale prices, along with actual incremental project cost, was the basis for the calculation of an economic internal rate of return (EIRR) of 46% over 20 years. A separate analysis was based on the incremental project-assisted investments made by the PC for the Aral PUU in the Talas region which used five micro-projects all aimed at improved pasture management. Various pieces of machinery and equipment, improved infrastructure such cattle dips and bridges together with enhanced veterinary services were used to increase meat and milk production. Based on the incremental increased income (of which about 10% came from leasing the acquired machinery and equipment) and the incremental increased costs, the internal rate of return was also estimated to be 46% over 10 years. However, for partial analysis of both the “project as a whole" and for the “typical PC” there was no information on the sources of data used for the “without project” assessment other than that they were estimates. There was also no sensitivity analysis to gauge the sensitivity of the estimated rates of return to changes in pasture productivity or meat and milk prices. The ex post EFA showed that the project achieved lower economic and financial rates of return compared to the appraisal figures. While the estimated ex-post rates of return were significantly higher than the discount rate of 15% assumed in the ICR for this project, as noted above there was no information in the ICR on the sources of data or the basis for estimates of the data used for the economic and financial analysis resulting in quite high rates of return. In particular the basis for the project benefits and costs without the project (and hence the basis for incremental income and costs) is not provided in the ICR. It was also not clear whether the ex-post EFA factored-in that at completion the utilizable pasture area increased by a total of only 3.1% rather than the 8% expected at appraisal. Administrative and Institutional Efficiency While the project experienced a relatively short extension of three months beyond the original closing date, there were significant delays at the effectiveness and start-up stages. Effectiveness was delayed one year after approval due to the long parliamentary ratification and associated clearance procedures in the government (ICR, para 49). Start-up delays resulted from spending the first year of implementation in on preparatory activities like development of the PUU assessment tool and CPMUPs. The preparation of CPMUPs was also negatively impacted by consultancy delays. Consequently, this resulted in delaying community micro-projects, because Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) disbursement was conditional on the Bank's approval of the CPMUPs. The conduct of an internal inventory of pastures was delayed and eventually cancelled due to the absence of bidders except for the State Institute on Land Resources. However, awarding the contract to the State Institute on Land Resources was contrary to the Bank's guidelines because it was a state-owned entity (ICR, par 51). Project implementation was disrupted by the sudden resignation of the director of the implementing agency as well as the project's M&E specialist. Procurement also suffered delays in awarding contracts which, according to the ICR (para 62), were beyond the control of the project. Despite the high estimated rates of return for this project its efficiency had a number of serious shortcomings (a) absence of any substantive information on how incremental benefits and costs were derived for the analysis of the internal rates of return; (b) the under achievement of the core PDO target, namely, the percentage increase in improved utilizable pastures of 3.1% compared with the target of 8%; and (c) weaknesses at the administrative and institutional levels in the project causing significant implementation delays. This review has therefore rated the efficiency of this project as Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  52.00  Not Applicable 100.00 ICR Estimate  46.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of Objectives was rated Substantial. The evidence provided in the ICR point to the expected success in improving community-based pasture and livestock management in the project area through implementing improved pasture and livestock management plans. Overall Efficacy was rated Substantial, despite some shortcomings in the efficiency with which these objectives were estimated to have been achieved. Efficiency was weak and hence rated Modest due to considerable under achievement of the target on utilizable pastures (3.1% at project completion compared to a target of 8%), and weaknesses at the administrative and institutional levels causing significant implementation delays. Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) Based on a Substantial rating for Relevance and Efficacy, and a Modest rating for Efficiency, the overall Outcome of the project is rated Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The ICR (para 69) discussed several issues that could impact the risk to development outcome:  Continued implementation of pasture management plans. The results showed that pasture users are satisfied with the positive effects of pasture management plans, and there is strong intention to continue implementation of such plans after project completion. Also, implementation of pasture plans would benefit from local government support and financial stability of Pasture User Unions (PUUs).  Financial stability of PUUs. The PUUs were expected to generate additional cash from the machinery procured during the implementation of community micro-projects, which was expected to strengthen their financial status.  Negative effects of climate change. This is a significant risk in the long-term because unfavorable climate conditions could result in extended drought periods that would negatively impact pastures and result in loss of productivity. The project activities would contribute partially to mitigating this risk through monitoring of pasture conditions and provision of alerts on weather changes to farmers. However, more studies on the climate change impact and investments in climate resilience activities are needed to help pasture communities mitigate climate change effects. Overall, the risk to the development outcome was rated Modest by the ICR. 8. Assessment of Bank Performance a. Quality-at-Entry This project was a follow-on operation to consolidate the achievements of the Bank-funded Agricultural Investments and Services Project (AISP, 2008-2013). AISP supported the design and initial implementation of pasture reforms, including social mobilization as a basis for the establishment of PUUs, and establishing governance arrangements and demarcating legal boundaries for pastures. The project aimed to contribute to reducing poverty and improving shared prosperity. The project was prepared in close collaboration with several Government agencies and local communities which helped to identify needs in pasture management and infrastructure. Design was informed by lessons from the experience of AISP. Notable lessons reflected in the design include: expanding investment in sub-grants including pasture infrastructure based on the strong demand under AISP, building the capacity of a team Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) of pasture consultants to provide technical advice to PUUs and encourage PUUs to contract their services, and intensifying public awareness efforts and strong logistical coordination at community level to ensure successful animal disease control. Design featured activities that were expected to increase access to pastures and animal health services with attention to the poorest livestock holders (ICR, para 5). Design also emphasized gender inclusion in governance activities through the PUUs as well as in training activities. Design featured well-structured components that contributed to achieving the PDO. Design could have benefited from consultancy support to cover the need for technical assistance and avoid lengthy delays. Six risks were identified at appraisal, with an overall moderate rating. The PAD included relevant risk management actions. However, the risk of delays of important preparatory activities was overlooked- specifically, the PUU assessment tool and pasture planning methodology which both required more effort than was originally thought (ICR, para 65). This situation combined with funding constraints contributed to significant start-up delays. M&E suffered from design weaknesses where the Results Framework lacked relevant indicators to capture certain project outputs/intermediate outcomes (see section 9 for more details). Based on the above-mentioned assessment, Quality at Entry is rated Moderately Satisfactory due to moderate shortcomings that stemmed mainly from overlooking risks as well as M&E design weaknesses. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The project experienced effectiveness delays followed by start-up delays. This created a challenging implementation environment at the earlier stages of the project. Supervision missions were conducted on a biannual schedule. The Bank team and the Country office provided technical and fiduciary support to the project. Also, social and environmental safeguards were regularly supervised. Implementation benefited from a comprehensive MTR that mapped out key measures to recover some of the lost time at the beginning of the project (ICR, para 66). However, the Bank should have worked earlier with the implementing agency to identify and rectify issues that caused start-up delays. Also, the project implementation could have benefited from dedicating more staff time and resources to support implementation (ICR, para 67). The coordination between the project and the IFAD-financed Livestock and Market Development project could have benefited from more attention to maximize the development effect. According to the ICR (para 46) the IFAD-funded project experienced implementation delays and "didn’t produce some instruments that the Bank team counted on." Finally, the project implementation would have benefited from a longer extension of the closing date to compensate for the start-up and initial implementation delays. Overall, the Quality of Supervision is rated Satisfactory despite the shortcomings that this review considered minor. Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) Overall Bank Performance is rated Moderately Satisfactory. This rating factors in the moderate shortcomings under Quality at Entry and minor shortcomings under Quality of Supervision. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The PAD did not include an explicit Theory of Change (ToC). Nonetheless, the ICR (para 7 & Figure 1) included a theory of change that laid out the relation between the project inputs, outputs, outcomes and longer-term outcomes. The overall responsibility of M&E activities would be under the Agricultural Projects the Implementation Unit (APIU). Design featured participatory monitoring and evaluation (M&E) system that would be with assisted by Rayon (district) Associations of PUUs to establish systems to monitor the performance of their member PUUs. The design called for conducting three surveys, a baseline survey followed by a survey at MTR and a final survey at the completion. The PDO was to be assessed through five PDO level indicators: the improvement of the community-based pasture management was expected to be seen in terms of: 1. number of Community Pasture Management Plans implemented satisfactorily based on the PUU performance criteria, 2. percentage increase in the area of utilizable pastures, and 3. percentage of pasture users satisfied with the management of pastures by PUUs. While the Improvement of the livestock management was expected to be seen in terms of: 1. number of animal health and husbandry plans implemented satisfactorily based on PUU performance criteria, and 2. percentage of pasture users satisfied with the quality of services provided by Private Veterinarians. The stated indicators were linked to the respective parts of the PDO and were measurable. However, two indicators were tied to the PUU assessment tool which according to the ICR (para 56) "additional coordination needed between APIU and PUUs and possible increase in maintenance cost." None of the indicators had a baseline. The Results Framework also included nine intermediate outcome indicators, two of which were capturing the gender dimension. These indicators were relevant and measurable. However, not all the project supported activities were covered under the RF. For example, the were no indicators to assess the impacts of the animal health and livestock improvements on mortality rates, lambing rates, and live-weight gain in sheep and cattle as well as milk yields. Also, the RF lacked intermediate indicators to assess impacts of the project-financed micro-projects including infrastructure. Finally, two indicators were redundant with the outcome level indicators (number of Community Pasture Management Plans and number of animal health and husbandry plans). Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) While M&E design was straightforward, assessing the actual improvement and testing the links in the results chain proved challenging (ICR, para 57). Also, the RF lacked intermediate indicators to cover the impacts of the animal health and livestock improvements supported by the project. b. M&E Implementation M&E implementation was challenging and suffered from start-up and implementation delays. The Agricultural Projects Implementation Unit (APIU) could not collect data after start-up because the PUU assessment tool was not developed at the preparation stage. Also, the methodology on the data to be collected and analyzed was not clear (ICR, para 57). Data on intermediate outcome indicators was collected to the extent possible, but tracking progress towards achieving the PDO was difficult to monitor (ICR, para 57). M&E challenges were exacerbated by the delay in conducting a baseline survey and the PUU assessment survey. While M&E functioning improved after the MTR, staff turnover was disruptive as new staff needed time to familiarize with the work. Despite these challenges, the project managed to track pasture unions in a comprehensive manner. This allowed reporting of various aspects of pasture management into a single evaluation system. M&E capacity at the PUU level benefited from training and consultancies provided under the project. Overall, however, M&E implementation was weak and tracking the achievement of outcomes had considerable shortcomings. c. M&E Utilization According to the ICR (para 58) "M&E findings were regularly communicated to the various stakeholders, which raised trust in the system." The project collected data on dry mass yields and overall pasture condition, which was used in the PUU assessment tool. While livestock and pasture management data was successfully collected by the project, "more time is needed to see the evidence that the system is fully used for decision making (ICR, para 59). "Furthermore, the ICR (para 59) stated that additional evidence was needed beyond the project's own M&E data to be able to reach "convincing conclusions." Overall, M&E Quality is rated modest. This rating reflects moderate design shortcomings, significant implementation challenges and limited utilization. M&E Quality Rating Modest 10. Other Issues Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) a. Safeguards The project was rated Environmental Category “B”, because of the potential limited environmental issues associated with the small-scale investments in pasture infrastructure. The project triggered three safeguard policies: Environmental Assessment OP/BP 4.01, Natural Habitats OP/BP 4.04, and Pest Management OP 4.09. The environmental impact of the project would be largely positive, but some adverse impacts might be generated. Positive environmental impacts of the project mainly would result from sustainable management of pastures by communities. Negative environmental impacts were expected to result from: increased pollution due to construction waste; increased pollution of ground and surface waters due to soil erosion and use of fertilizers and herbicides; potential increased use of pesticides for pasture management and fodder; and increased siltation of water bodies due to soil erosion. Mitigation measures were identified in the environmental assessment (EA) and the environmental and social management plan (ESMP) prepared for the project. The ICR (para 61) reported that environmental management benefited from a comprehensive environmental safeguards review as part of the Mid-Term Review. This improved environmental reporting and addressed issues related to the environmental sections in pasture management plans. According to the ICR (para 61) "the project was implemented in compliance with the Bank and national environmental assessment rules and procedures as well as existing environmental requirements." b. Fiduciary Compliance Financial Management. According to the ICR (para 63) financial management capacity "at both implementing agencies the Agricultural Projects Implementation Unit (APIU) and the Community Development and Investment Agency (ARIS), including organization and staffing was satisfactory." Both agencies had adequate internal control systems and provided timely reporting. The ICR did not report on the status of audit reports. Procurement. Procurement activities benefited from the presence of procurement specialists with international experience at the implementing agencies (the Agricultural Projects Implementation Unit and the Community Development and Investment Agency). Procurement ex post reviews were conducted regularly. While there were delays in awarding contracts, the ICR (para 62) reported that these delays "were due to factors beyond the control of the implementing agencies." There were minor procurement issues related to filing of signatures and storage of document copies, which according to the ICR (para 62) were resolved. c. Unintended impacts (Positive or Negative) --- d. Other Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) "The project raised awareness of the livestock sector stakeholders to climate risks and change, which is the first step in building resilience. In order to timely alert about inclement weather conditions, the project arranged for information exchanges from the Hydrometeorological Center to pasture users via cellular communication. This includes weather forecasts for the next 10-15 days and storm warnings transmitted 5 days in advance, allowing pasture users to be prepared for sudden changes in weather conditions. This was complemented by training sessions on climate change preparedness and information sharing on adaptation technologies. (ICR, para 44)." 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR included five lessons. Four of these lessons were project-specific reflecting on the implementation experience of this project. One lesson was of broader applicability to similar projects and is emphasized with some adaptation of language:  To ensure the sustainability of the development outcome, climate change risk needs to be addressed at the design stage. Initial pasture monitoring results in this project showed that climate change is becoming increasingly important in natural resources management and may pose a significant risk to development outcomes. Incorporating research into the effects of climate change and education of communities on climate change in project design will help communities to prepare better for environmental changes due to climate change. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain This project was a follow-on project to consolidate the achievements of the Bank's Agricultural Investments and Services Project (AISP, 2008-2013). The project experienced start-up/implementation delays and impacts needed more time to materialize, and more data are needed to assess efficacy and efficiency of this project Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) accurately. Also, an assessment would be a good opportunity to assess the impact of both projects on livestock and pasture management and the sustainability of outcomes, and generate useful lessons that would inform future operations. 14. Comments on Quality of ICR Quality of Evidence. The ICR acknowledged that M&E design had some shortcomings and implementation was weak. The ICR alternatively relied on the final evaluation survey and the supervision reports combined with information gathered at the ICR mission. Quality of Analysis. The ICR provided clear linking between evidence and findings. However, the impact of project activities on improving pasture management did not fully materialize due to start-up and implementation delays, which was beyond the control of the ICR. Also, efficiency analysis could have benefited from more details on the basis for the project's estimates of incremental benefits and costs. Extent to which lessons are based on evidence and analysis. Lessons were based on the project experience although they could have benefited from a better formulation of lessons that were broadly applicable to similar projects. Results Orientation. The ICR included a good discussion on the efficacy of outcomes despite concerns on M&E implementation and the assessment of overall outcome. The discussion of the first part of the PDO was thorough and went beyond reporting the achievement of outcome indicators. Discussion of the second part of the PDO was constrained by the limitation of the Results Framework design. Internal Consistency. Various parts of the ICR were logically linked and integrated. Consistency with OPCS Guidelines. The ICR rated Relevance of Objectives, overall Efficacy and Efficiency as Substantial. According to the Guidelines the overall outcome of this project would be rated Satisfactory. Nevertheless the ICR, using the flexibility in the Guidelines for downgrading overall outcome if there are “moderate shortcomings in the achievement of one of more of the objectives/outcomes used in the assessment of the overall efficacy”, rated overall outcome Moderately Satisfactory. While in principle this is consistent with the Guidelines, the ICR lacked explicit evidence on the moderate shortcomings of overall efficacy to justify the downgrading of overall outcome rating. In this case, IEG has assumed that the government and World Bank teams were aware of shortcomings in the project's efficacy that were not discussed in the ICR and has therefore accepted the downgrading of the overall outcome in the ICR. Conciseness. The ICR provided thorough coverage of the implementation experience and candidly reported on shortcomings. There was enough clarity in the report’s messaging. The performance story included a good amount of details despite M&E weaknesses. The ICR could, however, have provided more details on the risks identified at appraisal, whether any materialized and the efficacy of any risk mitigation measures. Finally, the Theory of Change could have benefited from a brief explanatory discussion rather than only presenting it as a chart. Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Pasture Management Improvement Project (P145162) Overall, this review concluded that the Quality of the ICR was marginally Substantial. a. Quality of ICR Rating Substantial Page 20 of 20