Document of The World Bank Report No: ICR4040 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-50030) ON A CREDIT IN THE AMOUNT OF SDR 24.4 MILLION (US$ 39 MILLION EQUIVALENT) TO THE PLURINATIONAL STATE OF BOLIVIA FOR THE AGRICULTURAL INNOVATION AND SERVICES PROJECT August 8, 2017 Agriculture Global Practice Latin American and Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective February 15, 2017 Currency Unit = Bolivianos Bs. 6.93 = US$1.00 US$1.00 = SDR 1.3500 FISCAL YEAR January 1 -- December 31 ABBREVIATIONS AND ACRONYMS Ag PER Agricultural Public Expenditure Review BCR Borrower Completion Report CIAT Centro Internacional de Agricultura Tropical (International Center for Tropical Agriculture) CIMMYT Centro Internacional de Mejoramiento de Maiz y Trigo (International Maize and Wheat Improvement Center) CIP Centro Internacional de la Papa (International Potato Center) CDI Departmental Innovation Council (Consejo Departamental de Innovación) CPI Plurinational Innovation Council (Consejo Plurinacional de Innovación) CRI Regional Innovation Council (Consejo Regional de Innovación) CPF Country Partnership Framework COSUDE Confederation Suisse de Developpement et Cooperation (Swiss Cooperation and Development Agency) Danida Danish Development Agency DAF INIAF’s Financial Administrative Directorate (Dirección Administrativa y Financiera) DIC Departmental Innovation Council DNAT INIAF’s National Technical Assistance Directorate (Dirección de Asistencia Técnica) DNI INIAF’s National Research Directorate (Dirección de Investigación) DNS INIAF’s National Seeds Directorate (Dirección de Semillas) DO Development Objective DPF Development Policy Financing EA Environmental Assessment ELC Local Coordination Spaces (Espacios Locales de Concertación) EMP Environmental Management Plan FAO United Nations Food and Agriculture Organization EIRR Economic Internal Rate of Return FM Financial Management FMISS Financial Management Implementation Support and Supervision Report FMR Financial Monitoring Report ii FRR Financial Rate of Return GDP Gross Domestic Product GoB Government of Bolivia GIZ German Cooperation Agency HR Human Resources IBRD International Bank for Reconstruction and Development IBTA Bolivian Institute of Agricultural Technology (Instituto Boliviano de Tecnología Agropecuaria) ICR Implementation Completion Report IDA International Development Association IFR Interim Financial Reports INIAF National Institute for Agricultural and Forestry Research Innovation (Instituto Nacional de Innovación Agrícola y Forestal) INSA Institute of Agrarian Insurance (Instituto del Seguro Agrario) IPP Indigenous Peoples Plan IRRI International Rice Research Institute LAC Latin America and the Caribbean MDRyT Ministry of Rural Development and Lands (Ministerio de Desarrollo Rural y Tierras) M&E Monitoring and Evaluation NCB National Competitive Bidding NGO Non-Governmental Organization NPV Net Present Value ORAF Operational Risk Assessment Framework PAD Project Appraisal Document PDO Project Development Objective POA Annual Operating Plan (Plan Operativo Annual) PISA Agricultural Innovation and Services Project (Proyecto de Innovación y Servicios Agropecuarios) PND National Development Plan (Plan Nacional de Desarrollo) PROINPA Fundación Promoción e Investigación de Productos Andinos PforR Program for Results R&D Research and Development R&E Research and Extension RIC Regional Innovation Council RF Results Framework SA Social Assessment SAFCO Government Administration and Control Law (Ley de Administración y Control Gubernamentales) SBI Bolivian Innovation System (Sistema Boliviano de Innovación) SENASAG Servicio Nacional de Sanidad Agropecuaria e Inocuidad Alimentaria (National Service for Agro-livestock Health and Food Safety) SIBTA Bolivian System for Agricultural Technology (Sistema Boliviano de Tecnología Agropecuaria SDC Swiss Development Corporation SISPOA Sistema para Plan Operativo Anual SISPOA-AE Sistema para Plan Operativo Anual – Alianzas Estratégicas iii SNIAF Agriculture and Forestry Innovation System (Sistema Nacional de Innovación Agropecuaria y Forestal) TA Technical Assistance WDI World Development Indicators Vice President: Jorge Familiar Country Director: Alberto Rodriguez Global Practice Senior Director Juergen Voegele Practice Manager: Preeti Ahuja Project Team Leaders: Francisco Obreque and Michael Morris ICR Team Leader: Griselle Vega iv PLURINATIONAL STATE OF BOLIVIA Agricultural Innovation and Services Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives, and Design .......................................................... 1 2. Key Factors Affecting Implementation and Outcomes .......................................................... 6 3. Assessment of Outcomes ...................................................................................................... 14 4. Assessment of Risk to Development Outcome..................................................................... 26 5. Assessment of Bank and Borrower Performance ................................................................. 26 6. Lessons Learned ................................................................................................................... 29 7. Comments on Issues Raised by Borrower/Implementing Agencies and Partners ................ 30 Annex 1. Project Costs and Financing .......................................................................................... 32 Annex 2. Outputs by Component .................................................................................................. 34 Annex 3. Economic and Financial Analysis.................................................................................. 40 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............................. 55 Annex 5. Beneficiary Survey Results ............................................................................................ 57 Annex 6. Stakeholder Workshop Report and Results ................................................................... 58 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ...................................... 61 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ........................................ 64 Annex 9. List of Supporting Documents ....................................................................................... 65 Map................................................................................................................................................ 67 v A. BASIC INFORMATION Agricultural Innovation Country: Bolivia Project Name: and Services Project ID: P106700 L/C/TF Number(s): IDA-50030 ICR Date: 07/27/2017 ICR Type: Core ICR GOVERNMENT OF Financing Instrument: IPF Borrower: BOLIVIA Original Total USD 39.00M Disbursed Amount: USD 34.87M Commitment: Revised Amount: USD 39.00M Environmental Category: B Implementing Agencies: INIAF Cofinanciers and Other External Partners: DANIDA, COSUDE B. KEY DATES Process Date Process Original Date Revised / Actual Date(s) Concept Review: 01/12/2009 Effectiveness: 01/30/2012 01/30/2012 05/22/2012 Appraisal: 01/14/2011 Restructurings: 06/10/2015 02/17/2016 Approval: 07/21/2011 Mid-term Review: 07/14/2014 07/15/2014 Closing: 02/15/2017 02/15/2017 C. RATINGS SUMMARY C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Unsatisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: vi C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments (if Indicators Rating Performance any) Potential Problem Project at Yes Quality at Entry (QEA): None any time (Yes/No): Problem Project at any time Quality of Supervision Yes None (Yes/No): (QSA): DO rating before Closing: Moderately Unsatisfactory D. SECTOR AND THEME CODES Original Actual Major Sector/Sector Agriculture, Fishing and Forestry Public Administration - Agriculture, Fishing & 32 32 Forestry Crops 11 11 Agricultural Extension, Research, Support Activities 57 57 Major Theme/Theme/Sub Theme Environment and Natural Resource Management Climate change 10 10 Adaptation 5 5 Mitigation 5 5 Renewable Natural Resources Asset Management 8 8 Biodiversity 4 4 Landscape Management 4 4 Finance Finance for Development 16 16 Agriculture Finance 16 16 Private Sector Development Jobs 100 100 Urban and Rural Development Rural Development 76 76 Land Administration and Management 4 4 Rural Infrastructure and service delivery 56 56 Rural Markets 16 16 vii E. BANK STAFF Positions At ICR At Approval Regional Vice President: Jorge Familiar Calderon Pamela Cox Country Director: Alberto Rodriguez Laura Frigenti Practice Manager: Preeti S. Ahuja Ethel Sennhauser Task Team Leader(s): Francisco Javier Obreque Arqueros Wilhelmus Gerardus Janssen ICR Team Leader: Griselle Felicita Vega Senior Global Practice Director: Juergen Voegele Co Task Team Leader1: Michael Morris ICR Primary Author: Griselle Felicita Vega F. RESULTS FRAMEWORK ANALYSIS Project Development Objectives (from Project Appraisal Document) To strengthen INIAF and the Recipient's National Agricultural and Forestry Innovation System (SNIAF) in order to contribute to productivity growth, food security, sustainable rural development and the income- earning potential of Recipient families dependent on agriculture and forestry. Revised Project Development Objectives (as approved by original approving authority) N/A (a) PDO Indicators Original Target Values Actual Value Achieved Formally Revised Indicator Baseline Value (from approval at Completion or Target Target Values documents) Years Indicator 1 : Participation of R&E organizations in annual SNIAF meeting (%) Value (quantitative or 0 40 70 71.32 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 02/15/2017 Comments Exceeded: 102%. At the time of the 2015 Restructuring, the population of R&E organizations (incl. % was defined precisely to allow measurement, and the end-of-project target was revised upward. achievement) Revised indicator: Participation of R&E organizations in SNIAF meetings at national and subnational level. This result is explained by INIAF’s successful promotion of strategic planning, collaboration and knowledge exchange among the R&E innovation agents, especially through the decentralized fora (DIC and RIC), supported at national level through INIAF’s annual INNOVARE events, which attracted even broader participation and support. Indicator 2 : INIAF-led agricultural innovations adopted by producers (#) Value (quantitative or 0 20 20 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 Comments Achieved: 100%. At the time of the 2015 Restructuring, two terms were precisely defined to (incl. % facilitate measurement: (i) “agricultural innovations” was defined to include: new plant achievement) varieties, new agricultural and forestry practices; and, other innovations (e.g., new equipment); viii and (ii) “adoption” was defined to mean adoption in process, to be measured annually through field verification by the M&E Unit with support from independent entities. The end-of-project target did not change. At completion, 20 innovations had been identified that met the criteria for adoption. (See discussion in Section 3.2 and Annex 2, Table A2.2) INIAF’s success in reaching this target is strong evidence of strengthened capacity. INIAF: (i) correctly diagnosed producers’ needs; (ii) generated or accessed innovations capable of addressing them; (iii) validated the effectiveness of the innovations in the field and determined their economic viability; and, (iv) devised an effective strategy for delivering the innovations to producers. Indicator 3 : Agricultural organizations which see INIAF as an effective leader of the agricultural innovation system Value (quantitative or 0 40 73.82 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 Comments Exceeded: 182%. At the time of the 2015 Restructuring, the population of agricultural (incl. % organizations was defined to include the R&E organizations used to measure PDO Indicator 1 achievement) plus a sample of farmer organizations and seed producers. Perceptions of INIAF leadership were based on INIAF’s performance with respect to three criteria: (i) coordination of stakeholders; (ii) generation of ideas and policy proposals; and (iii) mobilization of financial resources. The end-of-project target did not change. Agricultural organizations perceived that the SNIAF was a functioning innovation network providing open forums (DIC, RIC, workshops) for the expressions of needs, backed by financing instruments and mechanisms for delivering benefits to the field level. Participants in stakeholder workshops praised INIAF’s capacity building activities, emphasis on markets, and the value of INIAF’s seed certification activities to seed producers and users (b) Intermediate Outcome Indicators Original Target Values (from Formally Revised Actual Value Achieved at Indicator Baseline Value approval Target Values Completion or Target Years documents) Indicator 1 : Number of projects funded by the research fund (#) Value (quantitative or 0 30 24 23 qualitative) Date achieved 06/13/2011 02/15/2015 02/15/2017 02/15/2017 Comments Substantially achieved: 96%. At the time of the 2015 Restructuring, this indicator was (incl. % improved to count subprojects as having been funded by INIAF only when they had achievement) received the final payment in a planned series of payments. Revised indicator: Subprojects supported by the research fund that have received final payment (#). The end-of-project target was reduced. By closing, final payments had been made to all but one subproject. The end-of-project target was not quite met because one subproject had to be terminated early when the project closed, so the final payment for that subproject was not made. Indicator 2 : Share of priority themes covered by the funded projects Value (quantitative or 0 100 61.11 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 ix Comments Partially achieved: 61%. At the time of the 2015 Restructuring, this indicator was revised (incl. % to facilitate measurement. Revised indicator: Themes prioritized by the Departmental achievement) Innovation Councils that are covered by subprojects supported by the Research Fund (%). The end-of-project target was not changed. The target was only partially achieved because not all of the demand-driven sub-project proposals that were received were aligned with the priority themes identified by the DICs. Indicator 3 : Number of innovation policies developed by INIAF and approved by MDRyT (Ministry of Rural Development and Lands) Value (quantitative or 0 4 7 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 Comments Exceeded: 175% At the time of the 2015 Restructuring, the term “innovation policies” (incl. % was defined to mean policies, strategies and regulatory documents spelling out norms and achievement) standards as prepared by INIAF, DICs, and RICs at the national, regional and local levels. Revised indicator: Innovation policies (including those affecting official norms and standards) developed through the SNIAF and approved by the competent authority . The end-of-project target was increased. This result is explained by INIAF’s burgeoning capacity to support and encourage sub-national entities as forums for generating, demanding and advocating for policy and regulatory reforms. Indicator 4 : Number of national agricultural innovation fora Value (quantitative or 0 4 Indicator deleted qualitative) Date achieved 06/13/2011 02/15/2017 Comments At the time of the 2015 Restructuring, this indicator was deleted because it was considered (incl. % a poor indicator of the vitality of the SNIAF. achievement) Indicator 5 : Number of departmental technical assistance strategies defined with support of INIAF Value (quantitative or 0 12 10 10 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 02/15/2017 Comments Achieved: 100%. At the time of the 2015 Restructuring, the target was adjusted because (incl. % there were only nine Departmental Innovation Councils (DICs) and only one Regional achievement) Innovation Council (RIC) in the entire country. INIAF proactively supported these Councils in developing technical assistance strategies. Indicator 6 : Consolidated Departmental and Regional Innovation Councils contributing to the accomplishment of SNIAF objectives (#) Value (quantitative or 0 6 6 qualitative) Date achieved 06/10/2015 02/15/2017 02/15/2017 Comments Achieved: 100%. At the time of the 2015 Restructuring, this indicator was added because (incl. % the presence of functioning DICs and the RIC showed the impact the project was having in achievement) building the SNIAF. The sub-national entities were an important project achievement, as they played a decisive role in strengthening the SNIAF. x Indicator 7 : Percentage of targeted farmers satisfied with INIAF-supported technical assistance. Value (quantitative or 0 65 70 98 qualitative) Date achieved 06/13/2011 02/15/2011 02/15/2017 02/15/2017 Comments Exceeded: 140%. At the time of the 2015 Restructuring, the term “targeted farmers” was (incl. % defined to mean all farmers (male and female) participating in technical assistance achievement) subprojects. Revised indicator: Targeted farmers satisfied with INIAF-supported technical assistance. The end-of-project target was adjusted upward to harmonize (make identical) what previously had been different targets for male and female farmers. INIAF assumed a strong lead role in TA and assigned a high priority to direct services, delivering TA in 193 municipalities. INIAF’s program at closing had reached over 21,200 beneficiaries through a consolidated network of technical advisors and systematized innovation. TA increased farmers’ productivity and incomes in priority sectors, strengthened farmers’ management skills and capacity for collective marketing, and promoted alliances with other actors. Indicator 8 : Percentage of women farmers satisfied with INIAF-supported technical assistance in terms of their TA needs Value (quantitative or 0 70 70 98.5 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 02/15/2017 Comments Exceeded: 141% (Breakdown of Intermediate Indicator 7) At the time of the 2015 (incl. % Restructuring, this indicator was revised to more closely align it with Intermediate achievement) Indicator 7. Revised indicator: Targeted farmers satisfied with INIAF-supported technical assistance – female (%). The end-of-project target did not change. Women’s participation was disappointing in some areas (notably in managerial and technical roles within INIAF), but their participation in innovation subprojects was better. More than one-third of innovation subproject beneficiaries (36%, or 8,276 individuals) were women, and nearly 99 percent of those surveyed expressed satisfaction with the technical assistance they had received. Indicator 9 : Targeted farmers satisfied with INIAF-supported technical assistance – male (%) Value (quantitative or N/A 70 98.1 qualitative) Date achieved 06/10/2015 02/15/2017 02/15/2017 Comments Exceeded: 140% (Breakdown of Intermediate Indicator 7) At the time of the 2015 (incl. % Restructuring, this indicator was added to complement Intermediate Indicator 8. Similar to achievement) the case of women farmers, an extremely high proportion of men farmers (98%) expressed satisfaction with the TA they had received from INIAF. Indicator 10 : Volume of seed certified by INIAF (tons) Value (quantitative or 0 106,000 119,124 qualitative) (see comment) Date achieved 06/10/2015 02/15/2017 02/15/2017 Comments Exceeded: 112.4%. At the time of the 2015 Restructuring, this indicator was added to (incl. % provide a way of measuring the contribution made by INIAF in facilitating the diffusion of achievement) improved varieties and high-quality seed. The recorded baseline of zero (0) did not take into account the fact that INIAF was already certifying seed (78,700 tons in the first year of xi the project). This result shows the project’s success in strengthening INIAF seed laboratories and seed processing facilities, introducing new seed quality standards, strengthening the regulatory framework for seed marketing, and publicizing the benefits of certified seed covering an expanded range of crops. Indicator 11 : Percentage of seed producers satisfied with INIAF certification procedures Value (quantitative or 0 80 90.6 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 Comments Exceeded: 113.2%. At the time of the 2015 Restructuring, the term “seed producers” was (incl. % defined to include only individuals (not organizations), so the indicator could be achievement) disaggregated by gender. Seed producer organizations continued to be tracked by the project M&E system, however. By closing, about 4,500 seed producers and 300 producer organizations (representing about 14,000 families) had benefited from INIAF’s seed certification services. Seed producers showed high levels of satisfaction with INIAF’s seed certification services, which improved the quality of seed they sold, increasing their net incomes and their demand for seed certification services. The increased availability in the market of certified seed in turn increased yields and net incomes for final users of seed. Indicator 12 : Seed producers satisfied with INIAF certification procedures - female (%) Value (quantitative or 0 80 87.5 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 Comments Exceeded: 109.4% (Breakdown of Intermediate Indicator 11) Female seed producers who (incl. % participated in the beneficiary survey reported high levels of satisfaction with INIAF seed achievement) certification services, consistent with the overall population of seed producers. The slightly lower level of satisfaction reported by female compared to male seed producers is within the margin of error of the beneficiary survey. Indicator 13 : Seed producers satisfied with INIAF certification procedures - male (%) Value (quantitative or 0 80 91.1 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 Comments Exceeded: 113.9% (Breakdown of Intermediate Indicator 11) Male seed producers who (incl. % participated in the beneficiary survey reported high levels of satisfaction with INIAF seed achievement) certification services, consistent with the overall population of seed producers. Indicator 14 : Number of products for which Bolivia has established a genetic resources database Value (quantitative or 0 12 Indicator deleted qualitative) Date achieved 06/13/2011 02/15/2017 Comments At the time of the 2015 Restructuring, this indicator was deleted. By then, genetic (incl. % resources databases had been established for eight (8) products, but these databases were achievement) not being used because much better databases had become available on-line. Also, the indicator was believed to have little direct relevance to project objectives. xii Indicator 15 : Number of accessions in INIAF's germplasm banks Value (quantitative or 15,500 17,490 Indicator deleted qualitative) Date achieved 06/13/2011 02/15/2017 Comments At the time of the 2015 Restructuring, this indicator was deleted. At that time, INIAF’s (incl. % germplasm banks contained 16,436 accessions. The indicator was replaced by a new, achievement) similar indicator that better reflected project objectives: Progress achieved in characterizing accessions in INIAF’s germplasm banks. Indicator 16 : Accessions characterized in INIAF's germplasm banks Value (quantitative or 0 20 66.25 qualitative) Date achieved 06/10/2015 02/15/2017 02/15/2017 Comments Exceeded: 331.2%. At the time of the 2015 Restructuring, this indicator was added as a (incl. % replacement to Intermediate Indicator 15. The term “accessions characterized” was defined achievement) to mean accessions for which INIAF has created and recorded morphological descriptions. This indicator better measures the extent to which INIAF promotes innovation by increasing the usefulness of the accessions in its germplasm banks. No baseline was provided. This strong result is explained by INIAF’s creation of a National Germplasm Bank Network of active and working banks, with participation of 19 institutions. Government also signed two international treaties on genetic resources (Nagoya Protocol). These steps galvanized the achievement of technical objectives related to genetic resources including data conservation and management, germplasm characterization, and genetic improvement. Indicator 17 : Number of Genetic Resource Treaties subscribed by Bolivia Value (quantitative or 0 3 Indicator deleted qualitative) Date achieved 06/13/2011 02/15/2017 Comments At the time of the 2015 Restructuring, this indicator was deleted because INIAF has no (incl. % control over the number of genetic resource treaties subscribed to by the Government of achievement) Bolivia. Indicator 18 : Targeted clients satisfied with agricultural services (Core) Value (quantitative or 0 75 95.02 qualitative) when measured in Jan 2017 Date achieved 06/10/2015 02/15/2017 02/15/2017 Comments Exceeded: 126.7%. At the time of the 2015 Restructuring, this indicator was added (incl. % because it was a Core Indicator required in all World Bank projects. The method for achievement) calculating the indicator takes into account the level of satisfaction with two types of agricultural services provided under the Project: (i) technical assistance, and (ii) seed certification. xiii Indicator 19 : Targeted clients satisfied with agricultural services - male (Core supplement) Value (quantitative or 0 15,000 15,171 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 Comments Exceeded: 101% (Breakdown of Intermediate Indicator 18). (incl. % achievement) Indicator 20 : Targeted clients satisfied with agricultural services - female (Core supplement) Value (quantitative or 0 7,500 7,500 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 Comments Achieved: 100% (Breakdown of Intermediate Indicator 18). (incl. % achievement) Indicator 21 : Targeted clients - male (Core supplement) Value (quantitative or 0 20,000 19,442 qualitative) Date achieved 06/10/2015 02/15/2017 02/15/2017 Comments Substantially achieved: 97.2%. At the time of the 2015 Restructuring, this indicator was (incl. % added because it was a Core Indicator required in all World Bank projects. achievement) Indicator 22 : Targeted clients - female (Core supplement) Value (quantitative or 0 10,000 11,272 qualitative) Date achieved 06/10/2015 02/15/2017 02/15/2017 Comments Exceeded: 112.7%. At the time of the 2015 Restructuring, this indicator was added (incl. % because it was a Core Indicator required in all World Bank projects. Women’s participation achievement) in INIAF technical and administrative roles was low, but their participation as beneficiaries in subprojects was much higher. Indicator 23 : Agricultural innovations developed using participatory methods Value (quantitative or 0 10 18 qualitative) Date achieved 06/10/2015 02/15/2017 02/15/2017 Comments Exceeded: 180%. At the time of the 2015 Restructuring, this indicator was added to show (incl. % the degree to which INIAF was involving beneficiaries in the innovation process. This achievement) result is explained by project support for INIAF’s strategic communications efforts and funding competitions which encouraged wide participation of the agricultural R&E sector. INIAF also established a Competitive Research Fund through which INIAF implemented 23 applied and adaptive research subprojects based on priorities established through the participatory DICs. xiv Indicator 24 : HR, RM, FM, Procurement and Legal functions are effectively implemented Value (quantitative or 0 5 5 5 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 02/15/2017 Comments Achieved: 100%. At the time of the 2015 Restructuring, this indicator was revised to (incl. % facilitate measurement. Revised indicator: Management systems screened and approved by Achievement) INIAF’s internal audit. Management systems were defined to include the eight functions that had to be implemented under the SAFCO law. No baseline was provided. INIAF’s institutional capacity evolved with project support. Indicator 25 : INIAFs Management Information System (MIS) is fully effective Value (quantitative or 0 100 100 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 Comments Achieved: 100%. The management information system used by INIAF was called (incl. % SISPOA. Progress achieved with respect to this indicator was to be measured based on the achievement) percentage of SISPOA modules that were fully functional. At closing, nine (9) modules were operational, but INIAF was using only six (6) modules. Three (3) additional modules designed for use in managing subprojects were available, but INIAF was not using them regularly because subproject implementation was winding down. Indicator 26 : INIAF's network of national and departmental offices (established) Value (quantitative or 5 12 16 15 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 02/15/2017 Comments Substantially achieved: 94%. [Note: The word “established” was inadvertently omitted (incl. % from this indicator in the PAD.] This indicator was revised at the time of the 2015 achievement) Restructuring. Revised indicator: INIAF national and departmental offices that are performing their basic functions (#). The end-of-project target was increased to 16. The target was achieved, but then one departmental office was closed at project closing due to lack of budget. Indicator 27 : Number of formal collaborations INIAF had established Value (quantitative or 0 20 16 17 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 02/15/2017 Comments Exceeded: 106.25%. This indicator was revised at the time of the 2015 Restructuring. (incl. % Revised indicator: Formal collaborations with national and international counterparts achievement) under implementation. By that time, INIAF had established 37 collaborations, but many were not being actively implemented. The end-of-project target was revised to focus on active collaborations. Indicator 28 : Percentage of women and indigenous people in an institutional or research management position in INIAF Value (quantitative or N/A 20 20 3.85 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 02/15/2017 xv Comments Not achieved: 19.25%. At the time of the 2015 Restructuring, this indicator was revised to (incl. % reflect the fact that women and indigenous people were sometimes performing achievement) management functions in INIAF even if they were not formally classified as managers. Revised indicator: INIAF management positions held by women and/or indigenous people (%). The end-of-project target did not change. This result stems from the lack of effort by INIAF to increase the participation of women in technical and administrative positions /roles. The situation for indigenous people was much the same. Participation of women and indigenous people in innovation and technical assistance subprojects was much higher. Indicator 29 : User perception of the quality of organizational strengthening Value (quantitative or 0 N/A 50 31 qualitative) Date achieved 06/13/2011 02/15/2017 02/15/2017 02/15/2017 Comments Partly achieved: 62%. A qualitative Beneficiary Assessment was conducted for the Mid- (incl. % Term Review, and a quantitative Beneficiary Assessment was planned when subprojects achievement) reached a critical number of end-users. This indicator was revised at the time of the 2015 Restructuring. Revised indicator: INIAF internal users (staff) satisfied with the administrative support (%). The end-of-project target was set at 50%. The fact that many INIAF staff were not satisfied with the administrative support they received within their own institution reflects one of the challenges that INIAF should address in the near future. G. RATINGS OF PROJECT PERFORMANCE IN ISRs Date ISR Actual Disbursements No. DO IP Archived (USD millions) 0.00 1 09/18/2011 Satisfactory Satisfactory Moderately 0.70 2 03/28/2012 Satisfactory Satisfactory Moderately 2.09 3 11/09/2012 Moderately Satisfactory Satisfactory Moderately 3.16 4 02/24/2013 Moderately Satisfactory Satisfactory Moderately 6.35 5 09/24/2013 Moderately Satisfactory Satisfactory Moderately 11.29 6 04/11/2014 Moderately Satisfactory Satisfactory Moderately 14.71 7 11/07/2014 Moderately Satisfactory Satisfactory Moderately 20.68 8 06/29/2015 Moderately Satisfactory Satisfactory Moderately 26.63 9 12/31/2015 Moderately Unsatisfactory Unsatisfactory Moderately 29.17 10 03/23/2016 Moderately Unsatisfactory Unsatisfactory Moderately 30.55 11 05/26/2016 Moderately Unsatisfactory Satisfactory Moderately 34.11 12 12/07/2016 Moderately Satisfactory Satisfactory Moderately 34.88 13 02/11/2017 Moderately Unsatisfactory Satisfactory xvi H. RESTRUCTURING (IF ANY) ISR Ratings at Amount Restructuring Board Approved Restructuring Disbursed at Reason for Restructuring Date(s) PDO Change Restructuring in & Key Changes Made DO IP USD millions Level 2 Restructuring extended the co- financing effectiveness deadline of the co-financing agreement with the Swiss 05/22/2012 S MS 0.70 Development Corporation (SDC) by two years from June 1, 2012 to June 1, 2014 due to policy changes affecting SDC’s commitment to PISA of US$2.6 million. Level 2 Restructuring of IDA Credit 50030-BO to allow the following changes requested by Government: (i) expansion of the number of Departments in which TA subprojects 06/10/2015 MS MS 20.68 could be piloted; (ii) authorization to use an additional procurement method; and (iii) modification of the Results Framework to better align selected indicators with project objectives and facilitate indicator measurement. Level 2 Restructuring of IDA Credit 50030-BO to allow the reallocation of 02/17/2016 MU MU 29.17 Special Drawing Rights (SDR) 7.12 million from Category 2 to Category 1 to cover higher than expected expenditures. xvii I. DISBURSEMENT PROFILE xviii 1. Project Context, Development Objectives, and Design 1.1. Context at appraisal 1. Country and Sector Background. In 2011 when the Agricultural Innovation and Services Project was appraised, Bolivia was one of the poorest countries in Latin America. About 45 percent of the population of 10.1 million was living in poverty, including nearly 21 percent living in extreme poverty. Agriculture was one of the most important sectors in the economy, contributing just under 13 percent of GDP and accounting for just under 30 percent of total employment. Yet performance in the agricultural sector was disappointing; productivity in Bolivia was substantially lower than in neighboring countries due to the widespread use of antiquated technologies, limited uptake of modern inputs, and lack of knowledge and skills among farmers, among other factors. With many in the population dependent on agriculture as their principal livelihood source, increasing productivity in the agriculture sector was a national priority. 2. Government Sector Strategy. The 2006-2011 National Development Plan (NDP) of the Ministry of Planning and Development focused on four priority areas: (i) Social and Community Development (Bolivia Digna), (ii) Decentralization (Bolivia Democratica), (iii) Productive Development (Bolivia Productiva), and (iv) External Relations (Bolivia Soberana). The NDP, which stressed innovation and technological development as instruments for increasing sector productivity and competitiveness, called for the creation of a Bolivian innovation system capable of directing science and technology to the productive sector. The 2010-2015 Government Plan, continuing the general thrust of its predecessor, recognized that even though the agricultural sector had considerable potential, it was continuing to underperform because of low levels of technological development, weak institutional organization, and dependence on other productive sectors and internal demand. To address these challenges, the 2010-2015 Government Plan called for the creation of a National Institute for Innovation in Agriculture and Forestry (INIAF). 3. Rationale for Bank Assistance: The project as proposed by the Government was consistent with the NDP, giving priority to innovation and technology development as a means of improving agricultural productivity and competitiveness, and supporting the creation of a Bolivian innovation system capable of linking science and technology with the productive sector. The project as proposed was also well aligned with the Bank’s FY2010-2011 Interim Strategy Note for the Plurinational State of Bolivia (Report 48372-BO), which focused on four high level objectives or “pillars:” (i) Productive Development and Support to Production; (ii) Sustainable Development; (iii) Human Development; and (iv) Governance and Support to the Public Sector. The PISA project (Proyecto de Innovación y Servicios Agropecuarios) was intended to support primarily the first pillar of the Interim Strategy Note by strengthening the capacity of INIAF to lead the National System for Innovation in Agriculture and Forestry (SNIAF) and support research, technical assistance, and seed certification activities which would, over the medium to long-term, increase farm productivity and improve rural livelihoods. Bank financing in support of PISA thus responded to the shared vision of the Government and the Bank in which the public sector would assume a leading role in mobilizing a diverse coalition of partners to generate the innovations needed to drive development of the agricultural sector. 1.2. Original Project Development Objectives (PDO) and Key Indicators 4. The Project Development Objective (PDO) as stated in the Project Appraisal Document (PAD) was to strengthen INIAF and the Recipient’s National Agricultural and Forestry Innovation System (SNIAF) in order to contribute to productivity growth, food security, sustainable rural development and the income-earning potential of the Recipient’s families dependent on agriculture and forestry. 1 5. Achievement of the Project Development Objective was to be measured using three Key Indicators (PDO Indicators): PDO Indicator 1: Participation of research and extension (R&E) organizations in annual SNIAF meetings PDO Indicator 2: Adoption by producers of agricultural innovations generated by INIAF PDO Indicator 3: Perception by agricultural organizations that INIAF is an effective leader of the national agricultural innovation system 6. As stated in the PAD (paras 13 and 16), increases in agricultural productivity, food security, and the welfare of the rural population were recognized as higher level objectives to be achieved over the medium to long term, i.e., there was no expectation that they would be measurable within the life of the project. The support provided under the project was expected to lead eventually to the improved availability and use of innovations to benefit Bolivian families dependent on agriculture and forestry, but the primary emphasis during the project period was on pursuing shorter-term institutional objectives, specifically, the strengthening of INIAF and the SNIAF. 1.3. Revised PDO and Key Indicators 7. The original PDO did not change during the life of the project. The three PDO Indicators were not formally revised, although the wording of all three was adjusted in side notes as part of a Level 2 Restructuring carried out in May 2015. The adjustments were needed to facilitate measurement of the indicators, because the original wording lacked precision and was subject to multiple interpretations. In the case of PDO Indicator 1, the population of R&E organizations was precisely defined to include organizations active at both national and subnational levels, and the target was adjusted from 40 percent to 70 percent to reflect the higher expected participation rate of local organizations. In the case of PDO Indicator 2, it was specified that “agricultural innovations” consist of new plant varieties, new agricultural and forestry practices and other innovations such as equipment, and that “adopted” means adoption in process, to be measured annually through independent verification. In the case of PDO Indicator 3, it was clarified that the population of agricultural organizations among which the indicator should be measured includes the R&E organizations used for PDO Indicator 1 plus a sample of farmer organizations and seed producers.1 8. Subsequent to the 2015 Restructuring, by agreement between the Bank and the implementing agency, it was additionally specified that “perception of INIAF as an effective leader” would be measured taking into account the three dimensions of INIAF leadership covered in the PAD, namely INIAF’s ability to: (a) engage multiple stakeholders and coordinate their efforts; (b) lead public discussion around innovation and generate policy proposals; and (c) provide and mobilize financial resources for the engaged partners. 1 These adjustments do not fundamentally change the PDO Indicators and therefore do not justify an Appendix B split assessment of outcomes for this ICR. 2 1.4. Main Beneficiaries 9. The project’s intended direct beneficiaries were INIAF and the participants in the SNIAF (including universities, specialized research organizations, departmental and municipal extension systems, and NGOs involved in agricultural extension). 10. Indirect beneficiaries included all rural households dependent on agriculture or forestry, which were expected to benefit from increased production, increased incomes, and improved food security over the medium to longer term. The Social Assessment (SA) carried out during preparation estimated that potential project beneficiaries numbered around 3.1 million people, including about 1.6 million men and 1.5 million women. Nearly two-thirds (62 percent) of the target population identified themselves as indigenous peoples, which led to the entire operation being considered as an Indigenous Peoples’ Project (IPP). The practical implication of this was to obviate the need to screen project-supported activities one by one to determine whether they were likely to impact indigenous peoples and to prepare separate indigenous peoples plans for individual activities; instead, all project-supported activities were considered likely to impact indigenous peoples, so they had to be designed and implemented in conformity with the measures laid out in the project’s Indigenous Peoples Plan. 1.5. Original Components 11. The project consisted of four components. Component 1: Strengthening the National Agricultural and Forestry Innovation System (SNIAF) (estimated total cost US$ 6.1 million, of which IDA US$ 5.35 million) 12. Component 1 financed the strengthening of information and technology flows between people and institutions in the SNIAF, and it promoted the participation of research and innovation actors in the SNIAF. This was achieved through two sub-components: (a) Development of policies and strategies to improve governance of the SNIAF (US$ 1.18 million), which financed: (i) formulation of policies to support effective implementation of the SNIAF; (ii) development of a strategy to promote communication and information exchange among SNIAF participants; and (iii) establishment of a Plurinational Council for Agriculture and Forestry Innovation, made up of public and private stakeholders; (b) Establishment of a competitive research fund (US$ 4.92 million), whose purpose was to provide research subproject grants to support collaboration between national and international research entities and producer groups, for the purpose of identifying, testing, and disseminating innovations in areas other than those being covered by INIAF-led research programs/alliances financed under Component 2. Component 2: Strengthening core INIAF activities (estimated total cost US$ 26.17 million, of which IDA US$ 22.39 million) 13. Component 2 financed the strengthening of INIAF research, technical assistance, seed certification, and genetic resource management programs. This was achieved through four sub- components: (a) Support to INIAF research programs (US$ 12.44 million). This sub-component supported INIAF research programs in six strategic areas: wheat, potato, quinoa, dual purpose cattle, horticulture, and forestry. In addition to financing the salaries of INIAF staff and covering operational costs of INIAF research programs, project funds supported strategic alliances with national and international partners, which allowed INIAF to convene expertise and 3 resources not available in-house and achieve impacts more rapidly than could be achieved working alone. (b) Support to INIAF technical assistance services (US$ 4.85 million). This sub-component supported INIAF’s efforts to establish a national technical assistance system. Project funds supported: (i) preparation at national level of technical assistance capacity development programs and fine-tuning at sub-national level with input from local authorities; (ii) creation of nine Departmental Agricultural and Forestry Innovation Councils (DAFICs—which came to be called Departmental Innovation Councils—DICs) responsible for planning and carrying out agricultural innovation strategies and associated technical assistance programs; and (iii) implementation of two technical assistance and rural extension pilots identified by the DAFICs (DICs). (c) Consolidation of the national seed system (US$5.26 million). This sub-component supported: (i) promotion in the western districts of the country of use of quality seeds; (ii) upgrading INIAF seed program laboratories; and (iii) strengthening of the regulatory framework relating to seed production, certification, and marketing. (d) Management of genetic resources (US$3.63 million). This sub-component supported the establishment of a national network of germplasm banks, comprising core banks (used mainly for long-term conservation of genetic resources), active banks (used mainly to provide genetic materials for studies on genetic variability), and working banks (used mainly to provide genetic materials for breeding programs). Component 3: Supporting INIAF’s institutional development (estimated total cost US$16.38 million, of which IDA US$ 7.13 million) 14. Component 3 financed the strengthening of INIAF institutional and organizational capacity. This was achieved through two sub-components: (a) Improving the organizational and internal capacities of INIAF (US$ 15.23 million). This sub-component supported: (i) consolidation of an institutional management model reflecting INIAF’s national and sub-national roles; and (ii) training and capacity-building activities to improve the efficiency of administrative and institutional staff. Special attention was directed to Human Resource Management—especially the adoption of recruitment, personal development, and rewards policies enabling INIAF to compete with universities and other knowledge-based organizations—and to Financial Management and Resource Management. This sub-component also allowed INIAF to expand its geographic presence by opening or improving departmental offices. (b) Strengthening capacity for external leadership and collaboration (US$ 1.13 million). This sub-component supported activities designed to strengthen the capacity of INIAF to serve as the leader of the SNIAF, including: (i) training and capacity building activities for INIAF staff to strengthen their technical, administrative, and management skills and improve their ability to partner effectively with a range of external partners and stakeholders; and (ii) training and capacity building activities for members of the INIAF Board of Directors to improve the Board’s ability to carry out oversight functions for INIAF and the SNIAF. Component 4: Project Management (US$4.21 million, of which IDA US$4.12 million) 15. Component 4 financed activities related to the management of the project. This was achieved through two sub-components: 4 (a) Project implementation (US$ 2.85 million). This sub-component supported all activities related to project administration, fiduciary management, technical supervision, and safeguards compliance. (b) Monitoring and evaluation (US$ 1.36 million). This sub-component supported: (i) monitoring and evaluation functions, for the purpose of monitoring progress and documenting the results, outcomes, and impacts of project-supported activities; and (ii) building capacity within INIAF to measure and showcase the value added by INIAF and by the SNIAF. 1.6. Revised Components 16. During implementation, the scope of several components was adjusted in response to changing circumstances in the larger environment.  Sub-component 2(a): The original project design called for support to six INIAF research programs (wheat, potato, quinoa, dual purpose cattle, horticulture, and forestry). Based on a request received from the Ministry of Rural Development and Lands, INIAF decided (and the cofinanciers agreed) to provide support to two additional programs focusing on maize and rice, crops considered critical for food security in Bolivia.  Sub-component 2(b): The original design called for the project to support two large technical assistance pilots, to be co-financed by two Departmental governments. The use of local cofinancing was consistent with the Constitution of 2009 and the Decentralization Law of 2010, both of which had been enacted during a period when the central government was vying for power with decentralized governments, and which tasked Departmental governments with providing technical assistance to producers. By the time the project started, however, the tension between the central government and the decentralized governments had eased, leading the Ministry of Rural Development and Lands to ask INIAF to increase its role in providing technical assistance and reduce its reliance on external partners, including the Departmental governments. During the second year of implementation, before the two originally planned technical assistance pilots had been launched, INIAF proposed and the World Bank and Danida and COSUDE agreed to achieve greater geographical coverage by replacing the two pilots with nine smaller subprojects, to be co-financed by producer organizations (this was similar to what was being done under other World Bank and Danida projects). 17. The project was restructured on three occasions:  May 2012: A Level 2 Restructuring was carried out prior to effectiveness to extend the time available for the Swiss Development Corporation (SDC) to mobilize the financial resources it had committed to the project as cofinancing.  June 2015: A Level 2 Restructuring was carried out following the Mid-Term Review to implement three recommendations that had emerged from the Mid-Term Review: (i) the number of technical assistance subprojects that could be piloted was increased from two to 10; (ii) use of the Community Participation in Procurement method was authorized under subprojects; and, (iii) adjustments were made to the Results Framework to better align some performance indicators with project objectives and facilitate measurement of the indicators in selected cases. Three new Intermediate Indicators were added that would better demonstrate achievement of the PDO, as were two recently introduced Core Indicators that were required under World Bank policies. At the same time, several indicators over which the project had little control and which in any case could not be measured were dropped.  February 2016: A Level 2 Restructuring was carried out to reallocate US$ 7.12 million from Category 2 (subprojects) to Category 1 (INIAF direct costs). 5 2. Key Factors Affecting Implementation and Outcomes 2.1. Project Preparation, Design, and Quality at Entry 18. Soundness of the background analysis: The Bank team that prepared the project systematically reviewed earlier efforts that had been made in Bolivia to strengthen innovation in the agriculture sector and was well informed about the lessons that had been learned. Team members were aware that there had been a lack of continuity in research and extension services, which had led to an erosion in knowledge and a weakening of institutional capacity to support innovation. In 1975, the Bolivian Institute of Agricultural Technology (IBTA) was created with a mandate to conduct research and provide extension services, but research priorities were not demand-driven, because IBTA lacked the institutional capacity to identify and respond to the needs of producers (Hameleers et al., 2012). The Bolivian System for Agricultural Technology (SIBTA), created in 2000, replaced IBTA. SIBTA, a decentralized system comprising four research institutes focusing on different agro-ecologies, generated technology based on knowledge previously acquired or obtained from other countries, but there was little involvement of other institutional actors involved in innovation. Because SIBTA did not respond to producers’ actual needs, contributed little in the way of innovations, and generated limited benefits, it was closed in 2006. Two years later in 2008, the Government with the help of financing provided by Danida created a new institution, the National Institute for Agriculture and Forestry Innovation (INIAF). The Agricultural Innovation and Services Project (PISA) was conceived as an operation to strengthen the new institution. 19. Taking note of lessons learned during earlier attempts to promote research and extension in Bolivia, the Bank and the Government agreed to use an innovation system approach, rather than attempting to recreate an old style public national agricultural research service. The innovation system approach had two distinctive features. First, while INIAF was seen as the leader of the national innovation system, it was recognized that INIAF lacked the capacity to carry out all of the innovation activities that were needed in the country; therefore, implementation responsibility would be spread across a diversified innovation network made up of many decentralized partner organizations. Second, unlike earlier periods when agricultural research and extension activities were centrally decided and implemented in top-down fashion by IBTA, it was recognized that the innovation agenda should be demand-driven, responsive to the needs of food system participants. Throughout preparation, there was strong consensus around the critical importance of these two features. By the time implementation started, however, senior management within the Ministry of Rural Development and Lands and INIAF had changed, and later it became evident that the new leadership did not share the same vision regarding the role of INIAF and the way it would operate. 20. Assessment of project design: The project design was logical and coherent. The project components sensibly separated externally focused activities intended to build a strong SNIAF from internally focused activities designed to strengthen INIAF’s own capacities. The competitive grants program provided a suitable instrument for channeling resources to partners and stakeholders while at the same time leveraging additional resources by requiring matching contributions for subprojects. Numerous capacity strengthening activities supported by the project addressed the well-recognized lack of human capacity at all levels. 21. With the benefit of hindsight, it is clear that successful implementation of the project design required sustained commitment to the innovation network concept on the part of the Government, strong and stable leadership within INIAF, and competent and motivated INIAF staff familiar with the policies and procedures articulated in the Operational Manual (which reflected Bank administrative norms). In areas where those conditions prevailed (e.g., the INIAF Seed Program, several of the INIAF research programs), the project achieved or exceeded its objectives. But in areas where those conditions were lacking, implementation performance fell short of expectations. 6 22. Technical strategy: The decision by the Government and the Bank to jointly finance PISA reflected their shared belief that agricultural development in Bolivia was “knowledge dependent” and would depend critically on productivity gains driven by innovations. The central strategy of building a strong and dynamic SNIAF and strengthening INIAF’s leadership role in applied research, technical assistance, and seed certification reflected the multi-dimensional nature of the challenge and the need to make progress simultaneously on several fronts. The emphasis on collaborative models (partnerships for the competitive research fund, alliances for priority research products, technical assistance pilots with Departmental governments) acknowledged that INIAF alone would not be able to provide the required expertise in every area. 23. Financing: The project was financed with resources provided by the government of Bolivia (US$ 16.4 million), the World Bank (US$ 32.5 million), the Kingdom of Denmark through Danida (initially US$ 2.9 million later increased to US$ 3.3 million), and the Swiss Confederation through COSUDE (initially US$ 2.6 million, later increased to US$ 5.1 million). Responsibility for supervising the project was assigned to the World Bank under a joint supervision arrangement that called for the use of standard World Bank operational policies and procedures. 24. Implementation arrangements: During the life of the project, INIAF’s budget was financed almost entirely with project resources. As a result, PISA was not implemented as a discrete project within INIAF; rather, INIAF itself functioned as the de facto project implementation agency. The organization of INIAF was relatively straightforward; the Executive Director of INIAF oversaw a small coordination unit, a unit charged with managing the Research Fund, and national directorates for Research (DNI), Seeds (DNS), Technical Assistance (DNAT), and Administration and Finance (DAF). A PISA Project Coordinator reporting directly to the INIAF Executive Director theoretically was responsible for overseeing day-to-day activities of the project, but because these were basically indistinguishable from INIAF’s overall work program being managed by the INIAF Executive Director, the Project Coordinator ended up playing a limited role. In addition, a formal assistance role was vested in a Technical Assistance Committee (Comité de Acompañamiento) established to include the Bank, Danida, and COSUDE in their capacity as cofinanciers, as well as GTZ as a technical partner. General oversight of the project was to have been provided by the INIAF Board of Directors, which according to the governance procedures spelled out in the Operational Manual, was supposed to serve as the Project Steering Committee. In the end, the Board of Directors played a limited role, however, because the Ministers of Rural Development and Lands who held office during the life of the project continuously intervened directly with the INIAF Executive Directors. This points to a weakness in the implementation arrangements: the fact that Ministers of Rural Development and Lands viewed INIAF in general and the PISA project in particular as vehicles for implementing ministerial policies meant that INIAF, in practice, did not achieve the degree of autonomy envisaged at design. 25. Assessment of risks and mitigation: The Operational Risk Assessment Framework (ORAF) in the PAD rated Overall Project Risk as High, based on the uneven history of agricultural research and extension in Bolivia, weak institutional capacity in public agencies, potential lack of buy-in on the part of key stakeholders, and operating environment risks. In retrospect, the ORAF was accurate, as many of the risks identified at preparation materialized during implementation, e.g., implementation delays due to capacity constraints within INIAF, frequent changes in project management leading to a lack of continuity in operations, slow payment of counterpart funding by the Government, and challenges in ensuring the sustainability of project investments. Mitigation measures described in the ORAF were reasonable, but they proved inadequate in a number of cases, often because of capacity constraints within INIAF combined with a lack of political commitment on the part of the Borrower to take necessary remedial measures. 7 2.2. Factors affecting project implementation 26. Implementation progress ratings fluctuated between Satisfactory (S), Moderately Satisfactory (MS), and Moderately Unsatisfactory (MU) for diverse reasons including an inconsistent implementation trajectory characterized by long stretches of slow progress interspersed with brief periods of rapid gains. Some activities were adversely affected by institutional, administrative and technical capacity constraints, which was not unexpected given that INIAF was a new institution struggling to advance on many fronts simultaneously in a complex and rapidly evolving sector. Furthermore, weaknesses in the M&E system prevented the Borrower from demonstrating evidence of achievement until very late in the life of the project, which caused the supervision team to be very conservative in assessing implementation progress. As the project closing date approached, results reporting finally improved: in late 2016, INIAF presented solid evidence that the project was generating innovations and that these innovations were beginning to make their way into farmer’s fields and deliver benefits. This evidence confirmed that the project had many successes, even though INIAF for a long time had difficulty in documenting these successes. In January 2017, despite the availability of evidence showing that the PDO indicators and most of the Intermediate Outcome Indicators had been achieved, the rating was again downgraded from MS to MU, because at that time the sustainability of the development outcome seemed at risk. In the months leading up to closing, little evidence existed that the Government intended to allocate budget resources to implement the post-project institutional sustainability plan prepared by INIAF with support from COSUDE and Danida. 27. Lack of stable leadership in INIAF: Implementation was affected by a lack of stable leadership in the implementing agency. Over the life of the project, INIAF had five Executive Directors and four Project Coordinators. Every incoming Executive Director instituted sweeping changes to the mid-level management team. The constant turnover, which included protracted vacancies in key positions, had cascade effects down the chain of command. The resulting loss of continuity and institutional memory negatively affected the performance of administrative and technical staff responsible for project execution. The repeated need to reconstruct the leadership team, build understanding of the project’s objectives, and regain momentum in key programs and activities, caused delays. The erosion of institutional memory was especially pernicious, contributing in the early years to misinformation and contradictory messages about INIAF’s mandate, its activities, and the relationship between PISA and INIAF. Persistent confusion at the top reduced the motivation of staff whose stability and loyalty were critical to project success and more importantly to the long-term fortunes of INIAF and the SNIAF. The critical importance of leadership is reflected in the fact that the project achieved its greatest successes in areas that benefited from continuous, high-quality leadership (e.g. the strategic research program for wheat and the seed certification program). 28. Ineffective staff recruitment processes in INIAF: Throughout the life of the project, INIAF suffered from a shortage of well-qualified and highly motivated administrative and technical staff. Recruitment processes often resulted in the hiring of under-qualified candidates or in failure to identify any qualified candidates. This was due in part to a challenging external environment: the Bolivian market for qualified professionals is thin, and the skills sought by INIAF were often scarce, especially in rural areas where many INIAF staff were needed. But the recruitment process itself was imperfect. To shield hiring and firing decisions from political influence, it was agreed at negotiations that recruitment and evaluation of INIAF staff paid under the project would be managed by an external firm. While this is standard procedure for many donor-funded projects, it differs from the practices typically prevailing in Bolivian government agencies, where hiring decisions are often influenced by subjective and/or political factors. Entrusting recruitment and evaluation to an external firm frustrated the Borrower. Senior management in the Ministry of Rural Development and Lands and in INIAF consistently criticized the work of the external firm and on 8 several occasions was reluctant to accept candidates selected using the agreed criteria. This created tension between the Borrower and the development partners and on occasion resulted in long delays in filling key positions. 29. Change in the Borrower’s view about the role of INIAF: The Borrower’s view of INIAF’s role changed over time. The vision of senior officials in the Ministry of Rural Development and Lands and in INIAF who assumed office after project preparation diverged in three important respects from the vision of those who had prepared the project. First, the design called for INIAF to strengthen the SNIAF by transferring resources to partners to finance activities that INIAF itself was unable to carry out. By the time implementation started, however, senior MDRyT leadership was more interested in having INIAF itself conduct those activities, believing that INIAF’s primary role was to implement government policies directly. As a result of the change in attitude, transfers of subproject funds to partners never reached planned levels; nearly 70 percent of the funds originally allocated to subprojects were re-directed to financing INIAF self-executed activities. Second, while the design stipulated that the INIAF Board of Directors would provide oversight, once implementation started the line Ministry exerted control over INIAF and supervised implementation directly, marginalizing the INIAF Board of Directors and preventing INIAF from establishing the degree of autonomy envisaged. Third, although the design called for INIAF to focus on strategic issues requiring sustained, long-term effort, the Ministry re-oriented INIAF’s agenda toward urgent problems requiring immediate solutions. Even so, views within the Ministry were not homogeneous, and they changed over time as key ministerial staff changed. Some ministerial staff adhered to the view that prevailed when the project was designed, and these staff supported the efforts of the cofinanciers to empower INIAF to operate semi-autonomously, maintaining a focus on strategic, long-run objectives. 30. Lack of a strong champion for INIAF and ineffective communication strategy: INIAF lacked a strong champion to advocate for its interests within the Government. This problem was largely self-induced: INIAF did a poor job of raising its visibility and communicating its activities and accomplishments to the public and to influential politicians and policy-makers. INIAF was largely absent from many public spaces in which it could have demonstrated leadership in agriculture and forestry innovation. The development partners repeatedly urged INIAF management to prepare and implement a corporate communications strategy to establish a public presence; build awareness of its mission, activities, and achievements; exert itself as articulator of the SNIAF; establish a presence at national and departmental gatherings; and promote greater operational coordination between stakeholders. INIAF did eventually prepare and launch a communications strategy, but it was slow in building the political constituency needed to ensure continued funding at required levels following project closing. 31. Strong and sustained support from multiple development partners: Implementation benefited from strong and sustained support from the three cofinanciers and the development partner that provided technical assistance. In addition to participating in regular implementation support missions, World Bank staff based in La Paz or elsewhere in the region met weekly with INIAF counterparts to provide guidance, training and/or hands-on support. COSUDE and Danida provided not only financial resources, but also valuable administrative and technical assistance. They worked through the Technical Assistance Committee to engage numerous short-term and long-term consultants working with INIAF and the Bank to support INIAF’s efforts to establish the SNIAF, strengthen its strategic research programs, build capacity to provide technical assistance, and carry out M&E activities. German Technical Cooperation (GIZ) contributed technical expertise in several key areas. 9 2.3. Monitoring and Evaluation (M&E) Design, Implementation, and Utilization 32. M&E design: The project M&E system was designed for three purposes: (i) documenting the institutional development of INIAF; (ii) tracking operational procedures and human resources development; and (iii) monitoring and evaluating project-supported activities, results, and impacts. According to the M&E plan, the data needed to document progress achieved with respect to the PDO and Intermediate Outcome Indicators would be sourced from INIAF’s three technical units (Research, Technical Assistance, Seeds) and compiled by INIAF’s Planning and Organizational Development Unit. The PAD recognized that data collection and processing were likely to be challenging, because the technical units operated independently, each according to its own standards and formats. The M&E Plan therefore called for the introduction—as part of the project’s overall Management Information System—of an M&E module capable of compiling data from different sources into a homogeneous format. The PAD noted that the M&E plan was still a work in progress and would be finalized by INIAF during implementation. 33. The Results Framework (RF) had some design shortcomings. The PDO is reasonably clear, and although it makes reference to several higher level outcomes and impacts that will take time to materialize, the PAD helpfully included explicit statements indicating what could reasonably be measured within the life of the project. But if the PDO is reasonably clear, assessing achievement of the PDO is challenging, in the sense that it is difficult to identify easily measurable indicators of strengthened institutional capacity. In principle, the three PDO Indicators as defined in the PAD appropriately focused on factors of relevance to the PDO, but in practice several lacked definition, and methodologically sound baselines were not prepared, making measurement difficult. To resolve this issue, the definitions of two of the PDO Indicators were clarified as part of the 2015 Restructuring, and the procedures to be used to measure the indicators were precisely defined. 34. Several of the Intermediate Outcome Indicators also proved problematic, for three reasons. In some cases, the procedure to be used to measure the indicator was unclear (for example when the indicator related to the level of satisfaction prevailing among a set of beneficiaries, but the population of beneficiaries was not precisely defined). In other cases, the indicator referred to an outcome over which the project had little or no influence (for example, entering into international treaties relating to the use of genetic resources). And in still other cases, an indicator became obsolete due to changes in the environment in which the project was operating. To resolve these issues, as part of the 2015 Restructuring adjustments were made to several Intermediate Outcome Indicators through clarifications, modification of targets, elimination of some indicators and inclusion of new ones. 35. The adjustments made to the PDO Indicators and the Intermediate Outcome Indicators at the time of the 2015 Restructuring did not substantially alter the scope, focus, or level of achievement of the expected project outcomes. The effect of the adjustments was to more closely align the Results Framework with the PDO and allow measurement of a set of indicators capable of capturing the expected strengthened institutional capacity within INIAF to generate innovations and deliver them to the intended beneficiaries. 36. M&E implementation: M&E implementation was uneven and initially did not serve the project well. By the end of the first year, none of the performance indicators had been measured. At the insistence of the Bank, project management prepared an emergency action plan to jump-start the M&E function. The Bank, with support from M&E specialists secured either through the World Bank-FAO Collaborative Program or via consultancies, worked closely with INIAF to devise an M&E strategy, prepare methodological guides for the collection of M&E data, manage the preparation of the M&E module to form part of the MIS, and promote collaboration between INIAF’s technical directorates and departmental offices to improve data collection activities. 10 Despite these efforts, progress was uneven, due mainly to constant staff turnover within the INIAF M&E Unit compounded by the failure of the INIAF Planning Unit to complete agreed activities. 37. In the second year of implementation, after observing the slow rate of progress, the Bank strongly recommended outsourcing the development of the M&E system, with the goal of having a fully functional system in place by the time the first wave of subprojects was launched. The idea found little support in INIAF. Instead, project management decided to proceed with its plan to develop the M&E system in-house as part of the larger management information system (MIS). Consultants were hired to develop two software systems that could be used to support the planning and implementation of project-supported activities, as well as to generate M&E reports. The first software system, SISPOA, was designed mainly for use with activities being implemented directly by INIAF. The second software system, SISPOA-AE, was designed for use with activities being implemented by INIAF in collaboration with partners. Although SISPOA and SISPOA-AE did not come on line until late in the life of the project, the fact that they were eventually implemented shows that under PISA INIAF acquired tools for strengthening planning capacity, mainstreaming results-based management, and systematizing the use of computerized M&E systems. 38. With respect to the M&E system, an important legacy of the project dates back to the time of the 2015 Restructuring, when project management and the Bank team systematically reviewed the Results Framework and revisited the M&E Plan. As part of this effort, a cloud-based depository was created containing: (i) a detailed description of every performance indicator (definition, measurement procedure, formulas for calculation, etc.); (ii) data showing the historical evolution of the measurements for each indicator; and (iii) the latest report on the status of each indicator, including the means of verification. This mechanism, jointly administered by the Bank and INIAF, proved effective in systematizing information about project results and reducing losses of institutional memory due to staff turnover. 39. M&E utilization: Because the project M&E system became functional only very late in the life of the project, use of M&E data was relatively limited. At closing, data collected via the SISPOA and SISPOA-AE systems were being used to inform budgetary planning and administrative decision-making, but data being managed through SISPOA and SISPOA-AE were not being fed automatically into the results matrix. Still, by project closing the M&E system was in place, supported by manuals, procedures, and tools prepared under the project to support the M&E function. To the extent that the SISPOA and SISPOA-AE systems remain in use, INIAF will have an opportunity to benefit from their capacity to generate information that can be used to quantify and assess a range of output and impact indicators. For this opportunity to be realized, however, INIAF will have to strengthen the technical and operational capacities of the M&E Unit while continuing to improve the functionality of SISPOA and SISPOA-AE. 2.4. Safeguards and Fiduciary Compliance 40. With respect to safeguards, the Project was classified Category B. Five safeguards policies were triggered: Environmental Assessment (OP/BP 4.01), Natural Habitats (OP/BP 4.04), Forests (OP/BP 4.36), Pest Management (OP 4.09) and Indigenous Peoples (OP/BP 4.10). The Results Framework included no performance indicators related to the implementation of the Environmental Management Plan (EMP). The Results Framework contained one Intermediate Outcome Indicator measuring the percentage of women and indigenous people in a management or research position in INIAF (the definition of which was refined by the 2015 Restructuring to include women and indigenous people having a decision-making role in INIAF). 41. Environmental safeguards: Environmental safeguards compliance was rated Satisfactory throughout implementation. An Environmental Assessment (EA) was carried out during preparation, and an Environmental Management Plan (EMP) was prepared and disclosed in Bolivia 11 and on the World Bank website. The high staff turnover rate in INIAF on occasion affected safeguards monitoring, as there were brief periods when the Environmental Specialist position within INIAF remained vacant, but this was not considered problematic, because Bank Environmental Specialists were making regular visits to sub-project sites and had not detected any issues. The project’s research, technical assistance, and seeds certification activities, assessed during preparation as posing low risk to the environment, did not generate any significant negative environmental impacts. Moreover, INIAF took advantage of the project to promote environmental good practices, for example by producing manuals and guides and organizing field days and training events designed to build awareness of and familiarity with conservation agriculture, integrated pest management, and organic production methods, among others. 42. Social safeguards: Social safeguards compliance was rated Moderately Satisfactory throughout much of implementation. During preparation, a Social Assessment (SA) was carried out, and a Social Consultation and Participation Plan (SCPP) was prepared and disclosed in Bolivia and on the World Bank website. Because an estimated 61 percent of the target population self- identified as indigenous, the Social Assessment recommended that separate Indigenous People Plans not be prepared for individual subprojects; instead, the Social Assessment recommended that the entire project be considered subject to the Indigenous Peoples Policy and that all of its activities be conducted accordingly. At the time the project was prepared, this was standard practice for all rural development projects in Bolivia. During implementation, supervision of social safeguards compliance was deficient at times, again due to the high rate of staff turnover in INIAF, which prevented the Social Specialist position within INIAF from being continuously filled. This was somewhat problematic, because a small number of complaints were registered concerning alleged damages suffered as a result of project-supported activities, yet the mechanism for addressing complaints was weak. The Bank team responded to these constraints by visiting the concerned sites, accompanied whenever possible by the INIAF Social Specialist. It was determined that no clear protocol was in place for conducting effective prior consultations with local communities, and as a result multiple and inconsistent methods had been used to socialize local communities, often without appropriate documentation. Notwithstanding these issues, the Bank team was able to establish through field visits to the affected sites that, despite the inconsistent application of the Indigenous Peoples policy, no harm had been done to indigenous peoples. On the contrary, almost all of those contacted expressed their satisfaction with the support that they had received through the project, especially the technical assistance. 43. Procurement: Procurement was rated Moderately Satisfactory by every supervision mission but two. Procurement for the project was carried out in accordance with the provisions stipulated in the Legal Agreement, the World Bank “Guidelines: Procurement of Gods, Works and Non consulting Services under IRBD Loans and IDA Credits and Grants by World Bank Borrowers” dated January 2011, and the “Guidelines: Selection and Employment of Consultants under IRBD Loans and IDA Credits and Grants by World Bank Borrowers” dated January 2011. When the project first got under way, INIAF’s procurement capacity was weak. Previously INIAF procurement staff had handled mainly small contracts using national procurement norms, and they were unfamiliar with World Bank policies and procedures. Several action plans were prepared containing formal and informal training activities designed to improve the capacity of the INIAF procurement team, and Bank procurement staff provided intensive hands-on support to the procurement unit. In addition, during implementation the procurement procedures detailed in the Operational Manual were revised to simplify selected processes, which helped to streamline procurement activities. Over time, the capacity of the INIAF procurement unit improved considerably, and by project closing it was capable of conducting complex procurement processes using different methods, including International Competitive Bidding. This is not to say, however, that there were no challenges along the way. As in other areas, the high rate of staff turnover negatively affected procurement activities, as new staff were constantly coming on board who had 12 limited knowledge of World Bank policies and procedures and had to be trained on the job. Bidding documents often had to be returned for revisions, leading to delays. Finally, it is important to note that a World Bank investigation determined that a procurement process carried out in 2014 involving Danida funds had not followed World Bank procurement rules. INIAF was requested to remediate the issue. 44. Financial management and audit: The performance of the Financial Management and Audit functions varied throughout the life of the project. During the first three years, Financial Management was rated Moderately Unsatisfactory, because of chronic fiduciary problems that negatively impacted implementation. Perhaps not surprisingly in a new institution, INIAF took a long time to recruit qualified fiduciary staff, establish financial management and audit systems acceptable to the World Bank, adjust the Operational Manual, and finalize the formats of standard financial and audit reports. During the period when the Financial Management system was being put in place, performance suffered: Interim Financial Reports (IFRs) were consistently late and when submitted had to be returned for adjustment, audit reports were often late, reconciliation errors persisted in the designated account, accounting records were not updated, and funds were advanced to staff and to regional offices without appropriate supporting documentation. From late 2015 onwards, the Financial Management and Audit functions improved, in part because INIAF finally managed to stem the high rate of staff turnover and started to retain key fiduciary staff, which permitted significant capacity growth and resolution of long-standing issues. Following the Mid-Term Review, an action plan was put in place to address outstanding issues, including updating of accounting records, reconciliation of items in the designated account, recovery of advances and resolution of disbursement issues in subprojects. Reflecting the progress achieved as a result of the action plan, Financial Management was upgraded to Moderately Satisfactory in 2016. INIAF faced challenges during the closing period, particularly the loss of key fiduciary staff, but the fiduciary team focused on recovering and documenting outstanding advances and closely monitoring subprojects to ensure timely completion by the project closing date. At the time the ICR was being prepared, no IFRs or audit reports were due, and INIAF was preparing the final documentation due at the end of the grace period. 45. Project costs and financing: At appraisal, the total cost of the project was estimated as US$ 52.9 million, including funds from all sources (IDA, COSUDE, Danida, Government of Bolivia). At the time of the Mid-Term Review, the estimate was revised upward to US$ 58.38 million, based on additional resources that had been provided by the Government of Bolivia. According to INIAF, when the project closed the final cost stood at US$ 55.07 million, 104 percent of the estimate made at appraisal and 94 percent of the revised estimate made at the time of the Mid-Term Review. Of the final cost, 29.8 percent (US$ 16.44 million) came from the Government of Bolivia, 59 percent (US$ 32.5 million) came from IDA, 5.2 percent (US$ 2.82 million) came from COSUDE, and 6 percent (US$ 3.31 million) came from Danida. During implementation, the Government of Bolivia made an incremental contribution of US$ 7.44 million, citing the need for additional resources to strengthen human capacity in INIAF. COSUDE and Danida also increased their support, which was being provided under bilateral agreements between the two organizations and the Government of Bolivia that allowed for adjustments to be made if the amount of funding being provided. The agreement between COSUDE and the Government of Bolivia will expire in December 2017, and the agreement between Danida and the Government of Bolivia will expire in December 2018. 2.5. Post-completion Operation/Next Phase 46. Sustainability of INIAF’s institutional capacity: During the final year of implementation, INIAF management, with support from COSUDE and Danida, prepared a strategic plan for INIAF’s further development following the closing of the project. Covering the period 13 2017-2020, the strategic plan presents a vision for INIAF that is aligned with the Government’s development policies and programs, taking into account INIAF’s capacities and organizational structure, and building on lessons learned under PISA. The strategic plan, which proposes a series of reforms designed to better position INIAF to fulfill its mission over the longer-term, was adjusted during the first semester of 2017 to better align it with the sectoral development plan approved by the Minister of Rural Development and Lands in June 2017. These developments that took place following the final supervision mission have significantly improved sustainability prospects for the project’s achievements. Having prepared a clear strategy that is well aligned with sectoral policies and fully backed by the line ministry, INIAF is now much better positioned to secure funding in upcoming budget cycles. At the time the ICR was being finalized in July 2017, INIAF’s budget for 2017, including all financing sources, stood at around US$ 10 million, and the budgets proposed in the strategic plan call for this amount to increase modestly every year through 2020. These amounts are significantly larger than the budget projections reported during the final supervision mission carried out in January 2017, easing earlier concerns that the achievements under the project would not be sustainable. At the same time, the projected budgets (which it should be noted have not yet been approved) still seem insufficient to finance all of INIAF’s programs at the scale needed to have transformative impact at national level. For that reason, despite the recent positive developments and improved financing prospects for the immediate future, financing prospects over the longer term remain less certain. 3. Assessment of Outcomes 3.1. Relevance of Objectives, Design, and Implementation Overall rating: Modest 47. Relevance of objectives: The relevance of project objectives was High at appraisal, but it declined during implementation and was Modest at closing, resulting in an overall rating of Modest. At appraisal, the Government’s interest in supporting agricultural innovation was clear. As stated in numerous national economic development plans and other strategy documents, activities to be supported included generation of knowledge and technology in strategic productive and service areas, as well as building of systems capable of providing universal access to inputs, technology, technical assistance and other services to support production. Strengthening of strategically important institutions was specifically mentioned, including INIAF, the National Service for Agro- livestock Health and Food Safety (Servicio Nacional de Sanidad Agropecuaria e Inocuidad Alimentaria - SENASAG), and the Institute for Agrarian Insurance (Instituto del Seguro Agrario - INSA). In this context, project objectives were extremely relevant, as the PDO directly addressed two elements of the Government’s larger strategy: (i) establishing INIAF as the leader of a dynamic, diversified national innovation system (SNIAF) that could leverage the capacities of many different actors, and (ii) strengthening the capacity of INIAF to directly carry out critical research and extension activities. By closing, however, the strategy of the Government had shifted, as had its vision for INIAF, particularly regarding INIAF’s role in providing financial resources to external partners. After witnessing the difficulties that had been experienced in transferring financial resources to subprojects carried out by external partners, senior officials were less interested in establishing INIAF as a financier of the SNIAF. Moreover, while officials in the Ministry of Rural Development and Lands remained committed to strengthening INIAF’s capacity to directly carry out critical research and extension activities, they had not come up with a mechanism to ensure the long-term financing needed to make this happen. Because of the shift in Government strategy that occurred during implementation, by closing the project objective linked to strengthening the SNIAF (by providing financial resources through INIAF) had become less relevant. 14 48. While the relevance of the project objectives to the Government declined during implementation, the relevance to the World Bank’s strategy for Bolivia remains Substantial. The Country Partnership Framework (CPF, 2016-2020) includes among its objectives to “improve opportunities for income generation, market access and sustainable intensification.” The CPF supports the Government’s longer-term agenda to develop the non-extractive sectors by increasing productivity, especially in agriculture. In this context, the project objectives of building the SNIAF and strengthening INIAF’s own capacity remain relevant. 49. Relevance of design: The relevance of the project design is rated Substantial. The organization of the project components and the nature of the activities financed under the components were well aligned with the project objectives. Project-supported investments were oriented around strengthening “hardware” and “software” aspects of INIAF, for the purpose of achieving the objectives embedded in the PDO: (i) establishing INIAF as the leader of a dynamic, diversified national innovation system (SNIAF), and (ii) strengthening the capacity of INIAF to directly carry out critical research and extension activities. Investments to strengthen INIAF’s “hardware” included infrastructure upgrades for research stations, laboratories, and offices; acquisition of equipment; acquisition of vehicles; and purchases of consumable inputs used on research stations and in laboratories. Investments to strengthen INIAF’s “software” included the hiring of approximately 150 consultants, funding of training courses and other learning events, funding of knowledge exchanges, organization of technical meetings and professional conferences, sponsoring of innovation fairs and field days, and sponsoring DICs, among others. Despite these relevant design features, the project’s administrative and management instruments were not always well aligned with public norms governing the administration of goods and services. The Operational Manual by itself was not sufficient. 2 The instruments established to direct project resources to different uses were for the most part adequate, although several needed improvement. In some cases, the allocation of project resources was supply driven (e.g., direct allocation of funds to strengthen INIAF research programs), and in some cases it was demand-driven (e.g., use of competitive grants to finance innovation sub-projects proposed by beneficiaries). The use of competitive grants to fund innovation sub-projects was particularly noteworthy; experience in other countries had demonstrated the capacity of competitive grants to promote partnerships, encourage participation, and leverage additional resources from beneficiaries. 50. If the design was highly relevant in most respects for achieving the project objectives, it did have some shortcomings. Most notably, the institutional structure of INIAF pre-dated the project, so responsibility for implementing project-supported activities was assigned taking into account the institutional structure and division of responsibilities prevailing during preparation. Many project-supported activities aligned comfortably with INIAF’s established directorates and units, particularly in the areas of genetic resources conservation, research, extension and seed certification. But a few project-supported activities were new to INIAF, and they could not be linked easily to any existing directorate or unit. These included the activities relating to strengthening the SNIAF and establishing INIAF as its leader. This was a completely new role for INIAF, so when the project first got under way, there was little experience among INIAF staff and no institutional structure to support it. Initially, responsibility for these activities was assigned to the Technical Assistance Unit, which perhaps predictably made limited progress. Beginning in the fourth year, activities relating to strengthening the SNIAF and establishing INIAF as its leader were assigned to a semi-autonomous virtual unit, but by then much time had been lost. Another set of project-supported activities that did not fit easily into INIAF’s existing structure were those relating to institutional development. These were assigned to the INIAF Administrative Unit, but that unit 2 At the time of ICR finalization, INIAF was developing separate manuals for personnel administration, M&E, and procurement. 15 was so overwhelmed with day-to-day operational tasks that it was unable to devote much attention to the institutional development agenda. 51. Relevance of implementation: The relevance of Project Implementation is rated Modest. Implementation of project-supported activities was often delayed because INIAF and other SNIAF participants lacked capacity to execute the required policies and procedures. With the benefit of hindsight, it is clear that successful implementation of the project depended on the existence of a strong cadre of administrative and technical staff within INIAF, as well as on the availability of a deep pool of qualified technical experts throughout the country who could be mobilized at local level to accompany the innovation sub-projects. During implementation, it became evident that both of these critical resources were often in short supply, and without their contributions, achievement of the project objectives became challenging. The lack of capacity had particularly pronounced impacts on implementation in three areas: (i) procurement, (ii) monitoring and evaluation, and (iii) safeguards compliance. As discussed above, implementation arrangements proved deficient in two additional areas: (i) the INIAF Board of Directors never became fully empowered, and as a result it played a limited role during implementation; and (ii) the Project Coordinator position did not fit easily within the INIAF hierarchy, and as a result the Project Coordinator lacked authority to effectively manage project-supported activities. 52. The Bank team supervising the project responded proactively to the implementation delays resulting from capacity weaknesses, working with project management to prepare a series of action plans to address outstanding problems and providing extensive hands-on support. Despite these efforts, stronger Bank responses would have been helpful in several areas: (i) procurement support (e.g., providing even more training to project fiduciary staff); (ii) monitoring and evaluation (e.g., pushing INIAF to accelerate the roll-out of the MIS and the restructuring of the Results Framework); and (iii) social safeguards oversight (e.g., insisting more forcefully on filling the vacant safeguards position and establishing protocols for monitoring subprojects). 3.2. Achievement of Project Development Objective Rating: Substantial 53. The Project Development Objective was to strengthen INIAF and the SNIAF in order to contribute to productivity growth, food security, sustainable rural development, and the income- earning potential of Recipients’ families dependent on agriculture and forestry. The first two elements of the PDO are the most important (i.e., strengthening INIAF and strengthening the SNIAF). As noted earlier, the project was essentially an institution-building operation, and the PAD makes clear that there was no expectation that the higher-level objectives contained in the second part of the PDO would be measurable within the lifetime of the project. Efficacy thus is assessed in terms of the level of achievement of the two main elements of the PDO, based on the level of achievement of the PDO and Intermediate Outcome Indicators, supplemented by information contained in the independent final evaluation3 and information in the Borrower Completion Report (BCR). 4 (See also the Data Sheet for information on the full set of indicators, Annex 2 for a discussion of project outputs and outcomes, and Annexes 5 and 7 for summaries of the main findings of the final evaluation and the Borrower Completion Report, respectively.) It should be noted that there is significant overlap between the indicators measuring the strength of INIAF and the indicators measuring the strength of the SNIAF, as the strength of the SNIAF depends to a large 3 Sociedad Accidental CPDI-Servicios de Emprendimientos Globales. Evaluación final externa a la ejecución y resultados del PISA. December 2016. 4 Instituto Nacional de Innovación Agropecuaria y Forestal. Informe de Cierre: Proyecto de Innovación y Servicios Agropecuarios. Bolivia. January 2017. 16 extent on the emergence of INIAF as its leader. Summarizing across the entire Results Framework, all three PDO Indicators and 23 of 25 Intermediate Outcome Indicators were substantially achieved or exceeded. 54. The first element of the PDO (strengthening INIAF) is rated Substantial, based on the following considerations:  PDO Indicator 2: INIAF-led agricultural innovations adopted by producers . This indicator was measured based on the following parameters:5 (i) the technology must constitute an innovation, i.e., it must be new to the project beneficiaries; (ii) the technology must have been promoted by INIAF or by one of INIAF’s partners under the project; and (iii) field- based evidence must be available showing adoption of the innovation by project beneficiaries on their own initiative and using their own resources. Candidate adoption events were identified through project records and then verified in the field by an independent evaluator. Of the 48 candidate adoption events initially identified, 19 were determined by the independent evaluator to have met all three criteria. Five of the 19 innovations were reported as being at an initial stage of adoption, and the remaining 14 innovations showed a more advanced level of adoption in that producers had acquired the innovation on their own initiative and implemented it at scale using their own resources. During the final supervision mission carried out after the independent evaluator’s report had been delivered, INIAF provided evidence of adoption of an additional innovation, so the end-of-project target of 20 innovations adopted was achieved. A detailed review and final confirmation of successful adoption events was conducted jointly with the cofinanciers’ Technical Assistance Committee. INIAF’s success in reaching the target of 20 INIAF-led agricultural innovations adopted by producers provides clear evidence of strengthened capacity. The fact that adoption was taking place means several things had happened: (i) INIAF staff had correctly diagnosed the needs of producers; (ii) INIAF staff had generated or accessed innovations capable of addressing the needs that had been identified; (iii) INIAF staff had validated the effectiveness of the innovations in the field and determined that they are economically attractive; and (iv) INIAF staff had devised an effective strategy for delivering the innovation to producers.  Many of the Intermediate Outcome Indicators provide additional evidence showing that the first element of the PDO was achieved. The Intermediate Outcome Indicators, almost all of which substantially achieved or exceeded their end-of-project targets, show that at the time of project closing INIAF: (i) was actively involved in developing innovation policies and designing innovation regulatory frameworks; (ii) was setting strategic priorities for innovation; (iii) was conducting strategic and applied research; (iv) was formulating technical assistance strategies; (v) was delivering innovations and advisory services to the satisfaction of male and female producers; (vi) was certifying large quantities of seed to the satisfaction of clients; and (vii) was characterizing and conserving genetic resources, among other activities.  Additional evidence that INIAF’s capacity was strengthened as a result of the project was cited by many of those who participated in the Results Workshop held in La Paz in January 2017. The workshop participants, who included a diverse set of partners, stakeholders and clients, were asked if they had perceived any changes in the goods and services offered by 5 The method used to measure this indicator was clarified at the time of the 2015 Restructuring: (i) “agricultural innovations” was defined to include: new plant varieties, new agricultural and forestry practices; and other innovations; and (ii) “adoption” was defined to mean adoption in process, to be measured via annual surveys of selected R&E organizations at national and subnational levels. 17 INIAF and/or in INIAF’s operating modalities that could be attributed to the project. The workshop participants identified a number of goods and services now being offered by INIAF that were not being offered prior to the project, such as seed of improved varieties, improved agricultural machinery and implements, on-line genetic resources information, fee-based laboratory analyses, and customized technical assistance. In addition, they noted many positive changes in INIAF’s operating modalities, including the adoption of demand- driven priority setting, more systematic use of participatory research methods, enhanced collaboration with international research centers, greater use of information and communications technologies to disseminate extension messages, increased proactivity in organizing and sponsoring innovation fairs and similar learning events, and the sponsoring of a National Innovation Award competition. (See the Workshop Proceedings volume for additional details.) These changes in organizational behavior, while not captured in the performance indicators, provide additional qualitative evidence that the first element of the PDO was achieved. 55. The second element of the PDO (strengthening the SNIAF) is rated Substantial, based on the following considerations:  PDO Indicator 1: Participation of Research and Extension (R&E) organizations in SNIAF meetings at national and subnational level. This indicator was measured based on the participation in meetings convened by the Departmental Innovation Councils (DICs) and the Regional Innovation Councils (RICs), which were established through the efforts of INIAF and operated with INIAF funding. 6 The activities of the DICs and RICs were underwritten by INIAF as a way of promoting strategic planning, collaboration, and knowledge exchange among innovation agents at departmental and regional level. Building on the successes achieved at sub-national levels, beginning in 2013 national meetings known as INNOVARE events were held that attracted the same organizations as well as many others. In the final year of the project, just over 71 percent of the 136 R&E organizations registered as members of a DIC and/or RIC participated in one or more of the INIAF-sponsored events, exceeding the end-of-project target of 70 percent.  PDO Indicator 3: Agricultural organizations that see INIAF as an effective leader of the agricultural innovation system. This indicator was measured through a survey of 183 R&E organizations that were members of the SNIAF. Of these, 74 percent reported that they considered INIAF to be an effective leader of the SNIAF, nearly double the end-of-Project target of 40 percent. The fact that such a high percentage of the SNIAF membership recognized INIAF as an effective leader shows that by the time the Project closed, the SNIAF had emerged as a functioning innovation network and INIAF had established itself as its leader. The SNIAF succeeded in putting itself “on the map” by hosting the DICs and RICs and promoting innovation partnerships at the sub-national, national and international levels. The competitive grants instrument managed by INIAF played a key role in supporting these innovation partnerships, as many of the R&E organizations used it to access matching funds to pursue innovations in areas that had been identified as priorities by the DICs and the RICs.  The measured values of PDO Indicators 1 and 3 show that the second element of the PDO was substantially achieved by the time the project closed. It should be noted, however, that 6 The method used to measure this indicator was partly clarified at the time of the 2015 Restructuring. The population of R&E organizations to be sampled was clearly defined, but some questions remained about how to measure “participation in SNIAF meetings at national and subnational level.” 18 establishment of the SNIAF took longer than anticipated. During the first three years of implementation, INIAF management focused on strengthening INIAF’s internal capacities, and relatively little effort went into building the structures of the SNIAF and providing funding to support their activities. And when the SNIAF finally did get going beginning in Year 4, the level of activity was lower than anticipated. At project closing, only about 30 percent of the grant funding earmarked for subprojects had been transferred to partner R&E organizations. The low rate of transfers was attributable in part to regulations governing the transfer of public funds to non-governmental entities, which unexpectedly slowed initial disbursements to many external partners, but it also reflected the reluctance of INIAF senior management to channel project funds to third-party recipients when some areas within INIAF were pinched for funding.  While the SNIAF may have taken longer than expected to become established, all signs suggest that it will endure, as evidenced by the following: (i) the latest institutional organigram submitted by INIAF for ministerial approval includes a new operational unit tasked with leading the SNIAF; (ii) innovation councils have continued to operate at both regional and national levels, and INIAF is working with them to complete the remaining departmental innovation plans; and (iii) the next INNOVARE national innovation forum is planned for October 2017. Finally, in July 2017 when this ICR was being finalized, INIAF was preparing a proposal for a national innovation plan, which it intends to submit to the National Council for discussion and eventual endorsement. These activities attest to the commitment of INIAF and of the Ministry of Rural Development and Lands more generally to continue to support the SNIAF and build on the momentum that has been achieved under the project. 3.3. Efficiency Rating: Substantial 56. Estimating the efficiency of the project was challenging, for two reasons. First, as discussed previously, the project was essentially an institution-building operation, and the two main expected results described in the first part of the PDO (strengthened capacity in INIAF and strengthened SNIAF) do not lend themselves easily to quantification or valuation. Second, the higher-level objectives described in the second part of the PDO (contributions to productivity growth, food security, sustainable rural development, and incomes) take many years to materialize, so they were not measurable in any meaningful way at project closing. 57. In view of the above challenges, efficiency was assessed based on two types of benefits: (a) Value of increased production resulting from the adoption of innovations attributable to the project; and (b) Value of increased production attributable to the use of certified seed produced by INIAF using project resources. 58. The ex-ante economic and financial analysis carried out at appraisal focused on the first of these two benefits; it did not consider the second. The omission is puzzling, because an important objective of the project was to boost the service delivery capacity of INIAF, and one of the most prominent services provided by INIAF is seed certification. INIAF’s seed certification service is highly valued by the private sector, and it generates important revenues for the institution because seed producers pay to have their seed certified and conditioned at INIAF facilities. Importantly given the objectives of the project, seed certification activities are linked to research and innovation in two ways. First, they allow INIAF to introduce new varieties into the market and stimulate large- 19 scale adoption. Second, revenue generated through user fees is used to cross-subsidize other activities carried out by INIAF (e.g., conservation of genetic resources in the gene banks). (a) Benefits associated with adoption of INIAF-generated innovations 59. Methodology: The value of increased production resulting from the adoption of project- generated innovations was estimated based on analysis of 14 innovations generated by INIAF and/or partners using project funding. These 14 innovations included all those for which adoption had started by the final year of the project, making possible assessment of benefits actually being realized in farmers’ fields. The 14 innovations are considered representative of the 48 innovations generated under the project that were considered likely to be adopted (refer to the discussion in Section 3.2). The innovations are quite diverse; they include improved varieties of maize, wheat, rice, onion, potatoes, carrot, beetroot, and flowers; improved methods for producing maize- sorghum silage; and improved methods for producing seed. The assessments, which were carried out by independent consultants, involved field visits during which key parameters were discussed with focus groups that included producers with direct knowledge of the innovations, researchers, extension agents, and other informants. Conventional benefit-cost analysis methods were used to calculate standard measures of project worth for each innovation, including net present value (NPV), financial internal rate of return (FIRR), and economic internal rate of return (EIRR). 60. Results: On average, producers who have adopted project-generated innovations have benefited from significant increases in net revenue from their agricultural enterprises. Adoption has resulted in higher yields, better prices due to improved quality, and/or reduced production costs. EIRRs estimated for the 14 innovations range from about 18 percent associated with adoption of an improved carrot variety to more than 65 percent for the adoption of improved methods for producing seed of potato and the adoption of improved varieties of fava beans and peas. (See Annex 3 for detailed discussion of the results.) (b) Benefits associated with INIAF seed certification services 61. Methodology: The project promoted increased use of certified seed in a wide range of crops by strengthening the capacity of INIAF seed laboratories, upgrading seed processing facilities, introducing new standards for seed quality, strengthening the regulatory framework for seed marketing, and publicizing the benefits of certified seed. During the life of the project, the INIAF seed certification program significantly scaled up its activities and expanded coverage from what had previously been a concentrated focus on soybean to include a range of other crops. Increased use of certified seed attributable to the project led to increased yields and reduced production costs among users of that seed, which increased the net income derived from their cropping enterprises. The benefits attributable to project-supported seed certification services were estimated by calculating: (i) the incremental area planted to certified seed as a result of project investments in enhancing the capacities of INIAF’s seed program; and (ii) the yield gains associated with use of certified seed compared to non-certified seed, including farm-saved seed. The incremental area planted to certified seed was calculated by comparing the amount of seed that was certified during the life of the project to the amount of seed that was certified prior to the project. The yield gains associated with using certified seed were estimated based on measurements taken in farmers’ fields. The benefits were calculated based on the above-mentioned incremental area and yield gain estimates using farm budgets that compared with and without project scenarios. 62. Results: By the final year of the project, INIAF’s seed certification services were benefiting approximately 4,500 seed producers and 300 producer organizations annually, representing about 14,000 families in all. Based on the estimates of (i) the incremental area planted to certified seed and (ii) the yield gains associated with use of certified seed, it is estimated that producers realized Bs 11.7 million in incremental benefits over the life of the project. It is estimated 20 also that the resources invested in seed certification activities generated an EIRR of 54.4 percent. (See Annex 3 for detailed discussion of the results.) (c) Overall project benefits 63. Combined results: Returns to the overall project investment were estimated by summing across both types of benefits. Since the benefits associated with adoption of INIAF-generated innovations were estimated based on only 14 innovations, it was necessary to scale up the results to include the entire set of innovations generated under the project (including innovations that have not yet been adopted to any significant degree). The 14 innovations for which adoption has started represent approximately one-quarter of the 48 innovations that were identified as candidates for assessment, so the costs and benefits associated with the 14 innovations were multiplied by a factor of four. Implicit in this approach are two key assumptions: (i) the total number of innovations generated under the project will come to at least 56, and (ii) the costs and benefits associated with innovations other than the 14 innovations that were assessed will be similar to the costs and benefits associated with the 14 innovations that were assessed. While these assumptions are considered reasonable, sensitivity analysis was carried out to test the robustness of the results (see discussion below). To ensure that the efficiency analysis covered the entire project investment of US$ 55.1 million, the costs of all other project-supported activities were also included, even though no additional benefits were attributed to those activities because for the most part they were associated with strengthening of the SNIAF, the benefits of which are difficult to value. Using this extremely conservative approach, the NPV for the overall project investment is estimated at Bs. 109 million (about US$15.8 million), and the EIRR is estimated at 18.8 percent. These returns are consistent with returns documented for other similar projects, both within the LAC region and globally. (See Annex 3 for detailed discussion of the results.) 64. Sensitivity analysis: Sensitivity analysis was carried out to test the robustness of the efficiency analysis results. Three scenarios were explored: (i) a “low-adoption scenario” under which 50 percent of the innovations generated under the project are not adopted; (ii) an “unfavorable price scenario” under which the output prices received by users of project-generated innovations and incremental users of certified seed fall by 20 percent; and (iii) a “double shock scenario” combining low adoption of innovations and reduced prices for outputs. Under the “low adoption scenario,” the EIRR would fall to 13 percent, still an acceptable rate of return on public investment. Under the “unfavorable price scenario,” the EIRR would fall to 17.2 percent. Under the “double shock scenario,” the EIRR would fall to 11.8 percent. The results of the sensitivity analysis suggest that the returns to the overall project investment are quite robust. 3.4. Justification of Overall Outcome Rating Rating: Moderately Satisfactory 65. The Overall Outcome Rating of Moderately Satisfactory is based on the ratings of Modest for relevance, Substantial for efficacy, and Substantial for efficiency, justified as follows:  Relevance: The rating of Modest balances: (i) the diminished relevance of the project objectives at closing due to the evolution of the government’s priorities and vision for INIAF and the SNIAF; (ii) the strong and ongoing alignment of the project design with the PDO, which despite minor shortcomings merits a rating of Substantial; and (iii) the Modest rating for implementation relevance based on persistent capacity issues in INIAF and minor shortcomings in the Bank’s responsiveness on procurement, M&E, and social safeguards.  Efficacy: The rating of Substantial balances: (i) substantial achievement by INIAF in leading the promotion and adoption by producers of impactful innovations backed by continuous efforts by INIAF to develop innovation policies and regulatory frameworks, set 21 strategic priorities, conduct research and formulate/execute TA strategies; and (ii) clear evidence that INIAF came to be seen as an effective leader of the SNIAF and that the SNIAF itself was a stronger, better-organized universe of institutions and stakeholders, including at decentralized levels. As discussed above, all three PDO Indicators and 23 of 25 Intermediate Outcome Indicators were substantially achieved or exceeded.  Efficiency: The rating of Substantial is based on the positive assessment of benefits associated with the increased value of production resulting from producers’ adoption of project-generated innovations, as well as the value-added/incremental benefits generated by INIAF’s seed certification activities. 3.5. Overarching Themes, Other Outcomes and Impacts 66. The conclusions presented in the following sub-sections draw on the final evaluation (2016), views expressed by participants in the Stakeholders’ Workshop (2016), the Borrower Completion Report (2017), and monitoring data from the project M&E system. (a) Poverty Impacts, Gender Aspects, and Social Development 67. Poverty impacts: While there is no direct evidence of poverty reduction attributable to the project (which in any case is expected to occur over the longer-term), project records, the independent final evaluation carried out in late 2016, and the efficiency analysis carried out for this ICR all present evidence that adoption of innovations attributable to the project and incremental use of certified seed attributable to the project have led to increased productivity and production, lower production costs, and improvements in net income for more than 30,000 beneficiaries. These outcomes can be expected to contribute to reducing poverty among the project’s beneficiaries, many of whom depend heavily on agriculture for their livelihoods. (See also Annex 3 for evidence of productivity enhancement and poverty reduction potential.) 68. Gender: Promoting gender inclusion was an implied objective of the project and was intended to be pursued in two areas: (i) internally within INIAF; and (ii) externally in project- supported activities, for example, in events taking place within the SNIAF, in innovation subprojects, and around seed certification activities. Gender inclusion was to be measured based on women’s participation, but an explicit strategy for promoting women’s participation was not spelled out in the PAD. In both areas, the results were disappointing.  Internally within INIAF: The PAD states that a goal of the project was to increase the participation of women in INIAF, but insufficient efforts were made during implementation to achieve that goal. At project closing, barely one-quarter of all INIAF employees were women, with a heavy concentration in low- and mid-level administrative positions, as opposed to technical positions. The number of women in management positions was extremely small, ranging between 5 and 10 percent during the life of the project. INIAF management did not introduce any policies to encourage the incorporation of more women into the technical and scientific work force. Such policies are needed, as it is apparent that many women face particular barriers within the workplace, such as the inability to relocate on short notice due to family obligations and/or cultural pressures.  Externally in project-supported activities: Gender inclusion was monitored in external project-supported activities through a series of indicators that disaggregated participation by gender.7 Participants in the external activities consistently included fewer women than 7 The Client stated in preliminary comments on the Bank’s draft ICR, that the participation of women was monitored by the project but that project design did not focus on women and no strategy was applied to foster their participation. 22 men. For example, in the DICs and RICs promoted under the SNPA, only 14 percent of the participants were women. Participation by women in the innovation subprojects was somewhat more encouraging: project records show that about 36 percent of all subproject beneficiaries were women (8,276 individuals). 69. Indigenous people: Promoting inclusion by indigenous people was an explicit objective of the project, and the Results Framework contains several indicators designed to measure the participation of indigenous people as participants or beneficiaries. As with the case of gender, the results achieved were generally disappointing. Just how disappointing is difficult to say, however, because the project documentation (PAD and Results Framework) did not clearly define who should be considered an indigenous person. According to the final evaluation, only 8.4 percent of participants in innovation sub-projects self-identified as indigenous, but this figure seems improbably low in a country in which the majority of the population has indigenous ancestry and grew up speaking a local language other than Spanish. During implementation, it became evident that the term “indigenous person” lacked precision, but due to the cultural sensitivities surrounding matters of ethnicity, it was decided not to pursue the matter. (b) Institutional Change/Strengthening 75. As reflected in the PDO, the project was first and foremost designed to strengthen the capacity of INIAF and of the SNIAF, and in that context it is not surprising that it achieved a number of significant institutional changes. The most noteworthy included the following:  INIAF’s strategic research programs were considerably strengthened through upgrading of infrastructure, hiring of additional staff, and funding of expanded operating budgets. Infrastructure upgrades were particularly important in modernizing INIAF’s experimental stations (which were upgraded to innovation centers) and its germplasm banks. At the same time, procedures for planning, implementing, monitoring, and evaluating research were reformed, resulting in many operational efficiencies. An especially important change was the increased participation by INIAF scientists in collaborative research projects with a broader set of partners, not only domestic organizations but also international. Project funding helped support collaboration with international entities such as the International Maize and Wheat Improvement Center (CIMMYT), the International Potato Center (CIP), the International Center for Tropical Agriculture (CIAT) and the International Rice Research Institute (IRRI).  INIAF’s technical assistance programs were strengthened through upgrading of infrastructure, hiring of additional staff, and funding of expanded operating budgets. The technical assistance subprojects financed using the competitive grants mechanism increased demand for INIAF technical assistance services, which forced INIAF to re- examine and eventually overhaul its advisory services delivery system by putting in place new protocols for identifying, validating and delivering extension messages. The project was especially important in allowing INIAF to increase its presence at the departmental level, which allowed INIAF staff to interact more closely with target groups.8  INIAF’s administrative and fiduciary systems were strengthened through upgrading of infrastructure, hiring of additional staff and funding of expanded operating budgets. Many internal policies and procedures used by INIAF are different today that they were prior to the project, reflecting changes that were made during implementation in the areas of 8 INIAF’s research and technical assistance programs unquestionably were strengthened, but a persistent problem was the lack of coordination and productive working relationships between the Research Directorate and the Technical Assistance Directorate. 23 strategic planning, budgeting, and results monitoring. Two legacies of the project are the SISPOA and SISPO-AE management information systems. By project closing, the Planning Unit was one of the most capable units in INIAF.  INIAF’s Seed Program was strengthened through upgrading of infrastructure, hiring of additional staff, and funding of expanded operating budgets. Seed testing laboratories and seed conditioning plants benefited from major upgrades financed with project resources. Procedures for planning, implementing, and monitoring seed certification processes were completely overhauled and to a large extent automated, allowing the INIAF Seed Directorate to achieve major operating efficiencies even as it significantly scaled up the volume of seed certification activities.  Arguably the single most important institutional change attributable to the project was the creation of the SNIAF. As a result of INIAF’s efforts, considerable progress was made in formalizing regular interaction through the Departmental Innovation Councils (DICs) and the Regional Innovation Councils (RICs) of a wide range of actors, including local authorities, producer organizations, agribusiness firms, advisory services providers, and others. The DICs and RICs continue to function, albeit at a reduced level of activity, and they continue to provide venues for decentralized priority setting, planning, decision making, and knowledge exchange. 70. In addition to contributing to institutional changes, the project helped strengthen capacity within INIAF and throughout the SNIAF by supporting training events, learning activities, and knowledge exchanges in a number of areas:  INIAF researchers benefited from formal and informal training activities supported under the project, including technical workshops, seminars, short training courses and learning exchanges. The project also supported international travel that allowed INIAF scientists to exchange with and benefit from counterparts in other countries throughout Central and South America.  INIAF administrative and fiduciary staff benefited from technical training provided under the project and became familiar with World Bank procurement and financial management policies and procedures. While many of those who received training under INIAF have moved on to other jobs, they have taken their improved knowledge and skills with them and are applying them in other contexts.  Producer organizations and community groups that were able to access funding through the competitive grants mechanism benefited from the technical assistance and capacity building services provided under subprojects. Many subprojects included formal training activities designed to validate at farm level or scale up successful innovations.  Technical experts and project preparation specialists who assisted with the preparation of proposals submitted to the competitive grants facility benefited from the project by receiving training in proposal preparation and by participation in the proposal preparation cycle. During the early years of implementation, groups wishing to submit proposals often had difficulty mobilizing qualified experts; by the time the project closed, an entire cadre of specialists had emerged skilled in the preparation of proposals.  Participants in the activities sponsored under the SNIAF, especially in the DICs and RICs, became adept at organizing and implementing collective action. These participants included not only producer organizations and technical assistance organizations, but also local authorities and community leaders who often played an active role in the identification of local needs, priority setting, and resource mobilization. 24 (c) Other Unintended Outcomes and Impacts (positive or negative) N/A 3.6. Summary of Findings of Beneficiary Survey and/or Stakeholder Workshop 71. Beneficiary surveys: An overall beneficiary survey was not carried out covering the entire project, in part because the project generated different types of benefits for different target groups, and it would have been difficult to reach all of them through a single survey. Several smaller, more focused beneficiary surveys were carried out as part of the M&E plan, for the purpose of measuring some performance indicators. These smaller beneficiary surveys included: (i) the survey of R&E organizations participating in the SNIAF to determine the share that viewed INIAF as a dynamic leader (PDO Indicator 3); (ii) the survey of participants in technical assistance subprojects to measure the level of satisfaction with INIAF technical assistance services (Intermediate Outcome Indicators 7 and 8); and (iii) the survey of certified seed producers to determine the level of satisfaction with INIAF seed certification services (Intermediate Outcome Indicators 11 and 12). 72. Stakeholder workshop: A national stakeholder workshop was held in November 2016, shortly before project closing, with the goal of capturing perceptions concerning project successes, identifying bottlenecks encountered during implementation, and drawing out lessons learned. Approximately 170 people participated in the workshop, including representatives from producer organizations, service providers, NGOs, local governments and, private firms, among others. 73. Arguably the most important benefit identified by participants was the strengthening of innovation networks and the promotion of public-private and academic alliances. Other benefits identified by participants included: the technical assistance provided for strengthening capacity among producer organizations and community groups; the collective actions undertaken by producer associations and community groups in validating and scaling up innovations; the realization of the importance of producing for the market; and the potential for seed certification activities to bring economic benefits to seed producers.9 74. The participants also identified some shortcomings that negatively impacted project implementation. A recurring theme was the long delays experienced in the evaluation and approval of subproject proposals, which slowed the disbursement of funds to external partners and led to chronic implementation delays. Similarly, many participants expressed the view that the process for submitting subproject proposals was administratively complex and excessively bureaucratic, requiring applicants to comply with many requirements they viewed as unnecessary, as well as delaying subproject implementation and disbursements. Finally, participants flagged the low technical quality of INIAF staff in several regions and the high rate of staff turnover. 75. The outcomes of the stakeholder workshop are discussed in greater detail in Annex 6. 9 Instituto Nacional de Innovación Agropecuaria y Forestal. Informe de Cierre: Proyecto de Innovación y Servicios Agropecuarios. Bolivia. January 2017. 25 4. Assessment of Risk to Development Outcome Rating: Moderate 76. Sustainability of outcomes: Important legacies of the project include: (i) strengthened capacity and skills of the organizations and individuals that are participating in the SNIAF; (ii) strengthened capacity of INIAF to carry out research, provide technical assistance, carry out seed certification activities, and serve as the leader of the SNIAF; and (iii) strengthened capacity of producers and agribusiness firms that have benefited through their participation in project- supported activities. While these three things are expected to contribute to the sustainability of project outcomes, the full impacts of the institutional changes achieved under the project will take time to materialize. Although the project may have succeeded in substantively meeting its immediate objectives, for the expected longer term impacts to be realized, the activities initiated under the project will have to be continued in the years to come. In the months leading up to project closing, INIAF’s mid and long term financing remained uncertain. Faced with budget cuts, INIAF management had to scale back many programs and reduce the overall work force. After project closing, the Ministry of Rural Development and Lands undertook measures that have significantly improved the financing prospects for INIAF in the short to medium term (see discussion in Section 2.5). While funding prospects over the longer term remain less certain, sustainability of the development outcomes now seems assured. 5. Assessment of Bank and Borrower Performance 5.1. Bank Performance (a) Quality at Entry Rating: Moderately Satisfactory 77. Based on its analysis of previous experience promoting innovation systems in Bolivia and also in other countries in the region, the Bank/Borrower preparation team decided on a set of development objectives that were highly relevant for Bolivia and came up with a design that was appropriate given the local institutional context. The project management arrangements, fiduciary procedures, monitoring and evaluation systems, and safeguards enforcement measures were based on standard practices that had proven effective in countless Bank-funded projects worldwide, and the policies and procedures described in the Operational Manual prepared as a condition of effectiveness specified the implementation procedures in considerable detail. A separate manual was prepared for the competitive grants program describing the eligibility criteria for subprojects, identification and screening procedures for proposals, implementation support mechanisms, and evaluation procedures. 78. If the objectives of the project were relevant, the design appropriate, and the implementation arrangements reasonable, once implementation began, a number of problems arose that with the benefit of hindsight could have been addressed better during preparation and perhaps resolved by the time of effectiveness. Three of these problems are worth mentioning here. (a) Human resources management: At effectiveness and throughout the life of the project, INIAF experienced difficulties in recruiting and retaining capable staff. The Bank team that prepared the project recognized that staff management was potentially problematic, and it arranged for staff recruitment and evaluation to be entrusted to external firms, but this approach proved cumbersome and did not always produce satisfactory results. (b) Transfers of subproject funds: The procedure proposed for transferring subproject funds to external partners turned out not to be recognized under government fiscal management 26 regulations, and as a result transfers of funds for the first wave of subprojects were delayed. The problem was eventually resolved, but in the meantime valuable time was lost, and when the project closed the total amount of funds transferred was lower than originally expected. (c) Results Framework and M&E plan: The Results Framework had weaknesses related to the lack of relevance and the difficulty of measuring some indicators. These weaknesses were for the most part addressed at the time of the 2015 Restructuring, when adjustments were made to the Results Framework. In addition, at entry the project M&E plan lacked definition, in the sense that procedures for collecting M&E data had not been defined, and the associated instruments had not been developed. This delayed initiation of M&E activities, and some performance indicators were measured only shortly before closing. (b) Quality of Supervision Rating: Moderately Satisfactory 79. The project was supervised by a task team that included one or more Task Team Leaders supported by a full complement of technical, fiduciary, safeguards and M&E specialists. In addition to Bank staff, the team included several experts from the UN Food and Agriculture Organization (FAO), whose participation was supported through the World Bank-FAO Collaborative Program. The team also maintained a close and productive relationship with the two cofinanciers (COSUDE, Danida) and with the technical partner (GIZ), whose representatives participated in supervision missions, which permitted a rich interchange of ideas and provided a valuable source of technical support on specific issues. The cofinanciers’ Technical Assistance Committee provided a focal point for this collaboration. 80. Throughout the life of the project, the task team provided close implementation support, carrying out two to three supervision missions per year. In addition to formal supervision missions, Bank team members based in La Paz and Buenos Aires interacted with INIAF management on a near-continuous basis, with weekly face-to-face meetings and/or telephone calls. Aide Memoires from supervision missions document the intensive implementation support provided by the task team in all operational and fiduciary areas, the challenges faced in strengthening INIAF and the SNIAF, and the Bank’s pragmatic, problem-solving approach to supporting the client. 81. While the Bank’s supervision support was generally satisfactory, there was room for improvement. Three areas would have benefited from more proactive supervision: (a) Procurement: During the initial years of the project, the Bank team was sometimes slow in resolving procurement issues, and it had difficulty keeping up with the large number of No Objection requests generated under the procurement guidelines prevailing at the time. The problem was eventually resolved after the procurement guidelines were adjusted and the Bank Procurement Specialist provided supplementary training to INIAF fiduciary staff. (b) Monitoring and evaluation: The weaknesses in the Results Framework and the deficiencies in the project M&E plan have been discussed previously in this ICR. These problems were noted consistently in Aide Memoires and addressed during the Mid-Term Review, yet they were not resolved until shortly before closing. The Bank team invested a lot of time and effort trying to strengthen the M&E system, but these efforts were only partially successful. (c) Social safeguards compliance: INIAF had no previous experience working with World Bank safeguards, and it was slow to put in place systems for ensuring compliance with OP/BP 4.10 - Indigenous Peoples. The Bank team made note of the problem, but it was not proactive enough in finding a solution. The lack of proactivity was attributable in part to turnover within the Bank team (several people rotated through the Social Specialist position). 27 (c) Justification of Rating for Overall Bank Performance 82. Overall Bank performance is rated Moderately Satisfactory. The Bank team that prepared the project identified and prepared a project directly linked to government’s priorities and plans, and reflective of the Bank’s own analytics and strategy. The project design was logical, coherent, and straightforward, but even so challenges arose during implementation because some activities proved overly demanding for INIAF. The Bank’s supervision of the project was proactive and appropriately focused, but problems persisted in several areas—often due to the project’s excessive administrative and bureaucratic requirements, which INIAF as a young institution had difficulty following—despite the Bank’s proactive interventions to find solutions. 5.2. Borrower Performance (a) Government Performance Rating: Moderately Unsatisfactory 83. Government commitment to the Project varied throughout the course of implementation. Consistent with the Government’s stated objective of creating a modern, dynamic innovation system aligned with Bolivia’s long-term development vision, the Ministry of Rural Development and Lands was highly supportive during preparation. This support weakened noticeably during implementation, however, and even though the Executive Committee provided a powerful instrument through which the Government could have influenced the fortunes of the project, despite repeated appeals from the Bank team, during the first half of the project life little effort was made to slow the repeated turnover of leadership and to ensure that the project would be implemented as agreed by the Government and the Bank. From 2015 onwards, INIAF leadership became more stable, allowing the project to make significant progress. Finally, the budget allocated to INIAF during the post-project period, while significantly higher than anticipated at the time of the final supervision mission, falls short of what is required to maintain the momentum achieved and build on the progress already realized. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 84. INIAF was the implementation agency for the project. INIAF’s performance must be assessed taking into account that it was a very young institution tasked with re-inventing itself internally while working externally to establish and consolidate its leadership of the SNIAF. Despite the repeated changes in leadership that undermined institutional memory and led to delays while newly recruited staff became familiar with the project, INIAF performed reasonably well. The various INIAF management teams that were in place during the life of the project oversaw the successful implementation of a large and diverse work program, ensured that fiduciary functions were carried out in compliance with Bank and government policies and procedures, and maintained communications with SNIAF partners and stakeholders at national and departmental levels. Visits made during supervision missions to INIAF’s decentralized facilities and to subproject sites revealed that field staff were working hard to implement planned activities, despite facing many obstacles. At INIAF’s main offices in La Paz, considerable progress was made during the life of the project in putting in place streamlined administrative and fiduciary management systems, and by project closing financial management systems were functioning reasonably well, procurement staff were successfully managing complex bidding processes, and the monitoring and evaluation system was up and running. Despite the generally satisfactory performance, INIAF exhibited a number of weaknesses typical of project implementation units, some of which took longer than expected to resolve. These included the uneven performance on M&E during the initial years and the delays experienced in monitoring safeguards compliance. 28 (c) Justification of Rating for Overall Borrower Performance 85. Overall Borrower performance is rated Moderately Satisfactory. This rating reflects the Moderately Unsatisfactory rating for Government Performance and the Moderately Satisfactory rating for Implementing Agency, as moderated/influenced by the Moderately Satisfactory rating for Overall Outcome. 6. Lessons Learned 86. Changing the mental model of public authorities engaged in research and innovation is challenging. Public authorities who are used to deciding strategic research priorities and pursuing them in a top-down way using government implementing agencies may be reluctant to embrace a system in which innovation needs are identified through bottom-up, participatory processes and public funds are transferred to external partners so that solutions can be pursued in a decentralized way. Fearing a loss of power and control, they may resist implementing critical institutional changes. 87. By engaging in partnerships and strategic alliances, public agencies can leverage additional capacity and resources in pursuit of innovation and achieve much larger impacts. The subprojects financed under the project allowed INIAF to partner with a diverse set of actors including producer organizations, research institutes, universities, NGOs and private firms and produced results that INIAF could not have achieved working on its own. Partnerships also provide a mechanism that can be sued to sustain activities that may no longer be directly pursued by INIAF. 88. Selection of subprojects through an open, competitive process based on clearly defined technical evaluation criteria can be crucial for establishing credibility among stakeholders and avoiding political interference. Project beneficiaries—both producer organizations and service providers—have consistently indicated that the process used by INIAF ensured that subprojects were approved based on their merit, leaving few opportunities for financing to be steered by influential individuals to friends and associates. 89. Long-term vision backed up by sustained commitment of resources is needed to identify innovations, validate their effectiveness and promote adoption at scale. Unlike other types of donor-supported interventions, such as infrastructure projects, interventions designed to promote institutional change for the purpose of stimulating agricultural innovation do not deliver “quick wins.” On the contrary—the results take time to materialize. Maintaining a long-term vision and securing a sustained commitment of resources is challenging in situations where priorities change constantly and political leaders are under pressure to show results in the short run. 90. The success of institutions engaged in research and innovation depends critically on the quality and continuity of key staff. During the life of the project, the performance of INIAF suffered because at times positions were filled by staff who lacked the necessary qualifications, and because when capable staff were hired, often they did not stay long enough to have a noticeable impact. This points to the importance of having a good human resources management system, one that not only ensures recruitment of qualified staff but also ensures retention of staff who are performing well. Achievement of these objectives is easier when the institution has a certain degree of autonomy and can isolate itself and its staff from the volatility of political cycles. 91. Collaboration among multiple cofinanciers—in this case, World Bank, Danida, and COSUDE—can add value in several ways, some more obvious than others. Most obviously, collaboration among multiple cofinanciers can increase the total amount of donor funding made available to the Government. Not always appreciated is that financing from different cofinanciers may be provided under slightly different terms, which may provide the Recipient with some flexibility that otherwise would not exist. In addition, participation by multiple cofinanciers in 29 governance and supervision activities can broaden the range of perspectives being brought to bear on critical issues and lead to more informed and more balanced decision making. This was undoubtedly a key factor in project performance and results, and is critical for project sustainability. 92. Transition planning for World Bank-financed projects should begin early and not be left to the final months of implementation. Ideally, transition planning should begin at the time of the Mid-term Review and be monitored continuously thereafter. This will help to ensure that project-supported activities can be moved smoothly to government agencies following the closing of the World Bank loan, without suffering disruptions. 93. Investment Project Financing (IPF) may not be the best instrument for promoting institutional reforms. When IPF operations are used to support institutional reforms, Bank supervision teams cannot help getting involved in budget planning, personnel management, and other sensitive, easily politicized activities. When the project was prepared, IPF was the best available financing instrument, but in future teams preparing similar operations should consider whether alternative financing instruments that focus less on process and more on results might be more appropriate—such as Development Policy Financing (DPF) and Program for Results (PforR). 7. Comments on Issues Raised by Borrower/Implementing Agencies and Partners (a) Borrower/implementing agencies 94. The Bank shared a Spanish version of the ICR with the Ministry of Rural Development and Lands and with INIAF. At the time the ICR was submitted, no formal comments had been received. Should formal comments be received, they will be archived in the project files. 95. While the Spanish version of the ICR was being prepared, the ICR team met with representatives from MDRyT and from INIAF and shared the main findings of the ICR, which the representatives found to be reasonable. During the meeting, the Executive Director of INIAF provided an update on the progress that had been realized since the project closing date in terms of maintaining momentum on INIAF’s work program and fine tuning MDRyT’s strategy to ensure sustainability. In addition, INIAF management asked the ICR team to put on record the following two points: (a) Regarding the procurement case mentioned in paragraph 43, INIAF was frustrated by the level of information shared by the Bank. INIAF on several occasions requested that further details be provided regarding the case, but both INIAF and MDRyT deemed the information shared by the Bank insufficient to determine the extent to which mandatory procurement procedures had been followed. This prevented them from taking appropriate measures according to Bolivian law. The ICR team acknowledges INIAF’s assertion but upon consideration has chosen not to change the presentation in the ICR. (b) Regarding Intermediate Outcome Indicator 28 “INIAF management positions held by women and/or indigenous people (%),” INIAF believes the indicator does not reflect the level of diversity present in the institution. If all women working in INIAF are counted, including those engaged in research and those performing administrative tasks, they represent 29 percent of INIAF work force, which is higher than the level of representation in most Bolivian public institutions. The ICR team acknowledges INIAF’s point, but it reiterates that the purpose of the indicator was to document that more women were gaining access to management positions in which they have traditionally been underrepresented, and in this INIAF was less successful. 30 (b) Cofinanciers 96. COSUDE: The effort to contribute to the strengthening of the SNIAF in general and of INIAF in particular was a worthwhile exercise, aimed at consolidating the national innovation system and establishing the leadership role of INIAF. The participation of the development partners in the Technical Assistance Committee was very positive, as it facilitated decision-making by consensus and assured close supervision and strong implementation support during the different stages of the project. Bolivian legislation now attributes to INIAF various roles and responsibilities in research, technical assistance, and innovation, showing that the role of INIAF is formally recognized by the state. The level of financial support being provided to INIAF is very low, however, and far from the levels being provided to other similar institutes in the region. 97. Danida: During implementation, some confusion arose regarding the validity of the joint Memorandum of Understanding (MoU) that had been drawn up between INIAF, Danida, and the World Bank regarding the management of Danida funds. While Danida recognized the validity of a MoU signed by the Director of INIAF, the World Bank questioned the validity of the MoU, since it had not been signed by the Minister of Planning, the Government’s sole authorized representative for agreements with the World Bank. Regarding the 2014 procurement case involving Danida funds, the World Bank investigation took a substantial amount of time to complete, obliging Danida to suspend disbursements while awaiting the results. The investigation concluded that World Bank procurement rules had not been followed, but it has been difficult for Danida and INIAF to obtain documentation supporting this conclusion, which is needed to justify any request from Danida to INIAF to reimburse or reprogram Danida funds. (c) Other partners and stakeholders N/A 31 Annex 1. Project Costs and Financing (a) Project Cost by Component (US$ million equivalent) Appraisal MTR Actual Percentage of Project Components Estimate Estimate Cost Appraisal MTR (US$ million) Component 1: Strengthening the National Agricultural and Forestry Innovation System (SNIAF) (a) Developing policies, strategies and governance of SNIAF 1.18 0.72 0.68 58% 94% (b) Establishment of a Competitive Research Fund 4.92 2.52 2.48 50% 98% Subtotal Component 1 6.10 3.24 3.16 52% 98% Component 2: Strengthening core INIAF activities (a) Improving INIAF’s research programs 12.44 9.08 8.96 72% 99% (b) Improving INIAF’s technical assistance services 4.85 5.44 5.45 112% 100% (c) Consolidating the National Seeds System 5.26 6.08 5.9 112% 97% (d) Managing national genetic resources 3.63 2.64 1.91 53% 72% Subtotal Component 2 26.18 23.24 22.22 85% 96% Component 3: Supporting INIAF’s institutional development (a) Improving the organizational and internal capacities of INIAF 15.28 29.67 26.95 176% 91% (b)Developing external leadership and collaboration capacities 1.13 0.18 0.16 14% 89% Subtotal Component 3 16.41 29.85 27.11 165% 91% Component 4: Project management (a) Support to INIAF for implementing PISA 2.85 1.57 2.16 76% 138% (b) Developing an information and monitoring and evaluation 1.36 0.48 0.43 32% 90% system for INIAF Subtotal Component 4 4.21 2.05 2.59 62% 126% Total PROJECT COST 52.9 58.38 55.08 104% 94% Notes: Component 1: Actual expenditures fell short of appraisal estimates because (i) activities supporting the development of the SNIAF were late in getting started, and (ii) transfers of subproject resources to non-governmental entities were initially delayed pending clarification of government regulations. Components 2 and 3: Actual expenditures fell short of appraisal estimates for Component 2 and exceeded appraisal estimates for Component 3 because INIAF consultants engaged to perform activities supported under Component 2 were budgeted under Component 2 at appraisal, but the payments ended up being registered under Component 3. (b) Financing (US$ million equivalent) Appraisal Mid Term Actual/ Latest Percentage of Percentage of Source of Funds Estimate Review Estimate Appraisal MTR (US$ million) % Government of Bolivia 9.00 17.89 16.44 183% 92% IDA 39.00 34.34 32.50 83% 95% COSUDE 2.90 3.49 2.82 97% 81% Danida 2.00 2.66 3.31 166% 124% Total 52.90 58.38 55.07 104% 94% 32 (c) Project cost by Component (US$ million equivalent) Project Components 2011 2012 2013 2014 2015 2016 C1: Strengthening the National Agricultural and Forestry 0.00 0.11 0.22 0.76 0.81 1.23 Innovation System (SNIAF) C2: Strengthening core INIAF activities 0.62 0.95 4.71 6.48 6.30 3.25 C3: Supporting INIAF’s institutional development 0.13 2.91 4.89 5.99 6.88 6.22 C4: Project management 0.69 0.12 0.46 0.52 0.48 0.34 Total PROJECT COST 1.44 4.09 10.28 13.75 14.47 11.04 33 Annex 2. Outputs by Component 1. The Agricultural Innovation and Services Project comprised four components. This annex briefly summarizes the outputs achieved under the first three components (the fourth component was related to project management activities only). Table A2.1 lists the project’s achievements by component and subcomponent. Component 1: Strengthening the National Agricultural and Forestry Innovation System (SNIAF) (estimated total cost US$6.1 million, of which IBRD US$5.35 million) 2. The objective of this component was to support the organization of a national innovation system for agriculture and forestry (SNIAF) and promote the interaction and participation of its diverse constituents in agricultural research and innovation through competitively selected, collaborative subprojects. (a) Development of policies, strategies to improve governance of the SNIAF. INIAF established the Plurinational Council of Innovation (PCI) and strengthened six Departmental Innovation Councils (DICs), in which 211 public and private actors—including generators of knowledge, producers, and agricultural service providers—participate. INIAF also oversaw numerous other activities: approval of a National Law (No. 774), a Supreme Decree (No. 2454), and two Departmental Innovation Plans; formulation of a policy to establish a national innovation fund and regulate SNIAF’s operation; and the promotion of 4 national meetings (INNOVARE), 2 national competitions, 14 departmental competitions, 3 regional competitions, 3 enlarged meetings of the PCI with the participation of 9 DICs and 2 RICs, and 2 national forums on innovation in the sector. As a result of the project’s strategic communication efforts and funding competitions, 315 innovations emerged from SNIAF, of which 69 were systematized and disseminated. SNIAF also developed a web portal. (b) Establishment of a Competitive Research Fund. Through the creation of this fund, INIAF implemented 23 applied and adaptive research subprojects, based on priorities identified by the 9 CDIs. Of these subprojects, 58 percent have an agricultural orientation, 25 percent are oriented to livestock, 8 percent focus on aquaculture, and 8 percent support apiculture (beekeeping and honey production). Component 2: Strengthening Core INIAF Activities (estimated total cost US$26.17 million, of which IBRD US$22.39 million) 3. The objective of this component was to strengthen core INIAF programs (research, technical assistance, seed management, and genetic resources). (a) Support to INIAF research programs. INIAF substantially improved its research capacities, investing US$8.89 million in eight strategic programs (wheat, potato, quinoa, livestock, vegetables, and forestry, as well as rice and maize, which were introduced later). Wheat and potato research focused on genetic improvement, and livestock management research focused on prairie management. The forest program concentrated on implementing reproductive protocols, due to the long lead time required by innovation in this area. The partnership model supported under the project made it possible to marshal the knowledge and experience of different actors, including international peers and mentors, who accompanied the research programs either fully or partially as they implemented their improvements. The model also allowed INIAF to develop its own capabilities and, above all, to obtain impacts more quickly. In addition, support for INIAF under this component 34 strengthened the capacities of students and young professionals from national institutions, who participated in INIAF research programs through laboratory work and thesis research. (b) Support to INIAF technical assistance services. Technical assistance formed part of the project design with the expectation that it would allow INIAF to contribute to a system in which the Departmental Governments were key actors. Two technical assistance pilot subprojects were approved for financing in two departments. In the course of executing the project, however, INIAF assumed a stronger lead role in the system and assigned greater importance to direct technical assistance, eventually pursuing activities in 193 municipalities. INIAF also created subprojects throughout the country in alliance with numerous strategic public and private entities, with the exception of Departmental Governments. The key outputs were:  INIAF established a capacity development program with eight lines of action to date. The program reaches 21,206 beneficiaries, has consolidated a Network of Technical Advisors, and has systematized innovations.  Seven Departmental Technical Assistance Strategies were approved.  All activities under 13 subprojects were completed successfully. Users were highly satisfied with the technical assistance services that INIAF provided. (c) Consolidation of the National Seed System. This subcomponent was one of the most successful features of the project. The volume of seed certified by INIAF exceeded the annual target of 100,000 tons, and more than 90 percent of beneficiaries were highly satisfied with the seed. The volume of certified seed in western Bolivia has increased by 65.3 percent since 2011 and led to greater use of certified seed. This component made it possible to fully equip two seed laboratories and improve seven, as well as to refurbish and equip four seed conditioning plants. Seed certification, registration, control, and control of trade in seeds were improved through the implementation of 19 new regulations. Since 2012, certification has been extended from 210 to 284 varieties, including varieties of native crops, fruit, and medicinal and forest plants (such as tarwi, amaranth, citrus, peaches, apples, and moringa). The number of varieties registered in Bolivia since 1984 increased by 30 percent during the five years in which the project was active. The number of certified seedlings increased from 1 million to 10 million. (d) Management of genetic resources. INIAF created a National Germplasm Bank Network, comprising active banks and working banks, with the participation of 19 institutions. The government also signed two international treaties on genetic resources (The Nagoya Protocol). These efforts made it possible to meet technical objectives related to genetic resources, including genetic resource data conservation and management, germplasm characterization, and genetic improvement. Component 3: Supporting INIAF’s Institutional Development (estimated total cost US$16.38 million, of which IBRD US$7.13 million) 4. The objective of this component was to strengthen the institutional and organizational capacity of INIAF. Component 3 included two subcomponents focusing on internal organizational capacity and external leadership and collaboration capacity. (a) Improved the organizational and internal capacities of INIAF. The institutional development subcomponent partially met expectations, with INIAF fully achieving the goals of three out of six indicators. INIAF has consolidated a management model that respects national and departmental competencies and roles, although it will need to be revised in light of new ministerial guidelines for working at the macro-regional level. INIAF also introduced 35 regulations for human resource management and substantially improved financial management. Even so, a budget shortfall meant that INIAF had to close one of 16 fully operational offices. Some internal processes still take longer than the optimal time to complete, and the organization has struggled to maintain stable staffing, given the budgetary uncertainty following the closure of PISA. At the closing workshop with stakeholders, many noted their disappointment with delayed administrative processes at INIAF. INIAF, in planning for future institutional adjustments, needs to evaluate its internal management protocols in depth and propose ways of resolving its staffing and related difficulties. (b) Strengthening capacity for external leadership and collaboration. The first key achievement under this subcomponent is that INIAF successfully mobilized a broad network of institutions at the national and subnational level (the first outcome indicator). A second key achievement was that INIAF developed very fruitful partnerships with many national and international institutions, notably collaborative research programs (alliances, work with CIMMYT) and the subprojects supported through the research and technical assistance fund. It is very important for INIAF to continue to deepen its leadership capacities within SNIAF, as established in its Decree of Creation. Component 4: Project Management (US$4.21 million, of which IBRD US$4.12 million) 5. This component supported two subcomponents—one for INIAF to implement the project, and the other for developing an information, monitoring, and evaluation system for implementing results-based management. (a) Project implementation. The purpose of this subcomponent was to strengthen INIAF so that it could execute PISA, and with PISA support, INIAF met the operational, fiduciary, and safeguard management requirements for the project. Specialized staff were brought in to reinforce key technical positions in INIAF for implementing PISA, including the National Technical Directorates, the Administrative Unit, and the Management Unit. PISA met the objective of strengthening INIAF, but after the end of the project, personnel added through PISA did not join the INIAF structure. The budget allocated for 2017 is 15 percent of what it was during PISA. (b) Monitoring, and evaluation. PISA provided support to strengthen the INIAF Planning Unit and create the conditions for results-based management. Under PISA, manuals, procedures, and tools were developed to plan, monitor, and evaluate the various activities for which INIAF is responsible. Two electronic monitoring systems to facilitate the internal and external management of INIAF were implemented: SISPOA (for monitoring annual processes and plans) and SISPOA-AE (for monitoring research subprojects and technical assistance). Despite these advances, INIAF still must overcome several challenges to fully institutionalize and sustain the planning, monitoring, and evaluation system. It must strengthen the technical and operational capacities of the Planning Unit and advance the institutional arrangements to consolidate results-based management in the institution. It must also implement an integrated system of planning, monitoring, and evaluation that more fully and systematically links the different areas of INIAF, in addition to linking INIAF with its allies and SNIAF members. 36 Table A2.1: Project achievements by component and subcomponent Activity Achievement Comments Component 1: Strengthening the National Agricultural and Forestry Innovation System (SNIAF) Developing policies, strategies, Achieved  7 innovation policies or instruments were and governance of SNIAF developed and approved:  2 Supreme Decrees.  5 departmental agricultural and forestry innovation plans (developed by the Departmental Innovation Councils in Santa Cruz, Tarija, Chuquisaca, La Paz, and Cochabamba).  11 Departmental/Regional Innovation Councils were created: Cochabamba, Tarija, Santa Cruz, Potosí, Beni, Chuquisaca, La Paz, Oruro, Pando, CRI Chaco (Santa Cruz), CRI Chaco(Tarija).10 Establishing a Competitive Partially  INIAF established a Competitive Research Fund, Research Fund achieved concluding 23 of 24 planned subprojects. These 23 subprojects covered 61% (target: 100%) of the topics prioritized by CDIs. Component 2: Strengthening core INIAF activities Improving INIAF research Exceeded  18 innovations were reported using participatory programs methods developed by INIAF.11 Improving INIAF technical Exceeded  10 technical assistance strategies were prepared assistance services and approved in Oruro, Potosi, Chuquisaca, Tarija, Cochabamba, La Paz, Pando, Beni, Santa Cruz, and Chaco.  98.3% (target: 70%) of the producers are satisfied with the technical assistance provided by INIAF. Consolidating the National Seed Exceeded  119 tons of seed certified annually (target: 106 System tons/year), and 95% (target: 75%) of the clients were satisfied with INIAF’s services.  90.6% (target: 80%) of seed producers were satisfied with INIAF seed certification procedures. Managing national genetic Exceeded  Of the total 18,434 accessions registered in resources conservation, the morphological and agronomic traits of 66.25% (target: 20%) were characterized. Component 3: Supporting INIAF’s institutional development Improving the organizational Partially  5 (target: 5) audits were performed, complying and internal capacities of INIAF achieved with project technical requirements.  INIAF management information system is effective (100%). SISPOA and SISPOA AE are operational and enabled. 10 Reported by External Evaluator CPDI. 11 Final report by the Borrower. 37 Activity Achievement Comments  The use and functionality of some modules of both systems is not ensured after the project ends.  15 INIAF offices in operation at the national level.  31% (target: 50%) of internal users of INIAF (staff) are satisfied with INIAF's administrative support services.  Of the 26 managerial positions, only 1, which represents 3.26% (target: 20%), is occupied by a female self-identifying as an indigenous woman. Note that candidates for a position are chosen based on capacity and not necessarily based on their status as a female or indigenous person. Developing external leadership Achieved  37 (target: 16) agreements have been signed with and collaboration capacities (surpassed different institutions at the national level. target) Component 4: Project management Support to INIAF for Achieved  INIAF established and maintained a Coordination implementing PISA Unit for activities under the project, which had a close coordination relationship with the Organizational Development and Planning Unit of INIAF. Developing an information and Partially  INIAF has SISPOA for physical indicators, with monitoring and evaluation achieved an interface for financial monitoring. system for INIAF  Similarly, SISPOA-AE was created for physical and financial monitoring of the execution of subprojects with strategic allies. These systems were partially institutionalized at closing. 6. In addition, the project supported 56 subprojects, including: (i) 23 subprojects executed with Research Funds; (ii) 7 research subprojects under Component 2; and (iii) 26 subprojects in the areas of Technical Assistance and Seed Promotion. These subprojects, along with INIAF-executed strategic research programs, generated at least 20 innovations for which adoption had already started by the project closing date (for details Table A2.2.). Each innovation is documented by INIAF in its own technical file describing the innovation. 38 Table A2.2: Innovations adopted as a result of subprojects funded through PISA N° Crop Innovation 1 Wheat Introduction of wheat variety Totora 2 Wheat Introduction of wheat variety Yesera (biofortified) 3 Maize Introduction of maize hybrid H1 4 Maize Introduction of maize hybrid HQ2 5 Maize Introduction of maize variety Choclero 6 Horticulture Introduction of onion variety Navideña 7 Horticulture Introduction of beetroot variety Caperucita 8 Horticulture Introduction of onion variety Valencianita 9 Horticulture Introduction of onion variety Globosa 10 Horticulture Introduction of carrot variety Altiplano 11 Flowers Seed multiplication system for Lisianthus flowers 12 Flowers Seed multiplication system for Chrysanthemum flowers 13 Forage Silage systems for maize and sorghum 14 Quality seed Seed supply system (SLAS) in Cobija 15 Quality seed Seed supply system (SLAS) in 17 Mayo 16 Quality seed Potato seed supply system (SLAS) 17 Rice Introduction of rice variety Tayta VF 18 Rice Introduction of rice variety Chasqui 19 Quinoa Biofertilizers used for organic production of quinoa 20 Wheat Direct seeding and crop rotation in wheat production 39 Annex 3. Economic and Financial Analysis A3.1 Introduction: Assessment approach 1. The PAD makes no estimate of the project's economic internal rate of return (EIRR), but it mentions that evidence from a large number of studies worldwide shows that returns to investment in agricultural R&D are substantial. Many studies have documented the important role played by public investments in agricultural R&D and extension services in boosting growth in agricultural productivity and production. Comprehensive literature reviews suggest that the economic internal rate of return (EIRR) to investments in agricultural R&D usually exceeds 20 percent, and in two- fifths of the studies EIRR exceeds 60 percent. 2. Because investments in agricultural R&D take time to produce impacts, it is still early to attempt to measure the benefits that will be generated by the project over the longer term. A number of short-term outputs and outcomes can already be estimated, however. This Annex estimates benefits associated with two types of project-supported activities that are already generating positive outcomes, namely the R&D activities being supported under Components 1 and 2, and the seed certification services being supported under Component 2. These benefits include: (a) The value of increased production resulting from the adoption of innovations attributable to the project; and (b) The value of increased production attributable to the use of certified seed produced by INIAF using project resources. 3. The ex-ante economic and financial analysis carried out at appraisal focused on the first of these two benefits; it did not consider the second. The omission is puzzling, because an important objective of the project was to boost the capacity of INIAF to deliver services, and the most prominent service provided by INIAF is seed certification. INIAF’s seed certification service is highly valued by the private sector, and it generates important revenues for the institution because seed producers pay to have their seed certified and conditioned at INIAF facilities. INIAF’s seed certification activities are linked to research and innovation in two ways. First, they allow INIAF to introduce new varieties into the market and stimulate large-scale adoption. Second, revenue generated through user fees are used to cross-subsidize other activities carried out by INIAF (e.g., conservation and renewal of genetic resources through the gene banks) A3.2 Economic benefits from the adoption of innovations 4. Under Component 1, the project supported innovation through the Research Fund subprojects, whose objective was to meet innovation needs defined at subnational level. The Research Fund financed 23 subprojects for a total amount of US$2.48 million. These 23 subprojects financed research and extension activities focused on eco-regional priorities including climate change mitigation challenges, in many cases associated with the crops being targeted by INIAF’s own strategic research programs. Most of the innovations developed through the 23 subprojects have not yet been disseminated widely or adopted at a large scale. 5. Under Component 2, the project supported the strengthening of core INIAF activities. More than US$ 20 million was invested in: (i) setting priorities for INIAF research programs; (ii) developing linkages with providers of technical assistance (TA); (iii) expanding seed certification capacity; and (iv) improving genetic resources. Some of these investments are already generating quantifiable benefits. Under Component 2, US$ 9 million was invested in improving INIAF’s nine core research programs, which focused on maize, wheat, sugar cane, rice, potato, quinoa, dual purpose cattle, forestry, and horticulture. In addition, US$ 5.4 million was spent for 40 improvement of TA systems implemented by subnational authorities and NGOs participating in the SNIAF. Related to the TA activities, the production and dissemination of certified seeds and planting materials stands out as particularly noteworthy, as many of the farmers who are using these materials have realized significant improvement in yields and net incomes 6. For this analysis, a series of innovations with demonstrated adoption in the field was selected from Components 1 and 2, including innovations developed by partners and innovations developed by INIAF’s own researchers. The benefits attributable to these innovations were assessed with the help of independent consultants. The innovations whose benefits were estimated through the case studies included new varieties of maize, wheat, rice, onion, potatoes, carrots, beets, and flowers. In addition, the analysis included the value of certified seeds. Innovation case studies 7. When the project closed, adoption was under way for 20 innovations, some of which had been produced by partners, and some of which had been produced by INIAF’s own researchers. With all of these innovations, adoption had started, but it was still increasing, meaning that the peak adoption level had not yet been reached. 8. The economic and financial analysis covered 18 out of the 20 innovations for which adoption had started and the scale of adoption was assumed large enough for the analysis (Table A3.1). These innovations were quite diverse; they included: improved varieties of maize, wheat, rice, onion, potatoes, carrots, beetroot and flowers; improved methods for producing maize- sorghum silage; and improved seed production methods. In the end, after collecting data from the field, the team ruled out four additional innovations for which the degree of adoption in the field turned out to be too small. The economic and financial analysis thus ended up including 14 innovations. 41 Table A3.1. Innovations included in the economic and financial analysis N° Crop Innovation Data source 1 Wheat Wheat variety Totora INIAF 2** Wheat Wheat variety Yesera (biofortified) Independent consultant 3 Maize Maize hybrid H1 Independent consultant 4 Maize Maize hybrid HQ2 Independent consultant 5 Maize Maize variety Choclero Independent consultant 6 Horticulture Onion variety Navideña Independent consultant 7 Horticulture Beetroot variety Caperucita Independent consultant 8 Horticulture Onion variety Valencianita Independent consultant 9 Horticulture Onion variety Globosa Independent consultant 10 Horticulture Carrot variety Altiplano Independent consultant 11** Flowers Seed multiplication system for Lisianthus Independent consultant 12** Flowers Seed multiplication system for Chrysanthemum Independent consultant 13 Forage Silage systems for maize and sorghum Independent consultant 14 Quality Seeds Seed supply system (SLAS) in Cobija Independent consultant 15 Quality Seeds Seed supply system (SLAS) in 17 mayo Independent consultant 16 Quality Seeds Potato seed supply system (SLAS) INIAF 17** Rice Rice variety Tayta VF INIAF 18 Rice Rice variety Chasqui INIAF 19* Quinoa Biofertilizers for organic production of quinoa Not evaluated 20* Wheat Direct seeding and crop rotation in wheat Not evaluated * These two innovations were not considered because the level of adoption in terms of cultivated area was considered too small, based on the information provided by INIAF. ** These innovations were evaluated, but they were not included in the final calculation because the area on which the innovation had been adopted turned out to be too small and was not considered appropriate for a meaningful estimate. Data collection and methodology 9. With the support of external consultants, INIAF assessed the impact in the field of 14 innovation cases that were considered at a sufficiently advanced stage of adoption to be included in the assessment. For each of the 14 innovations, field activities were visited and analyzed by independent consultants, who quantified financial benefits. In every case, at least two focus groups were organized: one made up of farmers who had been exposed to the innovation and received TA, and the second made up of key informants. The first focus group provided farm-level information about production costs and results with and without the innovation. The so-called “CRP” technique12 was used to obtain information on costs, yields, and prices, all expressed in terms of a probability range (low – medium – high). The data were used to build annual cash flow budgets using the FARMOD software. The second focus group provided qualitative information about the innovations, adoption trends, and area covered / number of farmers. The information obtained from the second group was used to estimate current and future adoption paths and assess likely adoption ceilings (the latter defined in terms of area covered and number of farmers). 10. Based on the consultants’ observations, average crop budgets were developed and used to compare costs and revenues with and without adoption of the innovations. As shown in Table A3.2, 12 CRP refers to “Costos, Rendimientos y Precios” (Costs, Yield and Selling Prices) at the farm level. 42 it was determined that on average, adopters of the innovations realize significant increases in net revenue from cropping. These increases in net revenue result from higher crop yields, better prices obtained due to quality improvements, and/or reduced costs (or a combination of these factors). Innovation case study details (a) Improved maize cultivars (H1, HQ2, Choclero) 11. Three innovations consisted of improved maize cultivars: the INIAF hybrids H1 and HQ2 and the INIAF variety Choclero. These cultivars were developed by INIAF in collaboration with CIMMYT and introduced successfully through three subprojects implemented with partner organizations in the localities of Carapari, Palmito-Timboy, and Villa Montesin near Tarija. The three subprojects generated an EIRR of 60.4 percent and an NPV of Bs. 9.4 million (Table A3.3). Table A3.3. Benefits from adoption of improved maize varieties Without With project project Years Year 1 Year 2 Years 3 to 20 1 to 20 Main Production Traditional Maize 752,520 752,520 - - Maize Choclero - - 414,000 414,000 Maize CS INIAF H1 - - 1,656,000 1,656,000 Maize CS INIAF HQ2 - - 1,035,000 1,035,000 Sub-total Main Production 752,520 752,520 3,105,000 3,105,000 Production Cost Investment Project Investments - 2,400,000 1,200,000 - Operating Purchased Inputs 170,140 170,140 679,620 679,620 Labor 314,400 314,400 255,040 255,040 Sub-total Operating Costs 484,540 484,540 934,660 934,660 OUTFLOWS 484,540 2,884,540 2,134,660 934,660 Cash Flow 267,980 -2,132,020 970,340 2,170,340 Main Production (in MT) Traditional Maize 193. 193. - - Maize Choclero - - 18 18 Maize CS INIAF H1 - - 96 96 Maize CS INIAF HQ2 - - 30 30 EIRR = 60.4%, NPV = Bs. 9,411,545 Note: An average of US$171,400 was invested in each of the three subprojects. Around 20 families benefited in each of the three locations. (b) Improved wheat variety (Totora) 12. The wheat variety INIAF Totora, released in 2014, offers higher grain yields with increased protein and gluten content, making it attractive to the wheat milling industry in the valleys area. Totora shows tolerance to the main wheat diseases found in the area and matures in 135 days. The 43 subproject to develop Totora generated an EIRR of 20.1 percent and an NPV of Bs. 712 thousand (Table A3.4). Table A3.4. Benefits from adoption of improved wheat variety Without With Project project Years Years Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 8 to 20 1 to 20 Main Production Agricultural Products 4,569.348 4,569.348 4,569.348 4,569.348 4,626.258 4,796.022 5,090.022 5,678.022 6,854.022 Production Cost Investment Research Fund - 500.000 340.000 - - - - - - Operating Purchased inputs Inputs 969.821 969.821 969.821 969.821 991.013 1,030.113 1,108.313 1,264.713 1,577.513 Machinery and animal 373.008 373.008 373.008 373.008 390.352 422.352 486.352 614.352 870.352 traction Sub-total 1,342.829 1,342.829 1,342.829 1,342.829 1,381.365 1,452.465 1,594.665 1,879.065 2,447.865 Labor Labor 2,486.720 2,486.720 2,486.720 2,486.720 2,504.064 2,560.736 2,656.736 2,848.736 3,232.736 Sub-total Operating 3,829.549 3,829.549 3,829.549 3,829.549 3,885.429 4,013.201 4,251.401 4,727.801 5,680.601 Costs Sub-total Production 3,829.549 4,329.549 4,169.549 3,829.549 3,885.429 4,013.201 4,251.401 4,727.801 5,680.601 Costs OUTFLOWS 3,829.549 4,329.549 4,169.549 3,829.549 3,885.429 4,013.201 4,251.401 4,727.801 5,680.601 Cash Flow 739.799 239.799 399.799 739.799 740.829 782.821 838.621 950.221 1,173.421 Wheat Production (MT) 2,176 2,176 2,176 2,176 2,203 2,284 2,424 2,704 3,264 EIRR = 20.1%, NPV = Bs. 712,810 Notes: An average of US$ 120,400 was invested in this subproject. Around 30 families are expected to benefit. Area Planted: Years 1-5 (i.e., up though 2017) based on certified seed actually produced. Years 6-20 - it was assumed that 200 ha would be planted in 2018, 400 ha in 2019, and 800 ha thereafter. (c) Improved rice variety (Chasqui) 13. The rice variety INIAF Chasqui was developed from 2012 to 2016 in Beni and Santa Cruz from local materials. The variety is tolerant to major diseases, has a vigorous growth habit (which allows it to compete with weeds), and has a high yield potential of about 5 MT/ha in rainfed conditions and about 8 MT/ha under irrigation. INIAF Chasqui has proved so popular that demand for certified seed has outstripped INIAF’s ability to produce the seed. The adoption rates projected for the case study take into account the bottleneck in seed production. 13 The subproject to develop Chasqui generated an EIRR of 37.1 percent and an NPV of Bs. 2,757 (Table A3.5). 13 Multiplying basic seed into certified seed is time consuming, especially when only small amounts of basic seed are available initially. In this case study, it was estimated that five years will be needed to multiply 100 kg of basic seed into 5,000 MT of certified seed, the quantity required to plant 50,000 ha. 44 Table A3.5. Benefits from adoption of improved rice varieties Without With Project project Years Years Year 1 Year 2 Year 3 Year 4 Year 5 1 to 20 6 to 20 Main Production Agricultural Products 7,279 7,279 7,309 7,377 7,570 7,960 8,640 Production Cost Investment - 500 340 210 - - - Operating 3,640 3,640 3,652 3,678 3,753 3,904 4,168 Labor 192 192 192 192 192 192 192 Sub-total Operating Costs 3,832 3,832 3,844 3,870 3,945 4,096 4,360 OUTFLOWS 3,832 4,332 4,184 4,080 3,945 4,096 4,360 Cash Flow 3,447 2,947 3,125 3,297 3,625 3,863 4,280 Rice Production (MT) 2,022 2,022 2,030 2,049 2,102 2,211 2,400 EIRR = 37.1%, NPV = Bs. 2,757 thousands Note: An average of US$150,000 was invested in this subprojects. Around 60 families (300 ha) in the Tropics are using this new material, and area is expected to double in two years. 14. If the demand continues to grow as expected, and by 2020 about 10 percent of the rice area in Bolivia (15,000 ha) will be planted to this variety, in which case the EIRR of the resources invested its development would exceed 70 percent. (d) Improved beetroot variety (Caperucita) 15. The beetroot variety INIAF Caperucita costs the same to produce as traditional varieties but offers higher and more stable yields across multiple cropping seasons. It is also more resistant to damage than traditional varieties, resulting in a better quality product that fetches better prices in the market. Production of certified seed of Caperucita increased from 135 kg in 2012 to about 2,787 kg in 2016, which considering also the widespread use of farmer-saved seed of the same variety suggests that almost 50% of the national beetroot area is already under the new variety. The subproject to develop Caperucita generated an IRR of 35.8 percent and an NPV of Bs. 1.8 million (Table A3.6). 45 Table A3.6. Benefits from adoption of improved beetroot variety Without With project project Years Year Year Year Year Year Year Years 1 to 20 1 2 3 4 5 6 7 to 20 Main production 11,830 11,830 11,981 12,064 12,064 12,154 12,237 12,321 Agricultural Products Production Cost - 500 340 210 - - - - Investment 5,312 5,312 5,301 5,295 5,295 5,288 5,282 5,276 Operating 3,689 3,689 3,689 3,689 3,689 3,689 3,689 3,689 Labor 9,001 9,501 9,330 9,194 8,984 8,977 8,971 8,965 Sub-total Operating Costs 9,001 9,501 9,330 9,194 8,984 8,977 8,971 8,965 OUTFLOWS 2,830 2,330 2,651 2,870 3,080 3,176 3,266 3,355 Cash Flow 8,744 8,744 8,855 8,917 8,917 8,983 9,044 9,106 Beetroot Production EIRR = 35.8%, NPV = Bs. 1,782 thousands (e) Improved carrot variety (Altiplano) 16. The carrot variety INIAF Altiplano has been adopted by at least 38 farmers in the area around Sorocachi, Oruro. The variety is a substitute for the traditional Peruana variety, which is widely grown. The Altiplano variety shows similar yields to the traditional alternative, but the product commands prices that are 20 percent higher on average than the prices paid for local carrots because of its better appearance. The subproject to develop Altiplano generated an EIRR of 17.9 percent and an NPV of Bs. 276,103 (Table A3.7). Table A3.7. Benefits from adoption of improved carrot variety Without With project project Years Years Year 1 Year 2 1 to 20 3 to 20 Main Production Agricultural Products 957,600 972,000 1,010,400 1,140,000 Production Cost Investment - 500,000 340,000 - Operating 185,630 187,130 191,130 204,630 Labor 185,440 185,440 185,440 185,440 Sub-total Operating Costs 371,070 372,570 376,570 390,070 Sub-Total Production Cost 371,070 872,570 716,570 390,070 Cash Flow 586,530 99,430 293,830 749,930 EIRR = 17.9%, NPV = Bs. 276,103 Note: An average of US$120,400 was invested in this subproject. Around 38 families are benefiting, with about 0.5 ha planted per family. 46 (f) Improved onion varieties (Globosa, Navideña, Valencianita) 17. The onion variety INIAF Globosa, released in 2010, has proved popular in the altiplano zone due to its high yields and good quality bulbs. Crop losses due to early season cold are minimal, as premature flowering is reduced from about 15 percent to 2 percent. Production of certified seed increased from 45 kg in 2010 to 312 kg in 2016, and around 80 ha have already been planted with this new INIAF material. Demand has exceeded the availability of certified seed, and further area increases are expected in the near future as more seed becomes available. 18. Two other new white onion varieties, INIAF Navidena and INIAF Valencianita, are being promoted under the project in Sipe Sipe and Capinota localities in the Cochabamba valleys region. Compared to traditional varieties, these two varieties are characterized by lower production costs and higher yields. Seed multiplication for both varieties is still at an early stage. In both cases, there is a strong acceptance and growing demand for the seed, the goal being to substitute for white onion imports from Peru. The three subprojects to develop these three varieties generated a combined EIRR of 27.1 percent and a combined NPV of Bs. 2.5 million (Table A3.8). Table A3.8. Benefits from adoption of improved onion varieties Without Project With Project Years Year Year Year Year Year Year Years 1 to 20 1 2 3 4 5 6 7 to 20 Main Production Agricultural Products 7,202 7,202 7,235 7,502 7,727 7,916 8,078 8,106 Production Cost Investment - 500 600 1,020 630 400 - - Operating 2,545 2,545 2,547 2,549 2,548 2,549 2,542 2,526 Labor 2,600 2,600 2,600 2,600 2,595 2,590 2,574 2,537 Sub-Total Production Cost 5,144 5,644 5,747 6,168 5,773 5,539 5,116 5,064 OUTFLOWS 5,144 5,644 5,747 6,168 5,773 5,539 5,116 5,064 Cash Flow 2,058 1,558 1,488 1,334 1,953 2,377 2,962 3,043 Main Production (in MT) Onion (Globosa) - - - 1,725 3,105 4,140 5,175 5,175 Onion (Valencianita) - - - - 86 173 460 1,121 Onion (Navidena) - - 248 386 552 828 828 828 Onion (Perilla) 6,425 6,425 6,193 4,554 3,105 1,855 661 - EIRR = 27.1%, NPV = Bs. 2,461 thousands Note: An average of US$150,000 was invested in each of these three subprojects. Around 350 families are benefiting with the new material. (g) Quality seed production: Potatoes, fava beans and peas 19. Among the TA activities supported under the project, promotion of quality seed production among small communities was shown to be one of the most effective ways to promote adoption of innovations and increase the income of small farmers. The INIAF Subprogram Asistencia Tecnica Semillera - ATS (Seed Technical Assistance) worked with farmer groups and helped them to produce quality certified seed, for their own use or for sale to other farmers. The ATS Subprogram promoted certified seed production methods in potatoes, fava beans, peas, rice, wheat, maize and quinoa among other crops, targeting poor communities throughout the country. For purposes of the 47 ICR, the assessment focused on efforts to support quality seed production in potato, fava beans, and peas, with the costs and benefits aggregated and assessed jointly (Table A3.10). The three subprojects to promote quality seed production in these crops generated an EIRR of 65.3 percent and an NPV of Bs. 10.5 million (Table A3.9). Table A3.9. Benefits from adoption of quality seed production methods (potato, peas, fava beans) Without project With project Years Year Year Year Year Year Years 1 to 20 1 2 3 4 5 6 to 20 Main Production Green Peas and Fava Beans Seed 5,222 5,222 5,222 5,036 4,703 4,568 4,598 Potato Seed - - - 1,250 3,500 5,500 5,500 Sub-total Main Production 5,222 5,222 5,222 6,286 8,203 10,068 10,098 Investment IASP Investments - 500 600 1,020 630 400 - Operating Inputs 1,759 1,759 1,759 2,170 2,914 3,594 3,601 Machinery & Animal Traction 518 518 518 564 647 720 720 Labor 1,212 1,212 1,212 1,260 1,351 1,433 1,441 Sub-total Operating Costs 3,488 3,488 3,488 3,994 4,911 5,748 5,763 Sub-Total Production Cost 3,488 3,988 4,088 5,014 5,541 6,148 5,763 Cash Flow 1,734 1,234 1,134 1,272 2,661 3,921 4,336 EIRR = 65.3%, NPV = Bs. 10,629 thousands Note: An average of US$150,000 was invested in each of these three subprojects. Around 275 families were assisted with the production of seeds for potatoes, green peas and fava beans instead of the commercial produce. (h) Improved silage for feeding cattle 20. Livestock and dual-purpose cows are central components for the production systems prevailing in Cuevo, Boyuibe and Camiri municipalities of the Cordillera region in Santa Cruz and in the extensive drought-prone El Chaco ecosystems. Three TA subprojects were implemented in the three above-mentioned Santa Cruz municipalities, with the goal of introducing supplemental feeding with maize and sorghum silage during the dry season. According to the data collected in these localities, the introduction of maize and sorghum silage improves milk production significantly (as much as 40 to 50 percent). The three subprojects to promote supplemental feeding generated an EIRR of 16.0 percent and an NPV of Bs. 685 thousand (Table A3.10). As this new practice is refined and adjusted to fit the region, it is expected that these early results will be improved significantly. 48 Table A3.10. Benefits from adoption of methods for improved silage production Without project With project Years Year Year Year 1 to 20 1 2 3 to 20 Main Production Silage - - 315 315 Agricultural Products 1,890 1,890 2,678 2,678 Sub-total Main Production 1,890 1,890 2,993 2,993 On-Farm Use Silage - - 315 315 Net Value of Production 1,890 1,890 2,678 2,678 Production Cost Investment - 2,100 1,100 - Operating - 118 207 207 Labor 840 862 879 879 Sub-total Operating Costs 840 980 1,086 1,086 OUTFLOWS 840 3,080 2,186 1,086 Cash Flow 1,050 -1,190 491 1,592 Milk Production ('000 lts) 630 630 892.5 892.5 EIRR = 16.0%, NPV = Bs. 685 thousands A3.3 Economic benefits from certified seed production and use Incremental production from increased use of certified seed 21. The INIAF National Seeds Directorate has a long and illustrious history. Its activities were strengthened by the project through an investment of US$5.9 million to consolidate the National Seeds Program. The focus was on increasing the production and use of quality certified seed by improving the capacity and quality of INIAF seed laboratories and seed processing service plants, as well as by strengthening the regulatory framework and standards governing seed marketing. With project support, the INIAF National Seeds Directorate also expanded its coverage to include a broader range of crops—specifically, commodities grown in the Andean uplands and valley zones, where INIAF’s seed program previously had little or no penetration. As a result of project support, certified seed production in Bolivia was improved: prior to the project (2001-10), certified seed production was increasing at an average of 3,541 kg per year; with project support (2011-16), the seed production increases averaged 5,321 kg per year (Figure A3.1). 49 Figure A3.1: Trends in production of certified seed, pre- and post-project 22. From the incremental certified seed production figures mentioned above, it can be concluded that the current annual growth of certified seed production and use in Bolivia is partially due to the pre-existing growth tendency (3,541 kg/year out of the 5,321 kg/year, about two thirds of the increase), but also to the added certification capability enabled by the project (1.780 kg/year, about one third of the current growth rate). This enhanced support capacity of INIAF permitted increased annual incremental areas using certified seed, which jumped from about 41,000 ha to 61,500 ha from 2011 to 2016. Even if this increased production capacity induced by the project were to slow down again in the future, during the five- year implementation period, the project promoted 102,500 additional ha every year from 2016 onwards to use certified seeds as a result of the project support. 14 23. Numerous empirical studies conducted worldwide have demonstrated the impact of the use of certified seed and adoption of other agricultural technologies on crop yields and poverty reduction. The increased production and use of certified seed brought about by the project permitted increased yields and reduced production costs, raising the net income derived from cropping and contributing to poverty alleviation. For quantifying these benefits, crop budgets for the main crops were estimated both with and without the use of certified seed, based on the current results being obtained by farmers. Table A3.11 shows the main parameters used for the analysis. 14 Without the project and following the previous tendency, in 2016 it was expected to have a production of 93,370 MT of certified seeds per year. With the project, the certification capacity has now been increased to produce around 107,780 MT. So, today, Bolivia is producing about 14,410 MT more certified seed than the without project scenario, with which around 102,500 incremental ha per year are using certified seeds for commercial production. 50 Table A3.11. Benefits from adoption of certified seed, various crops Yield Gross income Production cost Net income Crop (kg/ha) (Bs/ha) (Bs/ha) (Bs/ha) Normal Certified Normal Certified Normal Certified Normal Certified seed seed seed seed seed seed seed seed Rice 2,500 2,880 4,500 5,185 4,161 4,297 339 888 Peas 1,500 2,100 6,000 8,400 4,370 6,340 1,630 2,015 Beans 700 810 4,200 4,860 3,729 3,917 689 943 Peanuts 1,530 1,760 12,240 14,080 9,680 11,280 6,128 7,180 Fava bean 1,650 2,100 5,775 7,350 3,375 4,710 1,484 1,725 Potatoes 5,900 8,600 14,180 20,640 10,210 15.990 3,950 4,650 Quinoa 540 650 4,752 5,720 4,480 5,207 272 513 Sunflower 1,670 1,920 2,906 3,341 1,230 1,397 1,676 1,944 Maize variety 2,200 2,600 4,303 5,086 4,024 4,570 279 516 Maize mech 2,500 2,900 4,890 5,672 3,433 3,844 1,457 1,828 Maize hybrid 4,200 4,800 8,215 9,389 4,336 4,920 3,879 4,469 Sorghum 1,670 1,920 2,923 3,360 1,790 2,022 1,133 1,338 Soybeans 1,723 1,900 4,077 4,495 3,158 3,429 1,723 1,900 Wheat 1,400 1,700 2,240 2,720 1,955 2,343 285 377 Wheat mech 1,600 1,900 2,560 3,040 2,497 2,767 63 273 Sesame 500 580 4,000 4,640 1,847 1,923 2,153 2,717 24. As can be seen in Table A3.11, average yields are increased by 15 to 46 percent while net income per cropped ha increases by 15 to 400 percent, depending on the crop. By adding up the 102,500 ha of additional crop area using the incremental certified seeds produced with project support, the aggregated net revenue obtained by farmers adds up to Bs137 million per year (about US$19.8 million). The following Table A3.12 summarizes the value added by the 102,500 ha producing with improved seed. Table A3.12. Benefits from improved capacity to produce certified seeds Without Project With Project Years Year Year Year Year Year Years 1 to 20 1 2 3 4 5 6 to 20 Main Production Agricultural Products from CS users 497,253 497,253 522,462 535,067 547,672 560,277 560,277 Main Production Support to INIAF Seed Direction - 11,000 11,000 11,000 11,000 11,000 - Operating inputs 374,558 374,558 390,318 398,198 406,078 413,958 413,958 Labor 8,825 8,825 9,043 9,152 9,261 9,370 9,370 Sub-total Operating Costs 383,383 383,383 399,361 407,350 415,339 423,328 423,328 OUTFLOWS 383,383 394,383 410,361 418,350 426,339 434,328 423,328 Cash Flow 113,870 102,870 112,101 116,717 121,333 125,949 136,949 Production CS users (in '000 MT) 221 221 232 238 243 249 249 EIRR = 56.0%, NPV = Bs. 91,585 thousands 51 Value of increased production of certified seed 25. Similar to the increased production resulting from new areas using certified seeds, the production of incremental quantities of certified seed per se (instead of commercial production) for supplying the growing demand by farmers for improved seed, required an additional 4,856 ha dedicated to the production of these improved seeds, which are sold at a value 2 to 4 times higher than the regular crop product. Crop budgets for certified seed production were also prepared (available in Project Files), and the added benefits captured by farmers were estimated at Bs11.7 million (net benefits increased from Bs4.7 million to Bs16.4 million) as shown in Table A3.13. Table A3.13. Incremental production of certified seeds due to subprojects Without With project project Years Year Year Year Year Year Years 1 to 20 1 2 3 4 5 6 to 20 Main Production 24,012 - 15,378 23,067 30,755 38,444 38,444 Support to INIAF Seed Directorate - 8,000 8,000 8,000 8,000 8,000 - Operating inputs 18,082 - 4,378 6,567 8,756 10,945 10,945 Labor 1,237 -0 4,437 6,655 8,873 11,091 11,091 Sub-total Operating Costs 19,319 - 8,815 13,222 17,629 22,036 22,036 OUTFLOWS 19,319 8,000 16,815 21,222 25,629 30,036 22,036 Cash Flow 4,693 -8,000 -1,437 1,845 5,126 8,408 16,408 EIRR = 54.4%, NPV = Bs. 64,464 thousands A3.4 Aggregated benefits 26. Returns to the overall project investment were estimated by summing across both types of benefits. Since the benefits associated with adoption of INIAF-generated innovations were estimated based on only 14 innovations, it was necessary to scale up the results to include the entire set of innovations generated under the project (including innovations that have not yet been adopted to any significant degree). The 14 innovations for which adoption has started represent approximately one-quarter of the 48 innovations that were identified as candidates for assessment, so the costs and benefits associated with the 14 innovations were multiplied by a factor of four. Implicit in this approach are two key assumptions: (i) the total number of innovations generated under the project will come to at least 56, and (ii) the costs and benefits associated with innovations other than the 14 innovations that were assessed will be similar to the costs and benefits associated with the 14 innovations that were assessed. While these assumptions are considered reasonable, sensitivity analysis was carried out to test the robustness of the results (see below). The estimated costs and benefits associated with adoption of INIAF-generated innovations were added to the costs and benefits associated with INIAF’s seed certification activities. To ensure that the efficiency analysis covered the entire project investment of US$ 55.1 million, the costs of all other project- supported activities were also included, even though no additional benefits were attributed to those activities because for the most part they were associated with strengthening of the SNIAF, the benefits of which are difficult to value. 27. Table A3.14 summarizes the estimated aggregated benefits of the project. Using this extremely conservative approach, the NPV for the overall project investment is estimated at Bs. 52 109 million (about US$15.8 million), and the EIRR is estimated at 18.8 percent. The benefit : cost ratio is estimated at 1.73. These returns are consistent with returns documented for other similar projects, both within the LAC region and globally. Table A3.14. Overall project benefits Without With project project Years Year Year Year Year Year Year Years 1 to 20 1 2 3 4 5 6 7 to 20 Main Production Agricultural Products 124.3 124.3 130.0 133.8 137.1 141.4 145.1 145.5 Agricultural Products CS Users 538.0 538.0 555.3 563.2 570.5 578.6 578.7 578.7 Certified Seeds - - 32.6 47.7 66.7 84.8 84.8 84.8 Sub-total Main Production 662.3 662.3 717.8 744.6 774.3 804.7 808.5 809.0 On-Farm Use (Silage) - - 1.3 1.3 1.3 1.3 1.3 1.3 Net Value of Production 662.3 662.3 716.5 743.4 773.0 803.4 807.3 807.7 Production Cost Investment Research Fund - 28.0 17.0 5.8 2.5 1.6 - - Technical Assistance Activities - 2.0 2.4 4.1 2.5 1.6 - - Support INIAF Seed Directorate - 19.0 19.0 19.0 19.0 19.0 - - SNIAF and INIAF Institutions - 43.5 43.5 43.5 43.5 43.5 - - Sub-total Investment Costs - 92.5 81.9 72.3 67.5 65.7 - - Operating Purchased Inputs Inputs 263.6 263.8 279.5 288.2 298.1 307.8 307.8 307.7 Machinery & Animal Traction 183.4 183.6 190.4 193.8 197.4 201.1 201.7 201.7 Sub-Total Purchased Inputs 449.4 449.9 472.5 484.5 498.2 511.9 512.9 512.8 Labor 46.4 46.5 51.0 53.5 56.2 58.9 58.8 58.7 Sub-total Operating Costs 495.8 496.4 523.5 538.1 554.4 570.7 571.7 571.5 OUTFLOWS 495.8 588.9 605.4 610.4 622.0 636.4 571.7 571.5 Cash Flow 166.4 73.4 111.1 133.0 151.0 167.0 235.5 236.2 EIRR = 18.8%, NPV = Bs 109 millions Sensitivity analysis 28. The assessment is considered conservative considering that in most of the innovation case studies, adoption is still in at an initial stage. As seen with the maize hybrids, the Caperucita beetroot and with the Globosa onion improved seeds where the process started five years ago in 2012 having more time for these materials to be adopted, the estimated EIRRs are significantly higher—often exceeding 50 percent—which is considerably above the overall 18.8 percent estimated for the project overall. 29. In spite of this conservative assessment, as a sensitivity analysis of the quantified partial results, more pessimistic assumptions were introduced to the analysis as follows: (i) about half of the analyzed subprojects fail to sustain the assumed adoption levels and are simple discarded in the near future; and (ii) all prices or agricultural products are reduced by 20 percent. In the first case, even if 50 percent of the analyzed subprojects were to end up failing to achieve the expected results, 53 the EIRR would still reach 13 percent. In the second case, if all prices of agricultural products were to fall by 20 percent, the EIRR would still be 17.2 percent. Finally, if both adverse situations were to occur at the same time, the EIRR would only fall to 11.8 percent. These results demonstrate that the project’s impact is highly robust even if the assumptions regarding expected results were to be highly pessimistic as in the adverse scenarios described. 54 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Francisco Obreque Agriculture Specialist GFA04 Team Leader Michael Morris Senior Agriculture Economist GFA04 Team Leader Armando Sanjines Senior Procurement Specialist GGO04 Lucas Carrer Financial Management Specialist GGO22 Patricia Gutierrez Team Assistant LCCBO Elena Segura Senior Counsel LEGLE Juan Carlos Enriquez Senior Environmental Specialist GEN04 Griselle Vega Senior Agriculture Specialist GFA04 Patricia de la Fuente Sr Financial Management Specialist OPSPF Maria Ruth Llanos Consultant GSU04 Safeguard specialist Patricia Velasco Temporary LCCBO Supervision/ICR Francisco Obreque Agriculture Specialist GFA04 Team Leader Michael Morris Senior Agriculture Economist GFA04 Team Leader Wilhelmus Janssen Senior Agriculture Economist GFA13 Team Leader Julio Sanjines Senior Procurement Specialist GGO04 Lucas Carrer Financial Management Specialist GGO22 Patricia Gutierrez Team Assistant LCCBO Juan Carlos Enriquez Senior Environmental Specialist GEN04 Griselle Vega Senior Agriculture Specialist GFA04 Maria Ruth Llanos Consultant GSU04 Safeguard specialist Patricia Velasco Temporary LCCBO Veronica Cronembold Consultant LCCBO Adam Behrendt Social Specialist José Rasmussen Senior Procurement Specialist GGO04 Marcelo Berthin Financial Management Specialist 55 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY08 1.25 19.88 FY09 20.58 98.47 FY10 44.75 184.46 FY11 56.18 171.30 FY12 5.80 21.90 Total 128.56 496.01 Supervision/ICR FY12 20.96 118.59 FY13 25.52 118.00 FY14 18.71 90.22 FY15 31.14 151.69 FY16 40.93 168.21 FY17 23.64 111.00 Total 160.90 757.71 56 Annex 5. Beneficiary Survey Results 1. The final evaluation of the project carried out by an independent consulting firm included formal and informal surveys of several beneficiary groups. The final evaluation is not discussed at length in this annex, because the key findings are reported in the Results Framework and discussed in the ICR Data Sheet and in Section 3.2 Achievement of Development Objectives. Final evaluation (a) Sociedad Accidental CPDI - Servicios de Emprendimientos Globales. 2016. Evaluación final externa a la ejecución y resultados del PISA, Producto 3. Documento informe final de consultoría. Diciembre. Assessed perceptions of INIAF’s performance as leader of the SNIAF (PDO Indicator 3). The assessment concluded that more than 70 percent of the external beneficiaries see INIAF as an effective leader of the SNIAF. 2. In addition to the final evaluation, three more narrowly focused beneficiary assessments were carried out as part of the project’s M&E function, for the purpose of measuring certain performance indicators. Two of these assessments focused on external beneficiaries, and one assessment focused on internal beneficiaries (INIAF staff). Again, the assessments are not discussed at length in this annex, because the key findings are reported in the Results Framework and discussed in the ICR Data Sheet and in Section 3.2 Achievement of Development Objectives. Assessments targeting external beneficiaries (b) NATBIO SRL. 2016. Evaluación de Indicadores del PIS, grado de satisfacción de agricultores y agricultoras con la asistencia técnica apoyada por el INIAF. Junio 3. Assessed the level of satisfaction of users of the technical assistance services provided by INIAF (Intermediate Output Indicators 7, 8, and 9). The assessment concluded that more than 98 percent of the beneficiaries were satisfied with the technical assistance services provided by INIAF. (c) Consultores Para el Desarrollo Integral (CPDI S.R.L.). 2016. Medición del Indicador de Satisfacción de los Productores de Semilla con el Servicio de Certificación de Semillas del INIAF. Mayo. Assessed the level of satisfaction of users of the seed certification services provided by INIAF (Intermediate Output Indicators 11, 12, and 13). The assessment concluded that more than 90 percent of seed producers were satisfied with the seed certification services provided by INIAF. Assessment targeting internal beneficiaries (d) INIAF. 2016. Informe técnico INF/INIAF/UPDO/No180/2016. Julio 28. Assessed the level of satisfaction of INIAF staff with internal administrative procedures (Intermediate Output Indicator 29). The assessment concluded that only 31 percent of INIAF staff were satisfied with the administrative support received within the institution. 57 Annex 6. Stakeholder Workshop Report and Results A6.1. Introduction 1. PISA stakeholders convened at the national closing workshop for the project held in La Paz in November 2016. The workshop was designed to present the main results of the project, capture stakeholders’ perceptions related to its implementation, identify the lessons learned, and derive recommendations to strengthen future activities. Among the 170 participants were representatives of producer organizations (generally the leaders), strategic partner institutions (generally the technicians supporting producer organizations), government agencies (chiefly the Ministry of Rural Development and Land and INIAF), and others. Cooperation agencies such as DANIDA, GIZ, and COSUDE participated as well, mainly as workshop facilitators. 2. Three working groups focused on assessing project achievements related to research, technical assistance, and seed, respectively, including contributions to productive technology, organizational development, access to markets, and income generation. Two other working groups focused on more general achievements of the competitive research fund and SNIAF and recommended ways to improve future interventions by INIAF and SNIAF. A6.2. Summary of main results 3. This annex summarizes the major conclusions developed by these working groups. For full details, see “Closing Workshop of PISA Agricultural Innovation and Services Project.” A6.2.1. Research  INIAF's alliances with producers and the creation of partnerships between public and private actors and academia were effective for promoting collaborative research. The development of networks, such as the wheat network, allowed the main organizations involved in the topic of interest to exchange knowledge and scientific information through research meetings.  A policy framework must accompany the dissemination and adoption of the innovations that are generated. INIAF must have the support of higher authorities and favorable public policies.  The work in the SNIAF venues and the partnerships developed by the public sector, private sector, and academia generate very rich spaces for innovation and the exchange of experiences. Alliances have positive aspects that must be combined, and negative aspects that must be managed, for greater effectiveness.  INIAF's research-oriented staff must possess the skills required to meet the challenges of innovation in the agricultural sector. It is essential to ensure the continuity of staff and actions to achieve the expected results. A6.2.2. Technical assistance  Technical assistance services provided under the project have increased productivity in the priority sectors, resulting in a direct improvement in production, and income for producers. Nevertheless, better market analysis is required, because increased production may not necessarily lead to higher incomes if market prices are low.  Producers strengthened their capacity for collective action to market their products; producer organizations improved their management capacity. The development of 58 alliances and linkages with other actors made it possible to obtain better results from technical assistance and to improve the dissemination of technologies and new knowledge.  Producer organization now require more intensive technical assistance to deepen their capabilities in collective action and business management. They also need assistance to improve the quality, timeliness, and market orientation of production, an develop a better grasp of the mechanisms of commercialization, such as collective branding and product certification. A6.2.3. Seed  The quality of the technical support and advice for seed production and verification is highly valued, especially considering the impact of improved seed on the productivity and incomes of producers.  The work of the technicians who provided assistance was arduous, given the poor logistical support for their work, the limited equipment available, and the instability of their tenure with INIAF. It is necessary to maintain and improve their capacity to provide technical assistance in seed production and certification so that producers can improve their knowledge.  Certified and quality seed must be promoted on a large scale to improve production. This activity must be accompanied by good technical advice for producers, not only from the government but from strategic allies.  The exchange of experiences among seed workers from different regions helps to improve knowledge and also highlights the need for regulations (such as seed legislation that takes regional characteristics into consideration), organization (such as establishing a national association of seed producers), and infrastructure (such as improved seed processing plants). A6.2.4. Competitive Research Fund  The research achievements attained through the alliances in priority areas (such as grapes, cocoa, cane, camelids, and strawberries) are evident. Research funding facilitated the transfer of technology to producers by stakeholders in the agricultural innovation system The development of negotiation capacities that was encouraged among the actors was beneficial for conflict resolution and strengthening strategic alliances.  Implementation of the research subprojects revealed the need for an electronic platform. Such a platform would enable researchers to spend less time on administrative tasks, because in one place it would assemble all monitoring information from implementation and make it easier to generate the reports required under the terms of the research grant.  In planning research projects, it is necessary to consider the real time required for the research, which is usually greater than anticipated. Planning must also make provisions for the time required for the whole process of applied research and technology transfer to unfold with producers and to arrive at results that can be used in the production system. A6.2.5. SNIAF  The participation of three kinds of institutions and organizations (knowledge generators, service providers, and producers in the CDIs) was adequate. INIAF improved its links 59 with other actors through its technical assistance services. The research and seed areas did not interact with CDIs.  Departmental innovation plans became planning documents for priority innovation themes. Interest was generated in the members of the CDI who participated actively in developing those plans. The potential for obtaining project resources and making an impact on public agricultural innovation policies was also of interest to them.  To improve future interventions, the SNIAF must expand its national identity and scope. The SNIAF requires leadership capable of energizing and motivating actors in the system and ensuring that they continue to have a space for coordination and agreement. INIAF must continue to strengthen its capacity as a leader of this system, especially with respect to maintaining a stable cadre of trained staff that can foster the coordination of actors in this space.  To achieve greater sustainability of the spaces for coordinating actors in the innovation system (such as the CDIs), it is necessary to give them some legal standing and framework to operate (through a Supreme Decree, for instance). 60 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR A7.1. Summary of Borrower’s ICR a. Background 1. The main strategy of INIAF to improve agricultural productivity and promote agricultural transformation is closely linked with improved innovation processes. Bolivia must secure steady increases in the yields of its main strategic crops, which have been lower than yields obtained elsewhere in the region. To do so, it must overcome the institutional challenges of reaching producing households efficiently, and of generating and capturing agricultural technologies and innovations oriented to impact. In this context, Agreement No. 5003-BO was signed for financing the Innovation and Agricultural Services Project (PISA). INIAF (the National Institute for Agricultural and Forestry Innovation) was responsible for implementing PISA. The activities carried out in PISA were part of the consolidation of INIAF, which is responsible for generating innovations that respond to the institutional priorities for supporting the objectives of the agricultural sector. b. Achievements 2. The Borrower acknowledges that many changes have taken place in INIAF following the implementation of the PISA project. These are summarized in the sections that follow. 3. National System of Agricultural and Forestry Innovation (SNIAF). Under PISA, the SNIAF achieved coordinated action through the promotion of CDIs and CRIs, which convened more than 300 representatives and actors on a recurring basis to coordinate issues of departmental and regional importance, such as the prioritization of innovation areas, approval of subprojects, calls for proposals, and innovation events. This activity made it possible to strengthen the capacity of the SNIAF (with political, public, technical and financial implications) to respond to prioritized and concerted demands. Within this framework, an Implementation Plan for the SNIAF feeds into INIAF's sustainability proposal. In addition, SNIAF staff have developed technical capacities, a systematized database, and innovation networks focused on priority areas. Networks for maize, potatoes, and wheat have been strengthened and consolidated. Links are being developed to form networks for rice, forests, and genetic resources. The theme of innovation has come to attention of groups of producers and private actors and is generating expectations and participation. The outlook for the future is for INIAF, through the SNIAF, to improve actions to increase productivity and catalyze agricultural innovation at the departmental and local level. 4. National Directorate of Seed. Thanks to PISA, many processes have been automated, such as the system for managing online seed certification (SISEM). Use of more efficient seed testing methods has also started (for example, the time needed for germination testing has been reduced from 30 days to 3 days). The seed laboratories have been improved with automatic seed germination cameras and more accurate equipment to measure seed moisture levels. An increase in processing capacity was achieved with respect to processes, equipment, and professional skills, and thus INIAF went from certifying 78,691 tons of seed in 2011 to 119,124 tons in 2015. 5. National Directorate of Technical Assistance. During the implementation of PISA, human resources were recruited to manage, structure, and operate the services portfolio in technical assistance and extension. Platforms for identifying demands for services and attending to them were strengthened. Methodological guides and protocols were developed, along with the capacity to implement subprojects with various actors. All of these achievements have contributed to strengthen the capacity to respond to demands for technical assistance in the agricultural sector, 61 focused on the development of capacities, efficient production systems, and sustainability in the face of climate change. 6. National Directorate of Research. Under PISA, the National Research Directorate generated technologies to improve productivity and quality in strategic areas, established populations of germplasm with broad genetic variability, improved infrastructure and equipment for research and the production of high-quality seed, strengthened laboratory capacity, and trained human resources to face research challenges. The project was able to consolidate national research networks for the management of elite lines for the identification and promotion of cultivars in multiple localities, which enables INIAF to develop stable, well-adapted technological alternatives. To achieve some of its research objectives, INIAF adopted a strategic alliance approach for building on technological advances in the areas of interest. 7. The research fund, prior to PISA, had no mechanism to consolidate applied research. Under PISA, the operations of the research fund have been initiated and consolidated, with the central objective of conducting applied research. The fund creates knowledge through public-private strategies to increase production and quality and develop disease control mechanisms. 8. The main activities were the development of mechanisms for convening with CDIs; the development of procedures, guides and protocols for the presentation, evaluation, selection, and monitoring of subprojects; and the organization of meetings of the Research Network. 9. Planning, Monitoring, and Evaluation. Prior to PISA, no clear procedures existed for results-oriented management. IT tools were not available for managing activities and particularly for implementing subprojects being carried out jointly by INIAF and partners. Specialized personnel were not available to carry out tasks related to planning, monitoring, evaluation, and knowledge management. With PISA, systems for adequate planning, monitoring and evaluation of projects (SISPOA and SISPO-AE) were consolidated. 10. Financial Management. Implementation of PISA has enabled the Financial Management Department to propose and consolidate procedures for results-based management, including operational manuals, monitoring and evaluation guides, and structured strategic planning and annual operational plans, complemented by professional expertise in planning, monitoring, evaluation, information technology, and organizational development. 11. The implementation of the PISA called for the Financial Management Department to take up the challenging task of meeting the highly specific financial management requirements of the World Bank. The Financial Management Department met and responded to the requirements. 12. Disbursement. Accounting staff were trained in procedures for World Bank disbursements, and the training was replicated at the national level. 13. Administration. At the departmental level, the Administrative Links have been empowered through training in managing the accounting system and preparing financial reports. The information generated and presented to funding agencies and public bodies was submitted for external financial auditing. 14. Human Resources. Prior to PISA, INIAF lacked trained technical personnel to perform its varied and dispersed activities and fulfill its mandate; many individuals performed activities that did not correspond to their position. PISA supported specialized personnel in all areas of INIAF. This was achieved through training in all areas of the institution and the hiring of 172 professionals. Administrative instruments and regulations were developed for the organization to manage human resources in the framework of recruitment, induction, integration, selection, job classification, internal demand and supply of staff, evaluation, training, and promotion. 62 15. Procurement. No specialized Procurement Area existed prior to PISA. The staff had no specialized training in procurement procedures and lacked capacity to carry out complex procurement processes (only the purchase process was considered). There was no fluid coordination between problem-solving areas, and there was no experience with strategic alliances on the scale of PISA. Under PISA, the Procurement Area was created, adequately structured, and capable of carrying out its operations, from the detection of the procurement need in the POA to the final payment, making national and international public procurement tenders possible. Procurement procedures and flowcharts were developed and incorporated in the Operational Manual, annual procurement plans were improved, and technical and procurement areas coordinated the development of TDRs and technical specifications. The information generated in the Procurement Area and the processes carried out during implementation were subjected to a special procurement audit, which was generally positive. All of these activities changed the concept of acquisitions in the organization. The process now begins with preparation of the POA and includes payments, control of penalties, and compliance with contractual clauses. A7.2. Borrower’s Comments on the Bank’s draft ICR: Note: At the time the ICR was submitted, the Borrower’s comments on the Bank’s draft ICR had not been received. 63 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Comments received from the cofinanciers appear in Section 7(b). Additional feedback was received in the form of comments made directly onto the draft manuscript and has been incorporated. 64 Annex 9. List of Supporting Documents Agencia Danesa de Desarrollo Internacional. 2013. Política de Asociación Dinamarca-Bolivia 2013-2018. Consultores Para el Desarrollo Integral (CPDI S.R.L.). 2016. Medición del Indicador de Satisfacción de los Productores de semilla con el Servicio de Certificación de Semillas del INIAF. Mayo. Government of Bolivia. 2015. Plan de Desarrollo Economico y Social, Estado Plurinacional de Bolivia. La Paz. Government of Bolivia. 2010. Environmental Assessment (published also as World Bank, Report No E2642). Government of Bolivia. 2010. Indigenous Peoples Plan. (Published also as World Bank, Report No IPP469). Hameleers, A.; S. Antezana; and B. Paz. 2012. Estrategias de Inversión en el Capital Humano para la Agricultura. Estudio de caso: Bolivia. La Paz. INIAF. 2016. Presentation on the Agricultural Innovation and Services Project. La Paz. INIAF. 2016. Propuesta de sostenibilidad institucional del INIAF. La Paz INIAF. 2016. Informe técnico INF/INIAF/UPDO/No180/2016. Julio 28. INIAF. 2017. Informe de Cierre: Proyecto de Innovación y Servicios Agropecuarios. January. NATBIO SRL. 2016. Evaluación de Indicadores del PIS, grado de satisfacción de agricultores y agricultoras con la asistencia técnica apoyada por el INIAF. Junio 3. Sociedad Accidental CPDI – Servicios de Emprendimientos Globales 2016. Evaluación Final Externa a la Ejecución y Resultados del PISA. Diciembre. World Bank. 2010. Project Concept Note, Bolivia Agricultural Innovation and Services Project. World Bank. 2011. Interim Strategy Note for the Plurinational State of Bolivia, 2010-2011, Report No. 48372-BO. World Bank. 2011. Project Appraisal Document, Bolivia Agricultural Innovation and Services Project. Report No. 57824-BO. April 26. World Bank. 2011. Loan Agreement, Bolivia Agricultural Innovation and Services Project. August 22. World Bank. 2012 - 2017 (various dates). Aide Memoires from Supervision Missions. World Bank. 2012- 2017 (various dates). Implementation Status Reports. 65 World Bank. 2012 - 2017 (various dates). Financial Management Supervision Reports. World Bank. 2012 - 2017 (various dates). Procurement Post-Review reports. World Bank. 2012 - 2017 (various dates). Project Restructuring Papers, Bolivia Agricultural Innovation and Services Project. World Bank. 2015. Country Partnership Framework, 2016-2020. Report No. 100985-BO. World Bank / FAO. 2017. Economic and Financial Analysis, Bolivia Agricultural Innovation and Services Project. World Development Indicators. http://datos.bancomundial.org/indicador/ 66 Map 67