Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) Report Number: ICRR0021813 1. Project Data Project ID Project Name P113273 LR-Smallholder Tree Crop Project Country Practice Area(Lead) Liberia Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-51010 31-Dec-2016 13,732,883.39 Bank Approval Date Closing Date (Actual) 05-Jun-2012 30-Nov-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 15,000,000.00 0.00 Revised Commitment 15,000,000.00 0.00 Actual 13,732,883.39 0.00 Prepared by Reviewed by ICR Review Coordinator Group Maria Vanessa Fernando Manibog Christopher David Nelson IEGSD (Unit 4) Corlazzoli 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the legal agreement, the objectives of the Smallholder Tree Crop Revitalization Support Project implemented in Liberia were “(i) to increase access to finance, inputs, technologies, and markets for smallholder tree crops in Liberia, and (ii) to develop a long term development program for the tree crops sector” (Legal Agreement IDA 51010, pg. 5). The objectives as stated in the legal agreement are the same as the ones presented in the Project Appraisal Document (PAD, para 16). No substantive changes to the objectives or project’s theory of change were made throughout the project. Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) While the project cost and IDA financing were revised downwards and the targets of the indicators were reduced, the outcomes were generally consistent, hence a split evaluation is not warranted. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 11-May-2016 c. Will a split evaluation be undertaken? No d. Components Component 1: Smallholder Tree Crops Revitalization (Appraisal cost was estimated at US$16.9 million, including an IDA contribution of US$9.5 million. At closing the cost was US$8.613 million in IDA contribution). This project was designed as a pilot with the aim of learning about larger-scale tree crop development program targeting smallholder farmers. The first component aimed at revitalizing the production and marketing of major tree crops (cocoa, coffee, oil palm, and rubber). The strategy included assisting smallholders to rapidly resume production through the rehabilitation of existing farms, replanting/planting, and capacity building. The component's activities also aimed at producing access to long-term credit to meet farm development costs (PAD, para. 21). The component included strengthening value chains. The component was made up of three sub-components each focused on a major tree crop. Sub-component 1 - Cocoa/Coffee Rehabilitation. This sub-component sought to adopt a value-chain approach with investments in production, storage, bulking, processing, and financial products development to facilitate cocoa and coffee certification to access preferential markets (PAD, para. 22). The sub- component aimed to rehabilitate cocoa based on two different models, namely, medium and high input. Activities included the selection of beneficiaries, co-financing agricultural inputs for farm development through start-up grants, the establishment and strengthening of farmers’ organization to enhance their abilities to provide services to members, capitalization of cocoa/coffee, facilitation of the access to financial services by cocoa/coffee smallholders and farmers' cooperatives, and rehabilitation of feeder roads. Sub-component 1.2 - Smallholders Oil Palm Rehabilitation. This component aimed at rehabilitating and replanting oil palm by (a) re-launching production and modern processing in one existing small holder plantation in Grand Gedeh and (b) establishing an outgrowers model around a concessionaire in Grand Bassa. The project would finance a small tool package, fast-yielding seedlings, long-term credit, technical assistance, and farm access road development (PAD, para. 23) Sub-component 1.3 - Smallholder Rubber Revitalization. This sub-component aimed at replanting and planting in smallholder rubber farms, improving the rubber farmer organization (FO), providing training and technical advice, and increasing access through limited feeder road and farm track rehabilitation. Component 2: Institution Building and Preparation of Future Large-Scale Tree Crop Development Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) Program. (Appraisal cost was estimated at US$3.3 million, including an IDA contribution of US$3.1 million. At closing the cost was US$2.078 million in IDA contribution). This component aimed at (a) strengthening institutional capacity of public and private institutions involved with the project, (b) supporting capacity building of the Ministry of Agriculture (MOA) and Cooperative Development Authority (CDA) and (c) supporting the MOA country agricultural offices (CAOs) to strengthen their operational capacity and project involvement. It also sought to (d) secure three crop smallholder’s land use rights and supporting related institutions, (e) commission applied tree crop research, (f) strengthen the MOA’s project management unit and (g) support the Environmental Protection Agency (EPA). This component also sought to prepare for a future large-scale smallholder tree crop development program and included a project preparation advance. Component 3: Project Coordination and Management (Appraisal cost was estimated at US$3.0 million, including an IDA contribution of US$2.5 million. At closing the cost was US$3.0 million in IDA contribution). This component aimed to ensure effective coordination, management, and monitoring and evaluation of the project through the support of steering bodies, the project coordination unit, and the implementation of the M&E system. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. At appraisal, the total cost of the project was estimated at US$23.1 million, of which US$15.0 million would be financed by the World Bank’s International Development Association (IDA) (PAD, pg. viii). During the project implementation, the total cost of the project was revised to US$19.5 million, and the IDA allocation decreased from US$15 million to US$13.7 million (ICR, para. 15). At closing, the total project cost was US$13.692 million (ICR, pg. 49). Financing. A total of US$15.0 million was financed through an IDA credit (IDA-51010). During the life of the project, the Liberian currency depreciated causing a loss of US$1,194,106 expected for implementation (World Bank staff interview, 2020). The IDA allocation available to achieve the PDO therefore shrunk from US$15M to US$13.7M. The actual disbursement at the end of the project was US$13,732,883 (ICR, pg. 2). Borrower Contribution. The PAD stated that Borrower would contribute US$1.1 million, while the beneficiaries would contribute US$6.2 million (PAD, pg. viii and para. 28). The legal agreement did not stipulate a contribution from the Borrower. At the end of the project, there was no evidence in the ICR to suggest that the Borrower or the beneficiaries made a financial contribution. Dates. The project was approved on June 5, 2012 and became effective on May 13, 2013. It underwent a midterm review on November 3, 2015. The project underwent three Level-2 financing restructuring. On July 28, 2014, changes to the legal covenants were made. On May 11, 2016, the project revised the results framework, components, costs, the credit closing date, and the allocations among disbursement categories. During this restructuring, there were also changes to the indicators including the lowering of targets. Some of these changes stemmed from the mid-term review (World Bank staff interview, 2020). At the time of the restructuring, the project was experiencing delays due Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) to the Ebola outbreak and changes to the design of the long-term credit scheme, which were addressed during the first restructuring. On October 10, 2018, the project again reallocated funds among disbursement categories due to the rising costs of consultant services (ICR, para. 20). The project’s original closing date was December 31, 2016. The project was extended at the May 11, 2016 restructuring and the project closed 23 months after the original closing date on November 30, 2018 (ICR, pg. 2). 3. Relevance of Objectives Rationale Country Context: At the time of appraisal, Liberia had emerged from two civil wars and was undergoing its second democratic election. The country was experiencing steady real growth rates and real GDP growth was forecasted at about 7 percent in 2012 (PAD, para. 2). In 2010, the country had reached the Highly Indebted Poor Country (HIPC) status, which provided for basic substantial debt relief and new public and private investments in all sectors. The agriculture sector had been the backbone of the economy throughout the conflict and in the post-war period, accounting for 61 percent of GDP in 2008 and providing employment for 4 million people (PAD, para. 4). Rice and cassava were the main food crops, while rubber, oil palm and cocoa comprised the majority share of the country’s agricultural exports. Despite favorable agro-climatic conditions and available labor force, the tree crop sub-sector was performing sub-optimally. There were little replanting or maintenance activities (PAD, para 5-6). Tree plantations were very old and productivity was low. Processing factories were operating below capacity, despite high international demand for tree crop products. The tree crop sub-sector also was experiencing supply-side challenges including relatively small–scale farms, remote location of farms, poor conditions of rural roads, limited market linkages, few market-oriented farmer organizations, and limited capacities of the Ministry of Agriculture (PAD, para. 10-12 and ICR, para. 3). Previous Sector Experience: At appraisal, the project design took into consideration support that was being provided by the International Finance Corporation (IFC) to the Ministry of Agriculture. The IFC was supporting the certification of the oil palm sector, the establishment of a credit line for small and medium enterprises, and the designing of an agri-finance program (PAD, 15). The project was developed following the recommendations of the Diagnostic Trade Integration Study (DTIS) and designed by taking into account lessons learned from previous tree crops projects financed in the World Bank in the 1970s and the 1980s in Liberia and other West African countries (PAD, para 16). Alignment with Government Priorities: The project’s development goal aimed to improve economic revitalization through the promotion of export-oriented economic growth, while simultaneously facilitating rural development (PAD, para. 23). The proposed project is aligned with the 2010-2020 Liberia Agriculture Sector Investment Program (LASIP) developed by the Government of Liberia. The project was meant to support three out of the four programs of the LASIP including supporting activities related to smallholder Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) tree crop development, rehabilitating and expanding rural roads, rebuilding the Ministry of Agriculture and improving coordination, developing the capacity of farm-based organizations, and revitalizing agriculture research (PAD, para. 15). The project’s PDOs supported the LASIP, which was implemented from 2011- 2015. In 2013, the Government of Liberia drafted the country’s vision and strategy for socio-economic transformation and developed the Agenda for Transformation – a five-year development strategy. Throughout the Agenda of Transformation, the government made commitments to improve the results of smallholders farmers and to support them in increasing agriculture productivity through access to markets, improvements in infrastructure, finance, skills, regulations and property rights (Agenda for Transformation, pg. 37 and 58). Alignment with World Bank Assistance Strategy: The PDOs of the project was aligned with the World Bank’s Country Assistance Strategies at inception and contributed towards the Country Partnership Strategy at completion. At appraisal, the PDOs aligned with the Joint Country Assistance Strategy for FY09-FY12. It contributed towards the expansion of agricultural development programs and aimed to address some of the structural and sectoral development challenges associated with agriculture. The project aimed to support CAS Outcome 4: Improve agriculture and natural resource management in ways to generate pro-poor growth (CAS FY09-12, para. 77-19). Similarly, the PDOs contribute towards attaining the Country Partnership Strategy FY 13-17, Outcome 4: Improved management and productivity in agriculture, forestry, and fishery (CPS FY13-17, pg. 21). While the overall focus of the Country Partnership Strategy for FY19-24 shifted towards the attainment of human development, the PDOs contributes towards Pillar I: Strengthening Institutions and Creating an Enabling Environment for Inclusive and Sustainable Growth (CPS FY19-24, pg. 35). Overall, the project’s PDOs aimed to close critical gaps that were affecting smallholder tree crop owners. The project’s PDOs were relevant to the context of the country and aligned with the government’s priority. While the PDOs were more closely aligned to the Country Assistance Strategy at the beginning of the project than towards the end of it, the relevance of the PDOs is rated as substantial overall. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To increase access to finance, inputs, technologies, and markets for smallholder tree crops in Liberia. Rationale Theory of Change: The aim of the project was to revitalize and grow Liberia’s tree crop sector by addressing smallholders tree farmers’ lack of access to finance, inputs, technology, and markets, while simultaneously improving infrastructure and strengthening government agency capacities. The retrospective Theory of Change suggested that if farmers applied the skills they learned, had the necessary tools, and cooperatives Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) purchased crops from the members, then there would be an increase in rehabilitation and replanting, that would lead to increase yields. The theory also stated that if there was improved infrastructure then farmers would be better able to transport their crops to markets. Finally, if government agencies increased their capacities in the areas of human resources, research, development of master plans, formulation of national policies, then a long-term and large-scale tree crop development program would be developed. An implicit assumption is that if there is increased small tree yields, and farmers have long-term financing as well as access to markets, the farmers would increase their income which will contribute to a decrease in poverty. The theory of change had significant shortcomings. Even though the PAD outlined the different ways in which the project incorporated lessons learned, the design of the project made several assumptions that did not materialize. These had major implications on the project implementation and the achievements of the PDOs. Assumptions related to access to long-term finance, which was part of the PDOs, did not come to fruition. For example, the theory of change assumed availability of counterpart cofinancing. The project expected US$8.1 million from the Government of Liberia, concessionaires, and farmers. These contributions were expected to be in-kind but they did not materialize (ICR, para 61i). Similarly the project assumed that farmer’s priorities included seeds, fertilizers, and agro-chemicals, whereas in reality, the farmers indicated hiring labor was their first-order priority, followed by mechanical services, and transportation (ICR, para. 61ii). The project also assumed long-term credit schemes would be available for farmers until new tree crops materialized. However, a large-scale program that would provide the credit did not materialize (ICR, para 61iv). The Ministry of Finance and Development Planning also did not provide financing after the project was completed (ICR, para 61iv). The project also assumed that there would be access to quality inputs, however seeds, fertilizers, and agro-chemicals were not readily available in Liberia and had to be imported (ICR, para 61v). The theory of change assumed that revitalizing the tree crop sector could be achieved by shifting outcomes at the individual level of analysis. Aside for some capacity strengthening at the government agency level, the project did not seek to change any power, political, or structural causes for the lack of growth in the small tree crop sector. The project indicators remained relatively the same throughout the life of the project. That said, the outcome targets were reduced during the second restructuring. One out of the six PDOs indicators was dropped and replaced with a new indicator, and two PDOs indicators were moved to the intermediate level. While the indicators in the project focused on the outcome (tree crops rehabilitated and yield increase), they did not measure the totality of the project's contribution to the different parts of the PDOs, such as access to market, technology, and inputs. The indicators did not measure knowledge gain or change in behavior of the targeted farmers. The ICR is very reflective and explicit about identifying gaps in information and lack of evidence. The monitoring and evaluation system did not close these gaps with additional qualitative or quantitative data. Increase access to Finance Outputs A total of US$62,390 in matching grants were provided to 12 cooperatives with 119 farmer organizations (Target of US$128,000 not met, Target of 40 farmer organizations met, ICR, para. 30 and pg. 43). The funding allocated was less than 50% of the minimum intended funding (ICR, para 31). However, the grants were able to reach more people than expected with a total of 2,974 farmers receiving matching grants (ICR, para. 31). Fewer funds were distributed to more farmer organizations, and as a result, more farmers were reached. Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) A total of U$115,526 was provided in in-kind credit to one rubber cooperative in Montserrado county, comprising of 150 farmers (ICR, para. 32 and pg. 43). The funds were used to purchase fertilizer and pesticides. Outcomes: At the beginning of the project, according to baseline data, none of the farmers had access to long-term credit or finance (ICR, para. 32). As a result of the project, a total of 3,518 project-supported farmers had access to credit or benefited from access to some form of matching grant (Target of 3,025 met, ICR, para. 33). However, the evidence on attribution is weak since the data on who received the long-term credit is inconclusive. While the project database states that 1,625 cocoa farmers from Nimba county received credit, project documents and the financial institution confirmed only one organization benefited from long-term credit scheme (ICR, para. 33). According to the ICR, long term credit was provided to 3 out-grower schemes (ICR, pg. 43). However, at least two of the three out-grower schemes were waiting for the project to fulfill its promise on access to long-term credit at the end of the project. At project closure, there were no long-term partnerships or financial arrangements in place to ensure continued farmer’s access to credit for maintaining their farms with tree crops (ICR, para. 33). The project “acknowledged that farmers would need financing from commercial banks at the end of the project to maintain the tree crop established” (ICR, para. 32). The project provided financing to rubber producers and farmers. According to World Bank staff the financing was used to establish a new rubber plantation (World Bank staff interview, 2020). While a larger number of farmers had access to finance during the project, the funds allocated were less than what was originally projected. The access to funding was not long-term or sustainable ,which was not conducive to long-term investment or growth of the sector. As a result this outcome is rated as modest. Increase access to inputs Outputs:  67 farmer field school facilitators were trained on agrochemicals (ICR, para. 35).  52 and 26 nurseries were set up in 2016 and 2017 respectively (ICR, para. 35).  5 people were trained (Target of 70 not met, ICR, pg. 39).  US$ 2,196,155 was provided in grants to purchase inputs (ICR, pg. 44).  A total of 3,518 farmers received at least one of the following: small tools, fertilizers, or chemicals. Outcomes: The project was able to provide farmers with inputs and training. Unfortunately, the monitoring and evaluation system did not capture how the farmers used the inputs and to what extent they helped with the tree crops. The project did not collect sufficient data on how the inputs were used, who used them, and whether productivity increased as a direct result of the project. No results were reported on the establishment of seed grants and relevant out-grower concessionaire linkage (ICR, para. 35). There were also no informal linkages arrangement with WIENCO – a Dutch cocoa company which provided technical training and input on credit to cocoa farmers under the rehabilitation scheme. Project data on WIENCO’s input and related farmers’ benefit was also absent (ICR, para. 35). In conclusion, while farmers received inputs and training, it is difficult to Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) assess the contribution or quality of the inputs due to the poor monitoring and evaluation system. As a result this outcome is rated as modest. Increase Access to technology Outputs  Construction of palm oil processing stations at the farmer-managed plantation in Grand Gedeh (ICR, para. 37).  Installation of two motorized mills with the processing capacity of at least 40 drums of fresh fruit bunches within 45 minutes of four mills (ICR, para. 37).  Sixteen cocoa cooperatives were reported to have received moisture testing meters (ICR, para. 37). Outcomes: The project reported an increased oil production at the factory which enhanced the plantation’s income generation capacity, although quantitative data on the pre- and post-processing activities incomes was difficult to obtain (ICR, para 37). No results were reported on the applied tree crop agricultural research activities. Data on the total number of farmers that benefited from training and use of the improved techniques provided was not available (ICR, para. 37). Thus, the results data in terms of increased access to technology are mixed or negligible. Increase markets for smallholder tree crops Outputs  5km of road were rehabilitated (Target of 17km not achieved, ICR, para. 39). Outcomes Anecdotal evidence gathered for the ICR, suggested that 48 beneficiary farmers in 9 communities living nearby the stretch of road rehabilitated were able to take their products to market, including in rainy season. Although limited in scale, the matching financial grants allowed some farmers to sell their crops directly to the cooperatives (ICR, para. 39). That said, the ICR is candid about the lack of data on formal market arrangements to guarantee the beneficiary farmers of a market once their tree crops were ready for harvest (ICR, para. 39). Out-grower schemes were also not established and cooperatives supported by the project also expressed concern about market prospects once trees were ready for harvesting (ICR, para. 39). Thus, as a result, the ICR did not provide conclusive evidence that the project achieved the outcome of increasing access to markets in a satisfactory manner – whether it was physical access or formal market arrangements. As a result, this outcome is rated as modest. Overall Outcome of the Objective: A total of 3,850 beneficiaries, of which 23% were female, were reached as a result of the project (Revised Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) target of 3,100 met, Original Target was 5,000, ICR, pg. 14). Of the beneficiaries, 3,518 were farmers of which 20% were women (ICR, para. 44). A total of 3,750 small tree crop farmers had access to finance, input, market, and technology as a result of the project (Target of 3,025 met, ICR, pg. 14). The project’s support rehabilitated, replanted or planted 5,828 hectares (Target of 6,056 not met, Original target 11,300, ICR, pg. 14). Of the 5,828 hectares, 4,194 hectares were rehabilitated, 1,004 hectares were replanted, and 630 hectares were of new-planted trees (Target of 300 for new planting met, Targets for the rehabilitation of 4,506 and replanting of 1,250 were not met, ICR, pg. 14). Moreover, the targeted yields per metric tons per hectare (mt/ha) were all not achieved, as follows: high input cocoa 1.00 (target of 1.30 not met), medium input cocoa 0.46 (target of 0.50 not met), coffee 0.30 (target of 0.50 not met), and for oil palm 1.50 (target of 5 not met) (ICR, pg. 14). The weighted averages are 0.63 mt/ha for the target yield and 0.48 mt/ha for the yield achieved, meeting 77% of the expected target (ICR, para. 41). The ICR speculates that the lower achievement in comparison to the yield indicator was due to lack of credit to hire labor and lack of access to market (failure to rehabilitate full road) (ICR, para. 41). That said, lack of access to markets is not necessarily the driving factor that contributes to the productivity that a hectare can yield in a farm, which is dependent on factors such as quality of the soil, application of fertilizers or pesticides, quality of the seeds, irrigation, and labor. While the ICR may have speculated that low achievement was due to lack of access to market (failure to rehabilitate road), this factor must have not been a direct determinant of tonnage per area planted. On balance, the project did not meet key inputs and outcomes. The targeted increases in productivity were not met. Moreover, the project failed to secure long-term credit for farmers, which puts the sustainability of outcomes at significant risk. Overall, key performance indicators were for the most part partially achieved. While the project reached the expected number of small tree crop farmers to engage with, there was only a modest increase of access to finance, inputs, technology, and markets. While taking into account the project's implementation in a fragile environment and the delays experienced related to the Ebola outbreak, the project did not lead to the expected number of hectares rehabilitated or replanted. The rehabilitation and replanting of small trees also did not lead to the expected increase in yields. The project also did not include any indication that the farmer's income had increased or that there was a reduction of poverty as a result of the project's increase yield. As a result, the project only modestly achieved this first objective. Rating Modest OBJECTIVE 2 Objective To develop a long term development program for the tree crops sector. Rationale Outputs  5 Country Agricultural offices were equipped (Target of 5 met, ICR, pg. 39).  5 CDA field offices equipped (Target of 5 met, ICR, pg. 40). Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273)  5 people were provided financial assistance to complete their final year of a Ph.D. program in human nutrition, entomology, agricultural economy and crop science (ICR, pg. 46).  Methodology for community land use rights validation agreed with stakeholders (Target met, ICR, pg. 41).  Tree crop master plans were formulated in a participatory process under Ministry of Agriculture leadership for cocoa, rubber and oil palm. (Target met, pg. 41). The master plans were accompanied by an action plan for promoting each tree crop (ICR, pg. 46). The master plans for coffee were not completed (Target not met, ICR, pg. 42).  The national policy and strategy for Farmer Organizations (FO) and cooperative development was not formulated and validated by all national stakeholders (Target not Met, ICR, pg. 40). That said, the project was requested to help update the Cooperative Development Act instead of the FO policy and strategy (ICR, pg. 40).  Support was provided to create the Liberia Agriculture Commodities Regulatory Agency (an agency within the Ministry of Agriculture) by assisting with feasibility studies, and the elaboration of the Liberia Agricultural Commodities Regulatory Authority (LACRA) Bill. (ICR, pg. 46.) Outcome The long-term, large-scale tree crop development program (STCRSP II) was only partially formulated and was not approved by the Ministry of Agriculture (Target not met, ICR, pg. 14). The project did help advance the master plans for three out of the four tree crops. These master plans were approved by the Ministry of Agriculture (ICR, para. 42). These master plans were comprehensive, according to the ICR, identifying long- term strategic tree crop sector goals, including a strategic road map. They also outline a set of recommendations and policies. The Government of Liberia has used the master plan to request for additional funding for a second phase of this project (ICR, para. 43). The ICR does not provide any evidence on the change of government capacity as it relates to human resources or research. In conclusion, while progress was made to create a master-plan that included long-term strategic tree crops sector goals, it fell short of developing a long-term development program for the tree crops sector, which is the project’s development objective. Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale In conclusion, the objectives were only modestly achieved. The project’s inability to secure long-term finance for the smallholder tree crop farmers will put a lot of the partially achieved outcomes at significant risks. While the project was able to provide access to finance to more farmers than originally expected, each farmer also received fewer funds. The results in terms of increased access to technology were inconclusive due to lack of data related to oil production. Similarly, while farmers received inputs and training, it is difficult to assess the contribution or quality of the inputs due to the poor monitoring and evaluation system. The project built only Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) 5km of road (target 17Km), and the ICR also did not provide conclusive evidence that the project achieved the outcome of increasing access to markets in a satisfactory manner – whether it was physical access or formal market arrangements. While the project reached the expected number of small tree crop farmers, the limited achievement of increased access to finance, inputs, and technology did not lead to the expected number of hectares rehabilitated or replanted. Moreover, targets on tree yield were also only partially met. The project also did not include any indication that the farmer's income had increased or that there was a reduction of poverty as a result of the project's increase yield. While the Ministry of Agriculture was able to create master plans for three out of the four tree crops, it did not meet the objective of developing a long-term development program for the tree crop sector. Moreover, in conclusion, even the partially achieved results cannot be attributed conclusively to the project’s interventions because of the weak M&E (discussed in Section 9) and insufficient causal evidence. Overall Efficacy Rating Primary Reason Modest Low achievement 5. Efficiency Scope of Analysis: A financial and economic analysis was carried out with and without simulations prior to the project’s implementation in 2012. The same methodology used in the appraisal was replicated during the end of the project analysis. Financial Analysis: The financial analysis was carried out to assess whether (a) the targeted smallholders would get sufficient cash income to justify their adherence to and participation in the project, (b) the proposed financial arrangements for farm development costs would be worth the risk to the different beneficiaries, and (c) the proposed models and investments would be attractive to other potential private partners. During the appraisal, financial internal rates of return and incremental incomes appeared to be attractive for all eleven crops tested (PAD, para 42). For crops with long gestation period, such as oil palm and rubber, the repayment could be achieved in 11 years. Economic Analysis: At appraisal, the economic analysis expected that the project would generate direct and indirect economic benefits related to (a) increased incomes of tree crop smallholders, (b) reduced transaction costs and post-harvest losses, (c) increased value of added tree crop production, (d) enhanced market/business opportunities and economies of scale benefiting all actors of the supply chain, (e) enhanced bargaining power, (f) incremental on and off-farm employment, (g) foreign exchange savings, and (h) improved social stability in project areas (PAD, para 43). Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) A cost-benefit analysis was conducted over a 25 year period. At appraisal, without considering the project and long-term tree crop program preparation costs, the project was expected to yield an Economic Internal Rate of Return (EIRR) of 33 percent and a Net Present Value (NPV) of US$28 million, at a 10 percent discount rate (PAD, para. 43). The EIRR would be 29% if all the project costs were considered. At project closing, the EIRR was 29.2%, with a NPV of US$26.7 million. The benefit-cost ratio (BCR) was 2.99 at a 10% discount rate (ICR, para. 47 and pg. 56). This implies that for every US$1 invested by the project and the beneficiaries, the tree cropping interventions would result in approximately US$3 in benefits for the smallholder farmers (ICR, pg. 56). One significant shortcoming of the ICR's efficiency section is that it did not disclose or explain why there was a significant difference in the reported EIRR and NPV vis-a-vis the calculated numbers in the project's Impact Evaluation report conducted by LTS Consulting. According to the ICR and an interview with Bank staff, the ICR's analysis was based on the Project Impact Evaluation results (ICR, Annex 4, pg.50). However, the two documents report different results. The ICR reported that the EIRR was 29.2%, with a NPV of US$26.7 million. and the benefit-cost ratio (BCR) was of 2.99 at a 10% discount rate (ICR, para. 47 and pg. 56). This implies that for every US$1 invested by the project and the beneficiaries, the tree cropping interventions would result in approximately US$3 in benefits for the smallholder farmers (ICR, pg. 56). Meanwhile, the project’s impact evaluation report estimated a NPV of US$17.7 million, with an internal rate of return of 17.5% (LTS Impact Evaluation Report, pg vi). The Impact Evaluation Report also reported a BCR of 2.12 indicates that for every $1 invested, smallholder farmers are likely to receive $2 in benefits (LTS Impact Evaluation Report, pg vi). This cost benefit analysis of the Impact Evaluation Report was estimated to run over a 25-year period with a 10% discount rate (LTS Impact Evaluation report, pg. vi). In sum, the EIRR and NPV reported in the ICR are higher than those in the Project Impact Evaluation Report. The Impact Evaluation Report numbers, while positive, are also significantly less than the projected numbers at appraisal. While both numbers showcase positive results, the gap in reported numbers further highlights inconsistencies in how data was reported in this project. The ICR also reports that the EFA results varied by tree crop models. The new-planting cocoa, high-input cocoa revitalization, and oil palm and rubber models were the most successful, while coffee results presented negative returns (ICR, pg. 58). As a result, the investments in coffee returns were a negative net present value of US$208,147 and a BCR of only 0.33. The economic analysis recommended that any future phases of the project should not continue to support the coffee model (ICR, pg. 58). Sensitivity Analysis: At project closing, the sensitivity analysis indicated that under the scenario of 20 percent or more cost increase, the corresponding EIRR remained high with a positive NPVs (ICR, para. 49). It would take a rise in costs of over 50 percent and a simultaneous reduction in benefits of over 55% for the entire project to not be economically viable (ICR, para. 49). Operational Efficiency: The project experienced almost a year delay (11.4 months) to become effective due to slow parliamentary ratifications (ICR, para. 50). Moreover, there were institutional weaknesses and capacity Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) challenges that led to slow project start-up, and weak implementation performance (ICR, pg. 58). These initial delays were also exacerbated by the Ebola virus outbreak, which were external and beyond the control of the project. The project's ex-post economic and financial analysis indicated positive returns of investments for the project and for the individual farmer. However, the project experienced operational efficiency delays, in particular in slow parliamentary ratifications of the project. Also, one significant shortcoming is the lack of clarity in the ICR's efficiency section with regards to why the reported EIRR, NPV, and BCR differ from the analysis conducted in the Impact Evaluation Report. Overall, the efficiency of this project is rated as Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  33.00  Not Applicable 100.00 ICR Estimate  29.20  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project’s PDOs aimed to close critical gaps that were affecting smallholder tree crop owners. The project’s PDOs were relevant to the context of the country and aligned with the government’s priority. The PDOs were more closely aligned to the Country Assistance Strategy at the beginning of the project than towards the end of it. Overall the relevance of objectives is rated as substantial. In the project's design and theory of change, assumptions were made related to smallholder farms gaining access to finance, and other critical assumptions that did not come to fruition. The project only partially achieved key indicators. While the project reached the expected number of small tree crop farmers, the increased access to finance, inputs, technology and markets did not lead to the expected number of hectares rehabilitated or replanted. Moreover, targets on tree yield were also only partially met. While the project was implemented in a fragile environment and experienced delays related to the Ebola outbreak, it did not fully achieve its objectives. A weak monitoring and evaluation system also contributed to the inability to measure conclusively the achievements of this pilot project. Overall, the efficacy of this project is rated as modest. The project experienced operational efficiency delays, in particular in slow parliamentary ratifications of the project. The project’s ex-post economic and financial analysis indicated positive returns of investments for the Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) project and for the individual farmer, although the Efficiency section primarily focused on NPV and EIRR and not on individual financial returns for the farmers. One significant shortcoming is the lack of clarity in the ICR's efficiency section with regards to why the reported EIRR, NPV, and BCR differs from the analysis conducted in the Impact Evaluation Report. Overall, the efficiency of this project is rated as substantial. On the basis of substantial relevance of objectives, modest efficacy, and substantial efficiency, the overall outcome of the project is rated as Moderately Unsatisfactory. (The overall outcome rating based on this combination of three sub-ratings is in line with the guidance from Appendix H, page 38 of the Bank's ICR preparation guidelines dated September 27, 2018). a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome The risks to the development outcome are significant due to the following factors: Financial – The farmers that benefited from this project do not have access to long-term credit or financing to continue to invest in tree crops, inputs, and technology (ICR, para. 80). As a result, it is possible that the gains created as a result of this project will not persist beyond the life of the project. Moreover the farmers may go into debt if they decide to pursue commercial financing on their own. Government Ownership and Commitment – The Government of Liberia demonstrated a commitment to continuing to support tree crop rehabilitation through the creation of the Tree Crop Master Plans. It is also supporting the Smallholder Agricultural Transformation and Agribusiness Revitalization Project (STAR-P, P160945) that aims to increase access to markets, finance, technology and inputs for smallholder farmers. The project focuses on oil palms, so some of the beneficiaries supported by the STCRSP project could potentially continue to receive support (ICR, para. 80). That said the project experienced delays, poor M&E systems, and unmet assumptions that had a negative effect on the outcome of the project. The ICR was unable to demonstrate an increase in government capacity and it is uncertain what is the government's long- term commitment to providing financing to small tree farmers. Without a shift in government priorities, future projects are also likely to experience similar challenges. 8. Assessment of Bank Performance a. Quality-at-Entry On paper, the theory of change of the project was logical and the project’s objectives reasonable. Unfortunately, the design did not take into account the operational conditions of working in a country that had a fragile, conflict, and violence status. According to the ICR, the scope of the project was ambitious with multiple activities, models, and targeted counties (ICR, para. 77ii). There were also flaws in the Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) results framework with performance indicators not fully aligning or measuring the objectives (ICR, para 77i). Moreover, the project was designed based on critical assumptions that did not materialize. For example, the project was expected to receive US$8.1 million in co-financing from the Government of Liberia, concessionaires, and farmers, but it did not materialize (ICR, para 61i). Priority investments needs of smallholder tree crop farmers to hire labor were not followed through by the project (ICR, para 61ii). Assumptions related to concessionaires and large-scale farm owners providing technical-support to smallholder tree crop farmers and participation in long-term credit schemes also never materialized (ICR, para. 61iii). The project also did not achieve the objective of creating a large-scale program. The Ministry of Finance and Development Planning did not provide subsidiary financial agreement (ICR, para 61iv). As a result, the long-term credit scheme needed to ensure that tree crop farms reach maturity was not accomplished (ICR, para 61iv). Sourcing and gaining access to quality seeds, fertilizers, and agrochemicals were challenging and lengthy. Expectations that input suppliers would extend farmers and their organizations with in-kind credit were not met (ICR, para 61v). As the ICR adequately reflects, there was insufficient attention to outlining risk mitigation in the operational risk assessment framework. For example, the risk assessment noted reluctant financial institutions as a risk but provided few specific details on how to mitigate this risk (ICR, para 77iii). As a result of the project’s design flaws, ambitious scope, faulty or unrealistic assumptions, and insufficient attention to outlining detailed risk mitigations, the quality of entry is rated as unsatisfactory. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision The Ebola crisis had a significant impact on the project implementation and delayed the timeframe for achieving results. Most of the project results were generated after the mid-term restructuring, which mitigated some challenges. The restructuring focused on revising indicators and reducing targets but did not reduce the scope or timing of the activities to address the design flaws (ICR, para. 78). The project also did not make alternative arrangements to ensure that farmers had access to credit to maintain their farms at the end of the project, particularly when it was evident that expected arrangements would not materialize (ICR, para. 78). The project experienced significant challenges in implementing the Monitoring and Evaluation (M&E) system. As the ICR states, “the shortcomings in the M&E were particularly consequential given the pilot nature of the project” (ICR, para. 78). While the Bank made important project changes in the restructuring, it was unable to sufficiently address project implementation weaknesses, under-development of the M&E system, and insufficient support to the Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) implementing partner to adequately mitigate project implementation weaknesses. As a result, the quality of supervision is rated as moderately unsatisfactory. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project’s theory of change aimed to support Liberia’s tree crop sector by addressing a lack of access to inputs, technology, markets, and finance, while simultaneously improving infrastructure and strengthening government agency capacities. The theory of Change suggested that if farmers applied the skills they learned, had the necessary tools, and cooperatives purchased crops from the members then there would be an increase in rehabilitation, replanting, and increase yields. The theory also stated that if there was improved infrastructure then farmers would be better able to transport their crops to markets. Finally, if government agencies increased their capacities in the areas of human resources, research, development of master plans, formulation of national policies, then a long-term and large-scale tree crop development program would be developed. The project had a handful of indicators that were modified through the restructuring, including the decrease in targets. The indicators did not accurately or adequately the outcomes or the achievement of the PDOs. For instance, there were no indicators to measure improved access to inputs, technology, or finances, or increased government capacity. Similarly, as the ICR pointed out, some of the indicators were not measurable or hard to attribute to the project, for example, “incremental yearly net cash-flow” (ICR, para. 65). The project’s M&E system was supposed to be established and implemented through annual participatory planning and evaluation workshops, as well as thematic evaluations (PAD, para 64 ). A baseline, a mid- term review, and a final evaluation were expected to be undertaken to identify lessons learned and integrate them into future programs (PAD, para 63 ). Moreover, impact evaluations that measured the increase yields and income derived were expected to take place during the life of the project (PAD, para. 34). The M&E system was supposed to embed a strong learning objective, given that this project was considered a pilot (PAD, para 63). The M&E system was also supposed to include a communication and knowledge sharing strategy to disseminate key information through various channels and target different audiences (PAD, para. 64). Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) According to Bank staff interview, all the M&E systems outlined in the PAD, including projected data collection activities (baseline, midterm and final evaluation), personnel, and data reporting process were in place during the life of the project. The responsibility of the M&E system was of the participating partners or service providers. The project had a full time Planning and M&E officer (PAD, para. 33). The Project Coordination Unit (PCU) also prepared quarterly reports on M&E findings (PAD, pg. 41). b. M&E Implementation During the course of the project implementation, a web-based M&E system was developed to document and capture outcomes, impacts, and facilitate the sharing of lessons learned (ICR, pg. 26). The management information system was completed by the mid-term and linked to the Geographic Information system (GIS), but unfortunately only one individual used the system, the M&E officer (ICR, para. 65). A consulting firm was also hired to carry out the baseline and support in the monitoring of key indicators (World Bank staff, interview). The mid-term review restructuring attempted to correct some of the M&E design flaws but challenges remained in terms of “interpretation and appropriate level of indicators” (ICR, para. 65). The ICR was very frank and upfront about the M&E challenges and its significant shortcomings. For example, the ICR noted there were gaps in data and analysis of key project activities that affected development outcomes. For example, there was a gap in information with regards to the use of the matching grants, the price paid to the farmers by the cooperatives, and price obtained by the cooperatives in the resale of crops (ICR, para. 66). There were also data gaps related to the unit cost of some of the investments, including feeder roads and/or delivering services compared to unit costs incurred by similar projects or standard costs commonly used in Liberia (ICR, para. 66). The ICR also pointed out that there were few project activities at the time that the mid-term assessment was conducted. After the mid-term assessment, data collection on the volumes sold and prices obtained for rehabilitated tree crops was not maintained(ICR, para. 66). Similarly, while the M&E system reported the aggregated number of farmers trained, feedback from farmers and implementing partners signaled that there were differences in the type and level of trainings (ICR, para 66). The M&E system did not maintain basic data such as information on the type of training provided and the beneficiaries (ICR, para 67). The MIS failed to deliver detailed information, including on the specific model smallholder farmer adopted, time of adoption, attrition of smallholder farmers (ICR, para. 68). c. M&E Utilization The ICR stated that the MIS generated materials that were used to inform project beneficiaries and other stakeholders on the status of the project activities. The information was used to create quarterly and semi-annual reports, as well as communication materials (ICR, para. 70). The MIS also generated socio-economic descriptions of beneficiaries and size of beneficiaries’ farms, which were used to write the ICR (ICR, para. 70). Despite these uses of the MIS, the M&E data collected were primarily inputs as opposed to the evidence of the achievements of the outcomes (ICR, para. 70). Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) The results framework, theory of change, and indicators had some design flaws, that were unfortunately not all fully addressed during the restructuring. The ICR is very honest and upfront about the serious limitations of the implementation of the M&E system, including data gaps, missing information, poor quality of data, and overall lack of rigor. While an MIS system was created and used, the M&E system was unable to collect or capture evidence that could demonstrate the achievements of outcomes, let alone lessons learned in a project that was meant to be a ‘pilot.’ As a result, of the significant M&E weaknesses, the overall quality of M&E is rated at negligible. M&E Quality Rating Negligible 10. Other Issues a. Safeguards The project was classified as Environmental Category “B” Partial Assessment. At appraisal, it triggered the following safeguards: Environmental Assessment (OP/BP 4.01), Natural Habitats (OB/BP 4.04), Forests (OP/BP 4.36), Pest Management (OP 4.09) and Involuntary Resettlement (OB/BP 4.12) (PAD, pg. ix). An Environmental and Social Management Team was set up under the project to monitor safeguard issues. This group had a strong collaboration with the Environmental Protection Agency of Liberia and the Ministry of Agriculture. The PIU unit was staffed with an Environmental Specialist. The project remained in compliance with the Bank’s environmental and social policies (ICR, para. 73). An Environmental and Social Management Framework (ESMF), Resettlement Policy Framework and site- specific ESMPs were prepared (ICR, para. 72). No specific environmental or social issues arose during the implementations (ICR, para. 73). There were also no issues with land acquisition, restriction on land use or benefit sharing. Subproject screenings also ensured that farms with contested ownership were not included in the project (ICR, para. 73). b. Fiduciary Compliance The financial management system, including accounting, auditing and reporting was adequate and consistent, according to the ICR (ICR, para. 74). There were three areas where the project’s financial management system could have been improved. First, manual accounting was being used despite numerous attempts to have the project team use accounting software (ICR, para. 74). Second, audited reports were submitted late in two financial years. Finally, a qualified accountant was only hired in the last year of the project implementation (ICR, para. 74). At the end of the project, financial management was rated by the Bank as moderately satisfactory due to funds being used for intended purposes, timely and reliable financial reports, and measurements in place to protect the assets of the project (ICR, para. 74). At project closing, the procurement rating was moderately satisfactory by the Bank. The procurement of Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) the project slowed down during the Ebola outbreak. The Ministry of Agriculture also did not renew the procurement specialist’s contract during the final year of implementation and hired a specialist with less experience (ICR, para. 76). The Bank provided hands-on support and training clinics for the new procurement specialist (ICR, para. 76). c. Unintended impacts (Positive or Negative) None d. Other None 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment The outcome is rated as Moderately Unsatisfactory mainly due to the lack of achievement on the objectives as demonstrated in the Efficacy Moderately Moderately Outcome section. Based on the Bank and Satisfactory Unsatisfactory IEG Guidelines, the substantial relevance, modest efficacy, and substantial efficiency correspond to the outcome rating of moderately unsatisfactory. There were appraisal design flaws and assumptions were made that did not materialize. Moderately Moderately Bank Performance The Bank was unable to provide Satisfactory Unsatisfactory sufficient support to address monitoring and evaluation weaknesses. The M&E system was weak and unable to capture or measure basic output data. There were also important inconsistencies in Quality of M&E Modest Negligible the data and poor data quality. Moreover, the poor M&E system impacted the efficacy of the project. Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) Quality of ICR --- Substantial 12. Lessons The following lessons were provided by the ICR and presented here with some adaptation: 1. Project activities need to be adequately timed and sequenced in order to yield the greatest possible outcomes. For example, in the case of building processing factories and warehouses, it is important to complete these activities early in the project. If they had been available at the beginning of the project, the benefits from access to the facilities could have contributed to generating increased income (ICR, para. 81iv). 2. Treecrop projects may need longer than four years to test results. For example, it takes several years for tree crops to mature, particularly if they were newly planted or replanting is involved. As a result, some of the positive or negative outcomes of the project will not be know for several more years (ICR, para. 81iv) The following lesson is provided by IEG: 3. At the design stage, it is important to test assumptions thoroughly and corrective actions mapped out accordingly. The project made several assumptions that did not materialize during the implementation of the project. For example, the project assumed that it could secure long-term credit schemes for the farmers until the new tree crops would mature, which did not materialize. Similarly, the project made unrealistic assumptions about in-kind contributions of seeds, fertilizers, and agrochemicals. The inability to meet these assumptions reduced the quality of outcomes and made the achievement of the objectives more difficult. To fully test assumptions, the project team needs to have extensive upfront consultations and negotiations with potential partners and private sector, particularly in FCV contexts (ICR, para 81iii). 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was clear, well written and supported by a logical outline. The ICR was evidence-based and refreshingly candid about the limitations of the project’s design and implementation. This is particularly evident when the ICR discusses unmet assumptions, issues with the indicators, gaps in data related to efficacy, and the M&E section. The ICR could have included more information on how the project attained specific safeguards and the results of the audit findings. The results framework (Annex 1) could have provided more explanatory Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LR-Smallholder Tree Crop Project (P113273) detail, particularly on the indicators that were not met. While the ICR narrative was very honest and upfront about the limitations of the project, the ICR’s ratings did not accurately reflect the narrative. One significant shortcoming is the lack of clarity in the ICR's efficiency section with regards to why the reported EIRR, NPV, and BCR differs from the analysis conducted in the Impact Evaluation Report. Overall, this ICR is rated as Substantial. a. Quality of ICR Rating Substantial Page 21 of 21