ICRR 14943 Report Number : ICRR14943 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/22/2016 Country : Panama Project ID : P064918 Appraisal Actual Project Name : Rural Productivity US$M): Project Costs (US$ 46.90 38.72 Project L/C Number : Loan/ US$M): Loan /Credit (US$ 39.40 38.80 Sector Board : Agriculture and Rural US$ Cofinancing (US $M): Development Cofinanciers : Board Approval Date : 03/21/2007 Closing Date : 01/31/2013 01/27/2015 Sector (s): Agro-industry; marketing; and trade (47%); General agriculture; fishing and forestry sector (33%); Central government administration (10%); Agricultural extension and research (10%) Theme (s): Rural policies and institutions (25%); Biodiversity (25%); Other rural development (24%); Rural services and infrastructure (13%); Rural non-farm income generation (13%) Prepared by : Reviewed by : ICR Review Group : Coordinator : Ridley Nelson Christopher David Christopher David IEGPS1 Nelson Nelson 2. Project Objectives and Components: a. Objectives: The objective of the Rural Productivity Project (PRORURAL), as specified in the Loan Agreement , was: to contribute to increased productivity among organized rural small -scale producers of the Borrower 's territory through their participation in productive alliances while ensuring the sustainable use of natural resources and the conservation of globally important biodiversity . In parallel to this project, starting nearly a year earlier due to processing differences , there was a GEF project, the Rural Productivity and Consolidation of the Atlantic Mesomerican Biological Corridor Project (CBMAP II), for which the Global Environmental Objectives were : to conserve globally important biodiversity and protect associated forest , mountain, coastal and marine ecosystems in the Recipient 's territory by: (a) improving the effective management of SINAP (National System of Protected Areas ) at the national, provincial, Comarca and district levels; and (b) supporting investments in natural resource management and productive opportunities for CBOs ( Community-based Organizations) of the Project Area. This was originally intended as a blended project , but, due to delays with PRORURAL and for other sequencing reasons , it became effective earlier. The GEF CBMAP II project was approved in June 2006 and the Bank-funded PRORURAL in March 2007. Given the substantial overlap , and the fact that both projects were treated together in the same Implementation Completion Report, this ICR Review covers them together . The text will be the same in each of the two ICR Reviews , but IEG has produced two ICRRs each with a different Section 1 and with some small differences in ratings , PRORURAL and CBMAP II are reported on under separate headings within each section except in a few sections where there are insignificant differences . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: PRORURAL Component 1: Support for Productive Alliances (Appraisal (all costs in this section are taken from the ICR cost table that excludes contingencies ) US$6.90 million; Actual US$5.97 million). This was to finance the preparation of productive alliance proposals , business plans and investment subprojects . It included a communication strategy and training for business skills for small -scale producers, technical service providers , technical studies and consultancies supporting business plan execution . Component 2: Productive Alliances (Appraisal US$24.00 million; Actual US$19.16 million). This was to finance about 70 subprojects (up to a maximum of US $500,000 each) implemented by Rural Producer Associations in three Provinces. The financing included working capital and Technical Assistance with a 10% contribution from the Associations. Component 3: Environmental Investments and Institutional Strengthening (Appraisal US$10.50 million; Actual US$10.50 million). This component was executed by the National Environmental Authority (ANAM) under the partially blended GEF Project. This was to finance matching grants for 450 Small-Scale Environmental Investments proposed and implemented by communities and Producer Associations . It covered 14 Protected Areas and buffer zones . The investments included natural resource management and productive activities with an impact on biodiversity . Funding included training, TA and mobilization. Beneficiaries were to contribute a minimum of 10% in cash or in kind. Component 4; Project Management , Monitoring and Evaluation (Appraisal US$2.90 million; Actual US$3.19 million). This financed the incremental operating costs of the Ministry of Agricultural Development to execute the project, including M&E. CBMAP II Component 1: Community Investments in Environmental Resources ( Appraisal US$12.10 million; Actual US$9.69 million). This financed investments proposed by rural community associations and Producer Organizations in targeted Protected Areas and buffer zones to improve the management and conservation of natural resources . It included matching grants for 450 demand-driven investments in small-scale infrastructure screened for their contribution to conserving biodiversity . It also financed support for the promotion of conservation and the sustainable use of natural resources. Component 2: Management of Natural Resources and Strengthening of SINAP (Note that SINAP is not an organization itself but is the inter -connected, inter-agency national system of protected areas ) (Appraisal US$2.70 million; Actual US$1.48 million). This supported the integration of social and environmental sustainability into development and poverty oriented strategies . It included participatory involvement in protected area management , promoted co-management and information systems . It also supported approaches to the development of alternative financing sources. Component 3: Monitoring , Evaluation and Project Management (Appraisal US$2.50 million; US$8.57 million). This component was to improve the Environmental Protection Agency capacity to monitor the national system of protected areas and to evaluate biodiversity conservation . It included the monitoring and evaluation activities and a national biodiversity monitoring system . It financed the coordination , planning and supervision elements of project management. There were some changes made to the above components in both projects . In particular, following the MTR, under PRORURAL the maximum subproject financing ceiling was reduced to US $250,000 due to weak absorptive capacity , allowing a doubling of subproject numbers . Under CBMAP II, the number of environmental investments was reduced from 450 to 350 due to inadequate funds to implement the larger number . There was also some refocusing of subproject funding to support other important protected areas . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Costs PRORURAL total project cost planned was US $46.90 million and actual was US$38.72 million. CBMAPII total project cost planned was US $18.10 million and actual was US$19.74 million. Financing PRORURAL IBRD funded US$39.40 million at appraisal and US$38.80 million equivalent actual. In US$ terms 98.48% of the loan was disbursed. Financing CBMAP II GEF financed US$6.00 million appraisal and actual. 100% was disbursed. In 2013, the Bank launched an INT investigation into allegations of fraud and collusion (ICR page 16). The findings cited poor record -keeping in procurement, and reported that there was a need to provide information to bidders on how they or other concerned parties could report suspected fraud . Measures taken to address the findings included training , more frequent ex post reviews, and the use of Panama Compra under the new GEF project to reduce the risk of subsidiaries under the same company competing for the same contract . Borrower Contribution PRORURAL The borrower contribution for PRORURAL was US $1.90 million at appraisal and US$1.09 million actual, mainly due to budget constraints. One of the main project financing issues was that slow disbursement in PRORURAL in 2011 for reasons other than reduced need resulted in further budget cuts from the center the following period . At the center, this was motivated by the need to comply with the annual fiscal deficit ceilings and a Fiscal and Social Responsibility Law (ICR page 10). This resulted in a budget cut of 98% for Component 2 subprojects. Under PRORURAL local farmer organizations contributed about 50% more than planned. This included contribution of land and labor. Borrower Contribution CBMAP II The borrower contribution for CBMAP II was US $1.20 million planned and US$4.10 million actual, over three times the appraisal estimate. The ICR notes several further additions not included in this figure , a US $2.42 million contribution for reforestation by the National Environmental Authority Investment Fund and US $0.69 million additional of government operating funds . Dates PRORURAL Under PRORURAL the closing date was extended twice , first for 18 months to enable the successful conclusion of subprojects and their associated monitoring and evaluation and the second time for six months to January 2015 to secure US$2.6 million of urgently needed additional resources at the time of renewed budget cuts to the project . This enabled completion of planned subprojects . Dates CBMAP II Under CBMAP II, there were three restructurings , in July 2012, May 2013, and June 2014, about one month before closing. The project was effectively one year ahead of PRORURAL but closed in July 2014 somewhat ahead of PRORURAL. One of the reasons for the delays in both projects (ICR, page 10) was that the projects were implemented over the periods of two government administrations , the change coming in 2009. Partly for this reason there were five project coordinators and a high turnover of staff . This was not helped by similar staff turnover on the Bank 's side. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Rated Substantial for both projects. With respect to both PRORURAL and CBMAP II, the Panama Country Partnership Framework of 2015 focused on three important areas related to the two projects : (i) the expansion of opportunities for the rural poor including indigenous people; (ii) increasing agricultural productivity ; and, (iii) the conservation of biodiversity . There was also a government strategic plan for the agriculture sector , a national biodiversity policy , and a national climate change policy that reflected similar goals . The objectives were consistent with the government 's global commitments under the Mesoamerican Biological Corridor agreement and the UN Framework on Climate Change . The objectives were consistent with the Bank 's country strategy of focusing on the eradication of extreme poverty and shared prosperity through sustainable livelihoods . The objectives were also consistent with the targets for the Convention on Biological Diversity . b. Relevance of Design: PRORURAL . Rated Substantial The design of PRORURAL was consistent with current state -of-the-art approaches aimed at bringing small farmers into commercial production and supporting value chain linkages . However, as noted by the ICR, (page 17), there was limited focus on the assessment of demand and the impact that demand patterns would have on the commercial linkages. The results framework was sufficiently outcome oriented and the internal logic linking training , technical support, business plans, productive alliances, and biodiversity support with the net revenue , sales, and reduced pressure on biodiversity outcomes indicators , as well as the supporting intermediate outcome indicators was largely sound apart from the demand element noted. CBMAP II . Rated Modest The design of CBMAP II was less satisfactory . The project design was consistent with GEF guidelines and a longer term objective of biodiversity conservation and there was generally sound gap analysis . It focused adequately on people/parks conflict issues and the resolution of different interests . It included alternative income sources to reduce pressure on protected areas . The focus on strengthening the management of the SINAP system was sound and the focus on moving towards financial independence of protected areas was appropriate and necessary both for people/parks conflict and for sustainability . However there were several weaknesses . First, funds provided for reforestation fell far short of the need to achieve the objective of that activity . Based on communication with the project team , it is still unclear why there was this substantial miscalculation. Second, there were also weaknesses in clarifying institutional responsibilities in the case of reforestation. Third, there was a weakness in the design with regards to the consolidation of the human capital development of the previous project . This seems to have been a missed opportunity to support capacity sustainability. 4. Achievement of Objectives (Efficacy): PRORURAL As noted above, the objective of PRORURAL was to contribute to increased productivity among organized rural small-scale producers of the Borrower 's territory through their participation in productive alliances while ensuring the sustainable use of natural resources and the conservation of globally important biodiversity . There are two elements to this objective assessed separately here . Increase productivity of rural producers through productive alliances . Rated Modest partly due to lack of evidence of increased productivity. Outputs 130 productive alliances for small-scale producers were formed (186% of target) in three provinces. 4,577 small-scale producers benefited from productive alliances . 80% of the productive alliances were still operating one year after receiving project support (the target for this was 75%). But this is a very modest goal to be still operating only one year after a substantial injection of funding that included a large share of operating costs . There was an increase of 54% in new productive alliance members (against a 20% target), nearly 3 times the target. The extent to which this is indicative of success is unclear since there was little up -front investment required by members. However, the Bank project team reports that , while the costs of joining were not prohibitive and free resources were an incentive , there was also a growing incentive of potential members seeing the benefits "of changing their productive culture to one stressing mutual benefits of collective action ". Outcomes and Intermediate Outcomes Beneficiary consultations found that being a member of a productive alliance increased incomes , improved quality of life, resulted in higher yields, offered new markets, and gave better prices through collective bargaining. However, there is limited quantitative evidence provided in the ICR for some of these claims . In a sample of 2,439 producers, 43% using project support showed a net profit compared to 33% of producers without project support (but see the note on methodology below ). Average net returns of the with -project producers increased from US $698 per year to US$1,180 a year, giving a 69% increase. The most profitable crops reportedly were milk , plantain, fish, corn and beans, but for some of these enterprises there were modest returns . According to the project team, the majority of production was individual but marketing was done increasingly through the associations . Collective production was often resisted, collective marketing and processing was the main aim . There was a 22.3% increase in sales receipts (90% of the target) by productive alliance members. Sales receipts were considered by the ICR to be a substitute for net revenue , the latter being hard to obtain from small farmers with limited record keeping. While this is true, measuring only sales omits the cost element of net revenue so it says little about net revenue after costs . Also, it is not clear what the sales receipt baseline was . The project team noted that alliances were not always the formal constructs that the Bank had originally envisaged. They were, in many cases, quite loose and not necessarily durable arrangements put together to qualify for subproject investment . The project team reported benefits in terms of : increased negotiating power; better opportunity to store and sell at the right time; professional processing; meeting phyto-sanitary requirements; and, better marketing linkages through higher tier association . In addition to sales through the producer associations , 58% of beneficiaries sold outside the alliances through commercial intermediaries. Since these sales are not recorded in the sales data reported above , the ICR notes that the alliance sales underestimate the marketing benefits . However, it is not clear from the ICR what the attribution of these unrecorded sales to the project investments was . It is very difficult to interpret the Beneficiary Survey evidence (ICR pages 67 to 70) for PRORURAL. The survey reports numbers of producers "with profits" and "without profits". There are a number of uncertainties about interpretation of these findings : (i) It is not clear from the ICR how "profit" is defined, whether it is total farm profit in terms of total costs deducted from total revenue or some other derivation ; (ii) It is not clear whether the average profit per producer includes the "losses" of the majority 1,417 deducted from the "profits" of the minority 1,076; (iii) It is not clear what the without project data refers to but it seems to be the before project situation rather than based on any control group of non-participants which would yield a "without project" comparator; (iv) It is not clear how the majority 57% who have no profits (and therefore presumably mostly have losses with some breaking even ) sustained their losses over time; (v) It is not clear whether changes in the value of home consumption were part of the assessment ; (vi) It appears that out of 2,493 sampled, only 255 producers (about 10%) shifted from making losses to making "profits". It is difficult to reconcile this with the 69% change in average profit per producer if this includes all producers. Thus, depending on interpretation, benefits seem only modest and this raises questions about sustainability discussed later . (Note that this productivity objective was not a CBMAP II objective whereas the next objective , especially the global biodiversity element, was largely common to both projects .) Ensure sustainable use of natural resources and conservation of globally important biodiversity . Rated Modest Outputs About 8% of the protected area of 3,781 hectares was transformed into forest /vegetation for sustainable use replacing previously (reportedly) unsustainable grazing of cattle and cropping in forested areas . (The target was 10%). How unsustainable the grazing and cropping was is unclear from the ICR . However, the project team reported that it included extensive cattle raising that included land clearing , burning off, and soil usage inconsistent with water conservation and erosion control , also unwise use of pesticides and fertilizer . The project alternative of improved practices was : reconstituting vegetation, terracing, no-till, and inter-planting. Outcomes and Intermediate Outcomes The ICR reports (page 19) that percentages of producers ranging from 47% to 100% by association adopted at least some improved farming practices supportive of sustainable resource management . These included avoiding burning, agroforestry systems, windbreaks, crop diversification, organic fertilizer, zero tillage, waste management, and terracing. However, the environmental impact of these on water , erosion, carbon storage or productivity was not measured , it is largely assumed. It is unclear whether these activities will be sustained in the absence of project funding . Some alliances were formed for the sake of getting project funds and may not be sustained in the absence of further support . There were a number of activities under PRORURAL that were blended with CBMAP II activities including reforestation or recovery on 1,900 hectares. These are reported to have generated carbon storage (see carbon value estimated below) and improved water infiltration alongside economic benefits . Quantitative evidence is presented for carbon but based on assumptions from other studies but not for water infiltration although hydrological benefits are plausible . Beneficiaries surveyed are reported to have become more positive and committed about conservation and more positive about the techniques learned . In terms of biodiversity impact, the ICR notes (page 19) that, in the context of globally important biodiversity these were small investments with predominantly local impact . CBMAP II The objective of PRORURAL was productivity and income with additional benefits for biodiversity and the environment whereas the objective of CBMAP II was mainly biodiversity and environmental with some additional benefits and incentives in ensuring productivity . As noted above, the Global Environmental Objective was (simplified): to conserve globally important biodiversity and protect ... ecosystems by improving the effective management of protected areas and by supporting investments in natural resource management and productive opportunities by community -based organizations. As in the ICR, the CBMAP II objective is split here into three elements : (i) conserving globally important biodiversity and protecting ecosystems ; (ii) improving the effectiveness of management of SINAP (the the national system of protected areas); and, (iii) supporting investments in natural resource management and productive opportunities for community organizations. Conserving globally important biodiversity and protecting ecosystems . Rated Modest Outputs 43,033 hectares of forest and related ecosystems are reported to have been reforested or restored and placed under sustainable management. This was 86% of the target of 50,000 ha. However, there is an attribution question about this achievement because it was not associated with the environmental investments and , according to the ICR, (page 19), would have been impossible to achieve even if it was because the cost greatly exceeded project funding. The high number of hectares was achieved by counting 41,076 ha of self- financed reforestation activities in buffer zones carried out by the National Environmental Authority . However, the ICR notes that the National Environmental Authority (ANAM) had to use other budgetary resources to leverage the project funding. The project team notes that there was considerable debate about this during the preparation of the ICR. The borrower was adamant that this should be included as a valid project benefit . The argument appears to be that this can be seen as a project achievement because the high project target , well beyond the funding provided, put pressure on the authority to do this reforestation . IEG's assessment is that this is only partly plausible and therefore the afforestation can only be considered partially attributable , a notional figure might be one third attributable. Using resources partly from the United Nations Program for Reducing Emissions from Deforestation and Degradation (UN-REDD) and FAO, the National Environmental Authority prepared a high resolution forest and soil use map giving up-to-date forest area in all protected areas . The ICR notes that this was extremely important to focus policy and institutions on environmental management and conservation . It showed a significant gain of forest//vegetative cover to about 62% (in 2013) over 65 protected areas compared to 45% in the year 2000. Outcomes and Intermediate Outcomes The ICR was not able to obtain direct evidence from the data for the 14 individual project protected areas supported. This seems to suggest weak data disaggregation from the study , since the 14 were among the total of 65 surveyed so the data must exist . The ICR team reported that they tried to get the data but failed . Nevertheless, the ICR argues (page 20) that the evidence strongly suggests that across the 14 protected areas there was improved management and also improved biodiversity protection in six of them covering some 248,000 hectares. The project team reports without quantification that there was an intensive study in 4 PAs prioritized by CBMAPII which found a number of new species and locations of species thought no longer viable . These were interpreted as early signs of improved management . With the support of sophisticated equipment for assessing biodiversity under the project and the National System for Information and Monitoring of Biological Diversity (SNIMDB), numerous species of conservation interest were documented in protected areas in general i .e. not specifically within the 14 areas supported. This included new species, species located beyond their previously known range , and indicator species showing effective conservation. The ICR notes (page 20) that "Some 6,758 species of flora and fauna were registered on now permanent biological monitoring lots in these (four) protected areas". While this does not show change in the absence of a baseline , the ICR argues that this project -supported database has far -reaching implications for the future as an important biodiversity database . Despite this being a second phase project , there is very little quantitative evidence of indicators of improved biodiversity in the particular PAs supported by the project . Improving the effectiveness of management of SINAP . Rated on balance Substantial . Outputs The ICR reports that co-management of protected areas was fostered by establishing 4 Municipal Environmental Units and training staff and consultants in 25 districts. The National Protected Areas System (SINAP) was strengthened through the preparation of Municipal Environmental Land Use Plans. These were ratified in 15 municipalities which represented about 60% of the project area districts (100% of the target). The aim was to integrate environmental management into local government practice. The plans were prepared with the help of participatory diagnostic workshops . There was also training for 754 representatives in the principles of environmental co -management. Three potable water sub-projects were executed as pilots in three locations with the support of the Environmental Management Units. 12 co-management agreements between the National Environment Authority and communities are under implementation in ecotourism, protection of endangered species , resource conservation, mangrove recovery, and agro-forestry. There are seven more under consideration . Outcomes and Intermediate Outcomes Management capacity achievement has been demonstrated by the WWF /World Bank Management Effectiveness Tracking Tool , an environmental management rating system that includes legal status , existence of operational plans, availability of resources, management systems, goods and services, and the effects of management in relation to conservation . However, it is not clear how this last coefficient on the conservation effects was rated given the lack of individual PA data on biodiversity changes . The ICR reports (page 21) that the average performance rated value for 13 of the project Protected Areas increased from 44.8 points in 2005 to 69.0 points in 2013. The ICR attributes much of this progress to a strong training effort reported in some detail in ICR Annex 2. The Forest, and Land Use Map noted above reportedly strengthened capacity to manage and monitor biodiversity changes and for the future to make strategic and policy decisions . Two studies were supported by the project . (i) a plan to establish fideicomiso to support PA management, the financing of investments, monitoring, and the exploration of the views of private sector firms located in buffer zones; (ii) a study of the financing needs of the national protected areas system . The latter is expected to be a valuable input to the follow-on GEF project. Supporting investments in natural resource management and productive opportunities for community organizations . Rated Modest Outputs Under both PRORURAL and CBMAP II some 350 investments were made (100% of the target) benefiting 10,760 beneficiaries directly and another estimated 40,000 indirectly. aimed at reducing pressures on protected areas mainly focussed on buffer zone areas . 42% of the investments were in indigenous communities . These included handicrafts, agro-forestry, plant nurseries, eco-tourism, and organic agriculture. The performance evidence is unclear. A sample of 147 of these found that the net increase over the average income of sampled households was between 12% and 53%, the lower values reportedly being where incomes were already higher . But, as with PRORURAL, it is not clear what the productivity impact was. In the case of PRORURAL many sub-projects gave very modest returns and although they may be positive for the environment may not be sustained unless returns can be substantially raised beyond what was found in the PRORURAL assessment .. The ICR claims greater environmental awareness through training , demonstrations, technical assistance and linkages to productive benefits . This is plausible but the evidence presented is limited . Outcomes and Intermediate Outcomes The 43,000 hectares of forest regeneration were calculated to give about $300,000 a year in carbon fixation value. As noted earlier, supported by both projects , there are soil management improvements reported by the ICR in agro-forestry, no burn practices, waste management, crop diversification, zero tillage (96% adopting), terracing, and other potentially beneficial practices . However, while such practices may be potentially beneficial to the soil and to biodiversity these impacts do not appear to have been measured even in selected cases to assess whether the anticipated potential was realized . (To take one example, zero tillage potentially can contribute to improved surface soil organic matter and reduced erosion but it is a difficult technique requiring good management and in some soils and with poor equipment or poor spray or fertilizer application it can fail to deliver in terms of yield and soil conservation . Only measurement can show whether it succeeded .) The ICR reports, again qualitatively (page 22), that in a rapid assessment, covering 50% of the sub-project beneficiaries, there were improvements in conservation practices "in all areas, depending on the conditions and soil units involved". The ICR also reports "environmental investments were not only an effective medium for improving family well-being but also a viable option for integrating producers within buffer zones in their conservation. 5. Efficiency: PRORURAL . Rated Modest The PAD analyzed three pilot productive alliance models finding , ex ante, a range of IRRs from 24% to 34%. For the ex-post analysis, according to the ICR, about US$21 million was spent on 130 sub-projects (Dairy or Cattle 20 subprojects, Maize 15 , Tubers 19, Artisanal Fishing 15, Pulses 14, Other 47). The number of families benefited was 4,577 giving an average cost per family of US $4,563. By global standards this appears to be a fairly high investment per household. However, the project team argues that it is lower than similar projects in Colombia , Bolivia, Honduras, and Nicaragua. The teams notes that the amount includes working capital (65% of the total) which is to be repaid to MIDA although the evidence suggests there have been some problems with repayment . A small random sample of 12 subprojects out of 111 was selected by the ICR for investigation stratified across six productive activities. As the ICR notes, these may not be representative . There were limitations in the number of sub-projects that could be usefully sampled because a few of them had only been operational for about one year at project closing leaving little chance to demonstrate measurable benefits . The average size of sample sub -projects was US$223,905 which was close to the average for all sub - projects suggesting that they were at least representative for size. 73% of the costs were funded by the project and 27% by association contributions . Participatory workshops were held with producer groups to supplement gathered data . Most projects had been operating for between two and four years giving limited time to reach full benefit stream and so requiring a substantial element of cost and benefit projections . Of the 8 out of 11 sub-projects with positive financial indicators only three had net present values per family in excess of US$150. As noted by the ICR (page 47) this is a fairly low return considering the risks of agriculture enterprises. Five were not expected to recuperate investments within the 10 year period of analysis, the other six were expected to recuperate investments in around seven years . The average internal rate of return for all the sampled sub-projects was only 11%. 8 of the sub-projects averaged 12% IRR or higher but three had negative rates of return. Overall the net present value of the sampled sub -projects was negative by an average of about US $36,000, leaving a negative net present value per family . With respect to sensitivity , only 2 out of 8 sub-projects with positive financial indicators could withstand an increase in costs or decrease in revenue of over 5% and only one over 10%. These results suggest very modest and also risky returns . As noted in Section 4, there are also questions about sustainability of any productivity gains in the absence of project funding . For sensitivity, the ICR also carried out a Monte Carlo simulation (randomizing risk variables over a distribution ) looking at net present value risk using a normal risk distribution and running 5,000 iterations. Based on this alternative sensitivity methodology another two additional sub -projects would be deemed not viable due to excessive risk. The main issues with the financially non -viable sub-projects were lower yields, lower prices, and smaller scale of production in relation to their original business plans . The ICR concludes that only 6 out of 12 sub-projects in the sample proved financially viable . However, the models suggest that several of these would be unable to face quite small adverse impacts. The ICR analysis did not aggregate these modest returns and apply project cost overheads for an aggregate whole project rate of return calculation but clearly it would be very low without the addition of large unquantified benefits . CBMAP II . Rated Modest Unlike PRORURAL, the efficiency analysis in the ICR for CBMAP II is difficult to follow being based on a study that had largely other purposes . The ICR states that the financial analysis of this project could not compare results with the PAD analysis because , at the time of the PAD, no environmental sub-projects had been identified leaving no basis for comparison . This CBMAPII project was analyzed by taking a sample of 42% of the Environmental Investments (EI) which constituted 146 EIs. Some had to be dropped for various reasons . Three main models were identified: silvo-pastoral models, organic production models, and models based on tourism and handicrafts . Beyond productivity, two additional environmental externalities were included and valued and projected over a five year period: carbon fixation and water infiltration in reforested areas . For carbon, the value in carbon markets was used and the methodology adopted was the standard Intergovernmental Panel on Climate Change approach . For this, the methodology is reasonably well established even if the values per ton of carbon are open to debate over a wide range. For the conservation costs per cubic meter in the areas of water recharge , establishing a methodology is more difficult. In the analysis reported in the ICR , it is unclear what the evidence is for volumes of water recharge . (With afforestation, since trees, while generally reducing run-off peaks and troughs, also move water to the atmosphere through transpiration, the net impacts over alternative vegetation are more likely to be dependent on variables such as slope, species, below canopy ground cover , rainfall pattern, soils, and downstream collection structures .) There are a number of uncertainties about methodology and definitions . The CBMAP II efficiency analysis is largely stated in terms of percentage of family income and percentages of cases with positive net present values or net benefits. It presents tables with data on "profit" but, as noted earlier, it is not clear how this is defined . . The average Social NPV by sub -project by region ranges from about US $22,000 to US$250,000. Only a little over half the investments showed positive commercial net present values although it is unclear how commercial and non-commercial benefits were differentiated in the study . 79% of investment cases exhibited a positive net present value including both commercial and household consumption benefits However , 76% had a "positive internal rate of return" from commercial activities. Since positive could apply to a 1% rate of return, this is not useful for an efficiency indicator. It is difficult from this analysis to understand what the average or aggregate rates of return for all cases or for particular regions would have been . As with PRORURAL, no aggregate project net benefit stream including project overhead costs was attempted to show overall project efficiency . The findings for CBMAP II, depending on how they are interpreted , are somewhat more positive on productivity than the findings of PRORURAL. However, since they both financed similar productive investments , it is not clear why this would be so other than a methodology difference . The data and methodology on PRORURAL is the more convincing . However, CBMAP II legitimately adds a carbon value and , less legitimately, because it is supported only by assumptions, hydrological benefits. Finally, there is the efficiency question of the limited attribution (for want of a better estimate we assume one third ) of the afforestation benefits including the carbon benefits from the planting by ANAM as discussed above in Section 4. ERR )/Financial Rate of Return (FRR) a. If available , enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The performance of both PRORURAL and CBMAPII was similar . There were significant shortcomings in both , partly due to weak impact evidence . Relevance of Objectives for both was Substantial , however Relevance of Design for CBMAPII was somewhat weaker and is rated Modest against Substantial for PRORURAL . On Efficacy, there was limited evidence on productivity . There are also concerns about the sustainability of productivity gains in the absence of project funding. There is also little evidence presented on biodiversity impact despite CBMAP being a second project. Both PRORURAL and CBMAPII are therefore rated Modest on all objectives for Efficacy other than the Substantial rating for improvements in SINAP . Efficiency is rated Modest. The returns from the cases studied for PRORURAL are modest and again there are sustainability questions about what can be expected in the absence of project grants and therefore questions about longer term net benefit streams . The efficiency evidence in the ICR based on "profit" assessments is difficult to interpret . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Risk is rated significant for both projects . While for both PRORURAL and CBMAPII there are follow -on projects (ICR page 16), there is uncertainty about the sustainability of the subprojects . There are a number of questions about the sustainability of the Revolving Funds . The original requirement was that, for the majority of subprojects, working capital would be repaid following a single agricultural cycle . The ICR reports (page 26) that, of 130 associations, 61% had fully repaid working capital obligations by the middle of 2015. However this was still only 30% of total working capital outlay. There were a number of agricultural subprojects that , at the time of project closing, had not started repayments. The concern was such that MIDA , who took over the management of PRORURAL , froze the revolving funds in 2014 for a stock taking of fiduciary performance and intends to continue a system of funds recovery and verification of funds release for another 10 years. While the ICR notes (page 27) that this may have been necessary as a prudent measure, it concludes that this could weaken sustainability . There are therefore serious questions about producer association sustainability . There are also some questions about the strength of supporting services , marketing issues, and infrastructure. More positively, producer associations have been expanding membership suggesting that they are seen by farmers to offer benefits. This included collective bargaining over prices and bulk sales , obtaining technical skills, and entrepreneurship fostering new association activities . The project team reports that the PMU has closed down with some members re -absorbed into MIDA and others being re-contracted for the new PRORURAL so that there is some sustained institutional memory to carry forward . For the activities supported by CBMAP II , some separate and some in partnership with PRORURAL , there is some qualitative evidence reported that farmers have started to change their attitudes towards sustaining soils and water supplies into the future, suggesting some internalization of the environmental understanding promulgated by the project. However, CBMAP II activities still face risks since beneficiaries tend to be poorer and live in more inaccessible locations and face greater organizational challenges . The project team notes that, "CBMAPII farmers were exceptionally poor/small, unused to operating collectively , and attrition is inevitable. The grant resources are finite - they end when the project ends - and if farmers know there will be no more , they drop out of the organization ." There are also concerns across both projects about operation and maintenance of the sub -project investments initiated. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: PRORURAL Rated Moderately Satisfactory . The project design adopted a participatory approach that was consistent with other state -of-the art approaches to sustainable production in agriculture and was consistent with the borrower 's strategy as well as the Bank 's. However, there were some weaknesses : (i) the ICR notes that the approach may have give too much emphasis to producer's wishes over more pragmatic criteria dictated partly by budgetary and implementation constraints and community needs; (ii) there was a disconnect (later addressed) between the US$500,000 expenditure ceiling for associations and the small size of the associations ; (iii) there was insufficient provision for linking associations to commercial companies and an over -estimation of the capacity of local technical assistance to provide support for this. CBMAP II Rated Moderately Unsatisfactory . The design that aimed at participatory approaches and co -management of natural resources was not unsound . However, there were a number of significant weaknesses : (i) there was a large underestimation of the costs of reforestation and environmental sub -projects (from exchanges with the project team , it is still unclear how this happened); (ii) the scale of the task of establishing poor associations in protected areas and buffer zones , some in remote locations, was under-estimated despite this being a second project which should have given better knowledge of the social and physical environment ; (iii) there was insufficient use made of existing communities trained under the previous project , possibly due to equity concerns ; (iv) there was inadequate design of reforestation responsibilities; and, (v) there was insufficient attention at entry to both community and investment sustainability issues and to an exit strategy . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: PRORURAL. Rated Moderately Unsatisfactory . The ICR notes (page 27) that reporting was thorough except for reporting on environmental safeguards prior to the MTR. There was strong client mentoring on financial and procurement issues . There was a high and disruptive level of turnover in the TTL leadership that government found difficult to deal with . (The ICR notes that the ICR rating of supervision is Moderately Satisfactory rather than Moderately Unsatisfactory because leadership changes were beyond the control of the task team . However, IEG notes that supervision ratings are not simply ratings of the task team but of the Bank 's role, as a whole, in supervision.) CBMAP II. Rated Moderately Satisfactory Initially supervision was strong with substantial support especially for the sub -project investments. There was also strong support for the SINAP protected areas system in terms of management and ownership . However, the ICR notes that, during implementation, there were changes in the Results Framework which were informal and confusing. Despite greater continuity of leadership than PRORURAL , there was a decline in the quality of reporting in the later period of the project particularly on safeguards . The ICR reports that there was good handling of the INT case reported above under Financing in Section 2d. Quality of Supervision Rating : Moderately Unsatisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: Both projects are rated Moderately Satisfactory . The ICR notes that government support for both projects was variable in the early years , made more difficult by the 2009 change of government with accompanying changes in government agencies . As noted earlier, budget allocations were insufficient in the later years , especially for PRORURAL . This affected sub-project support. However, the macro reasons for this tight budgetary situation were entirely laudable and consistent with the commitment under the national Fiscal and Social Responsibility Law and the associated accepted debt ceilings. In any case both the main government agencies for the two projects , with some difficulty, found ways to supplement the deficit by reallocating unused funds committed elsewhere . In the end, in the case of CBMAP II counterpart funding was high at 341% of the projections but in the case of PRORURAL it fell well short at 57%. The reasons for this difference are not clear . Looking to the future, the government has sustained its commitment to both follow -on projects. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: Both projects are rated Moderately Satisfactory . The coordination unit for PRORURAL had to function with five different coordinators over the project period . This was exacerbated by the changes on the Bank side . There was a process of continual adjustment in the implementing agencies to address issues . Responses included newly contracted specialists , improved technical support at community level, improved M&E capacity, enhanced skills in procurement and financial management , and institutional restructuring and changes in staff responsibilities . There was a challenge in the greater separation of the PRORURAL PMU from MIDA after the moving of the project from the National Environmental Authority . The ICR records the exceptional proactivity of the PMU coordinator in finding funds to complete the project . However, the PMU post-project was rapidly downsized due to budget constraints . The CBMAP II , initially run from the environmental agency , faced weak commitment and problems with the institutional structure until well into the project . This resulted in delays in getting sub -projects established and limited attention to these projects ' later sustainability. As noted in the ICR (page 29) this was partly a design fault due to over-estimation of the capacity of new beneficiary associations to handle the new sub -projects. However, after 2011 this situation changed and ANAM's commitment improved. As noted earlier, ANAM found additional resources to make up the shortfall for meeting afforestation targets . Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The Results Framework for PRORURAL was generally sound with a sufficient level of outcome focus . However, it was challenging for incremental changes measurement in the absence of baseline data , especially on more difficult parameters such as net income . A later shift towards sales receipts and a focus on beneficiary surveys adjusted for some of these problems but still left significant evidence interpretation issues . The CBMAP II GEO were stated in terms of global objectives and the M &E was more challenging than for PRORURAL although the M&E design was built on the experience of the previous project . Monitoring was not only expected to measure impacts on biodiversity but to track data that could feed into the WWF tracking tool . This had been used to establish baselines in 14 protected areas. Bank teams worked with USAID within the GEF Mainstreaming Biodiversity in Production Landscapes and Sectors program to design the M &E framework. M&E design was challenging also for the 50,000 ha of afforestation partly because the intermediate steps in the logical results chain were not well defined . The design included the updating of maps , strengthening of national databases, and monitoring indicator species . b. M&E Implementation: The Results Framework was revised during implementation . The ICR notes that both projects produced inputs for the MTR reports based on M&E data from the MIS systems. The training of associations in M &E and record keeping was uneven. This reduced the quality of data fed into the system . Sample surveys were undertaken and , for PRORURAL, a control group was reconstructed ex post . The participation of the University of Panama and Autonomous University of Chiriqui enabled the collection and analysis of data on biodiversity including data on indicator species in seven protected areas . The WWF Tracking Tool was applied during implementation . There was a weakness in implementation in measuring biodiversity for CBMAPII and for PRORURAL . Despite the monitoring supported, it is still not possible to present data on biodiversity changes in the individual project PAs or even causative output or intermediate outcome changes . It is unclear how the WWF tracking tool was applied in the apparent absence of the individual PA biodiversity data since impact on biodiversity is one of the parameters . c. M&E Utilization: The M&E findings appear to have been used in decision -making by both the borrower and the Bank . IEG found the analysis of the CBMAP II survey data difficult to interpret for the economic analysis but this may be because it had other purposes than overall project efficiency analysis . The M&E data was used quite widely in the ICR and several decisions during implementation appear to have ridden on data obtained through the M &E system, but this did not adequately inform conclusions with regards to the impact of project activities . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: Both projects were rated Category B at the outset . Both projects, after some later amendments, triggered the same safeguard policies: Environmental Assessments OP 4.01, Natural Habitats OP 4.04, Pest Management OP 4.09, Cultural Property OP 4.11, Involuntary Resettlement OP 4.12, Indigenous Peoples OP 4.10, and Forests OP 4.36. The Pest Management and Involuntary Resettlement OPs became applicable through amendments after the appraisal documents. Initially, each producer receiving support had to have an environmental management plan that cost about US $700, This was too burdensome so the law was changed allowing producers to comply with an Environmental Guide to filter out negative impacts. In some protected areas government retained sole rights to manage land but in others co -management was transferred to local authorities . Both projects aimed at positive environmental impact and in both projects there was a substantial training effort on environmental management. In CBMAP II there were a number of contracted environmental specialists . The ICR reports (page 14) a number of pro-environment actions including: a joint collaboration with several donors and partners for the preparation of REDD Readiness supported by the Forest Carbon Partnership Framework . There was also action on some hydro -electric infrastructure with on-site monitoring. Potential resettlement associated with this in the Palo Seco Protected Forest was also addressed . The Bank team monitored other hydro -electric activity in the another protected area. For social safeguards, business plans were screened for potential resettlement aspects . According to the ICR (page 14), sub-projects showed "no evidence of non-compliance". However, in 2012 and sustained into 2013, due to slowness in updating the Indigenous Peoples Plan , the Bank rating for Social Safeguards was downgraded to Moderately Unsatisfactory . This resulted in only four sub -projects in total being supported for indigenous peoples. Social safeguard compliance in CBMAPII was uneven early in the project but improved later following Bank training and involvement of the PMU social safeguards specialist in the formulation of municipal environmental land use plans. There is a considerable amount of additional detail on safeguards in the ICR (pages 14 to 15). b. Fiduciary Compliance: PRORURAL There were some fiduciary compliance weaknesses but over time these were addressed . Ratings were consistently Moderately Satisfactory with some weaknesses that were gradually resolved . The financial management system took time to become fully operational and there were staffing shortages . The ICR reports that agreed action plans were followed up efficiently. There was some delay in financial reports and independent audits . At one point, fidcuciary risk was downgraded to Substantial and financial management supervision was increased to twice yearly . CBMAP II Financial performance was adequate but even by 2010 accounts were still being maintained through Excel . spreadsheets. A planned PENTAGON system was slow to become operational . There were also staffing issues and the overall rating fell to Moderately Unsatisfactory . The situation did not improve much until 2014. Even in 2013 there were still issues with the new PENTAGON system not working well . By 2014 the ratings finally rose to Satisfactory . c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Significant shortcomings due to weak Satisfactory Unsatisfactory impact evidence, weak biodiversity change evidence even at output or intermediate outcome levels, and concerns about sustainability . Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately There were issues with regards to Satisfactory Unsatisfactory leadership changes and supervision that frustrated the government and prevented progress in various aspects of the project. Where there is a split rating (Moderately Satisfactory for PRORURAL QAE and Moderately Unsatisfactory for Supervision ) the Outcome rating becomes the tie-breaker, thus leading to a MU result. Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR offers a number of lessons of which three are given here with some adjustment of language . 1. Whether a Bank loan is blended with a GEF grant aimed at similar objectives is less important than ensuring synergies between the two projects . In this case, it would probably have been easier and less costly in terms of management to have blended the two but the two projects were still able to complement each other in strategy and in implementation. 2. Follow -on operations should build on previous success and , where needed , consolidate that success . This also allows monitoring performance over a sustained period . In this case, selecting entirely new organizations increased the challenge and ignored the need for consolidation of the achievements of the earlier groups while making longer term sustainability monitoring more difficult . . 3. To be used efficiently and sustainably , revolving funds for producer associations need a minimum level of organizational strength and cohesiveness at association level . In this case, there was variability and it was clear that some producer organizations had reached a minimum threshold and some had not and will continue to face sustainability problems. Defining a minimum set of standards or milestones to achieve organizational thresholds may aid producer organization support decisions and ensure better sustainability . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: Strictly there should be separate ICRs for each separate project number . Covering two projects in one, as was done in this ICR, had advantages and disadvantages . It allowed for reporting on the complementarities but it made reporting within the required Bank formats and procedures more complicated . The presentational complications would have been compounded had the two projects diverged more in terms of achievement and ratings . The ICR is thorough and open about issues . It made a serious effort to make up for substantial data limitations . The lessons are thoughtful and the more detailed and country -specific lessons in the ICR are worth referring to in addition to the more generic lessons. The economic analysis for PRORURAL is very thorough (it is rare to see in an ICR the application of a Monte Carlo Simulation to a risk analysis .). The economic analysis for CBMAP II, drawing from a project study that clearly had largely other purposes , is difficult to interpret and to apply to net present values or rates of return. a.Quality of ICR Rating : Satisfactory