ICRR 11907 Report Number : ICRR11907 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 09/21/2004 PROJ ID : P007700 Appraisal Actual Project Name : Mx Community Forestry Project Costs 23.57 18.44 US$M ) (US$M) Country : Mexico Loan /Credit (US$M) Loan/ US$M ) 15.0 13.12 Sector (s): Board: RDV - Forestry Cofinancing -- -- (42%), Agricultural US$M ) (US$M) extension and research (29%), Central government administration (29%) L/C Number : L4137 Board Approval 97 FY ) (FY) Partners involved : -- Closing Date 06/30/2002 12/31/2003 Prepared by : Reviewed by : Group Manager : Group : Soniya Carvalho Nalini B. Kumar Alain A. Barbu OEDSG 2. Project Objectives and Components a. Objectives The overall objective of the project is to support the implementation of key elements in the Government's redefined natural resource management strategy . Specific objectives of the project are to : (a) improve natural resource management and conservation by community and ejido forestry resource owners; and (b) increase the range of forestry-based income generating options available to them . b. Components Assistance to Communities and Ejidos (Appraisal: US $10.25 million; Actual: US $12.22 million). Was to finance promotional activities, private technical assistance, short and long term training, and community -to-community training. Strengthening of the Capacity of Private Service Providers (Appraisal: US $1.61 million; Actual: US $0.59 million). Was to finance, inter alia, training necessary to enable private sector providers to work with communities and ejidos in sustainable integrated forest management practices, multiple -use forestry etc. Promotion of Non-timber Forest Products (Appraisal: US $3.22 million; Actual: US $1.46 million). Was to finance diagnostic studies, grants for purchase of tools, materials, equipment, temporary civil works, technical assistance etc . and salaries, consultants, equipment, materials and publications for the evaluation and dissemination of results . Institutional Strengthening (Appraisal: US $8.49 million; Actual: US $4.17 million). Was to finance consultants, salaries, studies, equipment, training programs, vehicles, travel -related expenditures, and workshops to strengthen national and state forestry agencies to carry out their mandated forestry activities, and to manage and implement the project. c. Comments on Project Cost, Financing and Dates Actual total project cost was about US $ 5 million lower than estimated at appraisal . The government counterpart contribution was estimated at US $ 8.6 million at appraisal while the actual contribution was US $ 4.73 million (including beneficiary contributions ). There was a one-and-a-half year delay in project closing, partly because of reduced availability of timely funds . 3. Achievement of Relevant Objectives: Based on the (weak) evidence in the ICR, the project appears to have achieved its objectives but with significant underachievement of the targets set at appraisal (ICR Annex I). With regard to the first objective, the ICR states that the project increased the forest area under improved management by 271, 731 hectares (but there is no discussion of the extent and nature of improvement ). The ICR notes that Mexican timber is facing an increasingly competitive world market and the key to successful competition lies in the development, inter alia, of strategic alliances with the private sector (ICR page 14). But the ICR also notes that the historical experience with private and parastatal companies has left many communities with a reluctance to engage in partnership with the private sector (ICR page 21), something that was confirmed in subsequent discussions with the Region . With regard to the second objective, studies were carried out by private service providers, NGOs, or consulting companies on the potential of investment projects and technical assistance was provided to ejidos and communities on the feasibility of such investments which provided them forest-based investment options. At the time of the mid-term review, however, there was concern regarding underutilization of the results of the technical studies and frustration among some communities and ejidos because of lack of investment funds to develop non -timber forest products (ICR page 9). The ICR page 5 states that the identification of income options for the communities and ejidos by the project facilitated communities to seek additional resources from other federal, state, and municipal programs as well as to invest their own resources. An estimated 90.5 million pesos were leveraged by the project through grants from other agencies to carry out investment projects . 4. Significant Outcomes/Impacts: 1. The project helped shape legislation --an international seminar on community forestry sponsored by the project, contributed significantly to shaping national forestry policy and particularly the National Forestry Law passed by Congress in December 2002 (ICR page 10). 2. The ICR page 5 states that the identification of income options for the communities and ejidos by the project facilitated communities to seek additional resources from other federal, state, and municipal programs as well as to invest their own resources. An estimated 90.5 million pesos were leveraged by the project through grants from other agencies to carry out investment projects . 3. The project induced adoption of some of its methodologies and instruments in other National Forest Commission (CONAFOR) programs (ICR page 17). 4. The project financed evaluations that led to the certification of some 147,314 hectares. According to the ICR, certification has made it possible to improve the relationships among forest producers, civil society, and the government and to provide recognition of the communities' and ejidos' capacity to manage their resources (ICR page 9). 5. The project helped to develop the National Forest Information System (SNIF) which provides information on forest inventories, data on plantations, information on forest fire prevention, pest control, legislation etc . 5. Significant Shortcomings (including non-compliance with safeguard policies): 1. There was a significant underachievement of the targets set at appraisal (ICR Annex 1). 2. There was little monitoring of the quality of the training, technical assistance, and capacity building activities of the project. Hence it is not clear how much capacity was built . 3. The roster of private service providers was somewhat skewed by discipline and location (ICR page 8). Business and financial management skills remained particularly weak (ICR page 8). 4. There were concerns at the mid -term review about the utilization of the results of the studies financed by the project. 5. Some communities indicated that the procedures for accessing technical assistance through the project are excessively bureaucratic and time -consuming (ICR page 17). 6. Ratings : ICR OED Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Satisfactory OED rates a project's outcome as moderately satisfactory [a rating which does not exist under the ICR's 4-point rating scale] when it achieves most of its major relevant objectives but with significant shortcomings (in this case, a significant underachievement of targets set at appraisal and some other shortcomings all of which taken together add-up to significant shortcomings, see Sections 3 and 5 above). Institutional Dev .: High Substantial Substantial at the central government and community levels, but modest at the state government level. According to the Region, the ICR's modest rating on page 32 refers to state government level institutional development. Sustainability : Highly Likely Likely While the follow-on project will continue assistance to some of the 300 original communities, underutilization of the results of the technical studies (noted during the mid-term review) if not effectively addressed and constraints on markets for forest products may affect sustainability. Bank Performance : Satisfactory Satisfactory Satisfactory overall but with shortfalls . Despite the PAD's recognition that "the project could have a potentially negative environmental impact..." (ICR page 33), the ICR did not report on environmental and other safeguards although the Region has subsequently provided information on compliance. M&E of training and technical assistance activities was overly numeric (e.g., number of communities assisted) and severely deficient in not assessing the quality and impact of training/technical assistance--there were no beneficiary assessments, focus groups, or key informant interviews . A stronger business orientation was needed in Bank supervision (ICR page 18). Borrower Perf .: Satisfactory Satisfactory Quality of ICR : Unsatisfactory NOTE ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness. NOTE: 7. Lessons of Broad Applicability: 1. In projects involving training, technical assistance, and other forms of capacity building, monitoring and evaluation must include quantitative measurement but also in -depth qualitative assessment . 2. Projects aimed at enhancing income -generating options cannot afford to ignore the larger economic context in which project activities take place, especially access to markets . Otherwise, any improvements in income would only be short-term. 8. Assessment Recommended? Yes No Why? To verify the ratings. 9. Comments on Quality of ICR: ICR quality is rated unsatisfactory for a number of reasons : The ICR does not report on the extent of compliance with safeguard policies . A substantial proportion of project cost (US $10.3 million out of US $13.12) financed capacity-building activities for communities and ejidos (ICR page 26). Yet, the ICR provides little assessment of the quality of these capacity-building activities. The ICR does not adequately substantiate its conclusions about project performance, for example, it states that the project contributed to building up private service providers' technical skills (ICR page 5), but provides no evidence. The ICR also states that the project "contributed to strengthening social organization and governance of ejido and forest communities" (ICR page 17), but again does not provide evidence to back -up this assertion. Despite noting that at the time of the mid -term review, there was concern regarding underutilization of the results of the technical studies (ICR page 9), the ICR does not asses the extent of utilization of studies produced under the project and report progress since the mid -term review. The evidence in the ICR raises doubts about the reliability of the internal rates of return (IRR) calculations. For example, ICR page 14 mentions IRRs of 25-26 percent for pine resin but ICR page 22 says of pine resin that "although the positive impact is generally recognized, only one case has so far demonstrated positive effects on employment and income generation". Also, the relatively low IRR for the community (Pueblos Mancomunados) where production technology was more advanced compared to the other communities (ICR pages 27 and 28) needed more explanation. There are discrepancies in the ICR : (i) different total (actual) project cost figures are provided on page 16 and 25; (ii) page 1 mentions the original closing date as 06/30/2002 whereas page 15 mentions the project period as going only upto 2001 and page 10 talks about a one year delay . Page 32 of the ICR rates the achievement of the institutional development and public sector management objectives as modest but does not indicate that these refer only to state government level as subsequently clarified by the Region. The summary institutional development rating on page 2 of the ICR is high. Page 15 of the ICR notes that the shortage of budget funds as well as problems with their timely release were factors in explaining the relatively slow pace of development of the activities and the low rate of disbursements but the Region subsequently pointed out that this statement was not accurate since the delays did not affect project implementation (given that Bank funds are not additional in Mexico ).