Document of The World Bank FOR OFFICIAL USE ONLY Report No.: 20517 PERFORMANCE AUDIT REPORT PAKISTAN NORTHERN RESOURCE MANAGEMENT PROJECT (CREDIT 2512-PAK) June 6, 2000 "We were in a deep slumber and the project has helped us come out of it" - Project beneficiary. Sector and Thematic Studies Group Operations Evaluation Department This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Currency Equivalents Currency Unit = Rupee (Rs) Annual Average Exchange Rate(Rs to US$): 1990 21.4 1991 22.4 1992 24.9 1993 26.0 1994 30.1 1995 30.8 1996 33.5 1997 40.0 Abbreviations and Acronyms AJK Azad Jammu and Kashmir DOA Department of Agriculture DOAH Department of Animal Husbandry ERR Economic rate of return GOAJK Government of Azad Jammu and Kashmir GOP Government of Pakistan ICR Implementation Completion Report IHFDP Integrated Hill Farming Development Project NGO Nongovernmental organization NRMP Northern Resource Management Project NRSP National Rural Support Program OED Operations Evaluation Department P&DD Planning and Development Department PAR Performance Audit Report PWD Public Works Department SDR Special drawing rights SMU Social Mobilization Unit VDC Village Development Committee WDC Women's Development Committee Fiscal Year July 1 to June 30 Director-General, Operations Evaluation Robert Picciotto Director, Operations Evaluation Department Gregory Ingram Manager, Sector and Thematic Evaluations Ridley Nelson (Acting) Task Manager : Ridley Nelson FOR OFFICIAL USE ONLY The World Bank Washington, D.C. 20433 U.S.A. Office of the Director-General Operations Evaluation June 6, 2000 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Performance Audit Report on Pakistan Northern Resource Management Project (Credit 2512-PAK) Attached is the Performance Audit Report (PAR) for the Pakistan Northern Resource Management Project for which a credit in the amount of SDR 20.4 million (US$28.8 million equivalent) was approved on June 15, 1993. The project closed on June 30, 1999, six months behind schedule. The final total disbursed was US$25.6 million equivalent, 89% of the original amount. There was no cofinancing. This audit was carried out in association with an Intensive Learning ICR which included beneficiary assessments. The main objective of the five-year Northern Resource Management Project (NRMP) was to provide for the sustainable and economically efficient use of land resources in one of the country's poorest, most environmentally vulnerable areas through: (a) improving the policy framework within Azad Jammu and Kashmir; (b) restructuring and strengthening the main sectoral institutions; and (c) testing, on a pilot basis, programs whereby communities took management and financial responsibility for planning and implementation of activities for sustainable resource management. Components were: policy reforms, especially reduced subsidies and transfer of input distribution to the private sector; institutional strengthening in agriculture and animal husbandry; strengthening basic technical services, including extension and training; support for community participation activities, including support for women's group formation, in five pilot areas facilitated by a Social Mobilization Unit; and, improvement of 165 kilometers of rural roads. The project was generally well-designed, drawing from the experience of the earlier Integrated Hill Farming Development Project, audited by OED in 1996, and from the experiences of the Aga Khan Rural Support Program in another region, evaluated by OED in 1986, 1989 and 1996. The main objective of the NRMP - providing for the sustainable and economically efficient use of land resources - was largely met. The policy framework was modified. Most planned actions to transfer input production and distribution to the private sector, to eliminate subsidies on inputs, and to re-deploy staff were completed. Institutional strengthening was carried out as planned. Communities were successfully organized in the five pilot areas, and a majority of the Village Development Committees (VDC), and the later-formed Women's Development Committees (WDC), were functioning satisfactorily. Most of the technical support services were provided and planned studies carried out. Road improvement work was substantially delayed, but was largely completed in the end. This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. 2 However, the Bank failed to ensure the application of the Involuntary Resettlement Operational Directive in the roads component, although the mission, in its limited sample, only encountered families who were satisfied that road access had compensated for the loss of some land. Sustainability, has been enhanced by the policy and institutional reforms and carries the greatest weight in the sustainability assessment. However, long term sustainability of the community arrangements is uncertain without continued support. The ICR rated outcome as satisfactory, sustainability likely, institutional development as substantial, Bank performance as satisfactory, and Borrower performance as satisfactory. The audit agrees with these ratings but downgrades the Bank performance rating to unsatisfactory because of lack of compliance with the Bank's safeguard resettlement policy. The project offers six main lessons. First, multiple component rural development projects can work if they focus on efficiency at the village level through community participation backed by efficiency of services through institutional reform and privatization. Second, sustainability can be enhanced by keeping community mobilization to a manageable scale, by planning for the post-project period, by developing NGO capacity to play an increasing role and, now, by the use of the Bank's new project instruments with longer commitments. Third, community-requested training programs on agricultural technologies, delivered within the village, can be a very effective technology transfer mechanism and may be appreciated more by beneficiaries than routine extension visits. Fourth, appraisal reports should analyze project and post-project local funds demand, compare these with recent budget allocations, and explicitly size the project against these findings. Fifth, road construction, even road improvement, frequently involves some measure of land acquisition and therefore falls under the Involuntary Resettlement Operational Directive, calling for careful consideration of the compensation arrangements. Sixth, the absence of a suitable credit line limits the options in community driven development. This raises the importance of addressing credit delivery issues in a pragmatic way. Attachment Contents Principal Ratings........................................................... Preface..............................1m 1. Background .................................................... 2. Findings...................................................... 2 Lessons .......2.........2 Future Directions ..................3 3. Analysis...........................................................4 Relevance.4 Efficacy.......................................................5 Efficiency............................................ ......8 Institutional Development. .........................................9 Bank Performance ................................................9 Borrower Performance.. .................................. ......10 Sustainability................................................... 11 Annex A. Basic Data Sheet................................................. 13 Attachments .. Supplementary Background Information.................................15 2. Comments from the Government...................................... 17 Tables Table 1. Physical Targets............................................... 7 Boxes Box 1. Stakeholder Comments.................... 6 Box 2. Arnila - A Typical Village Development Committee ................... .... 7 This report was prepared by Ridley Nelson (Task Manager) who audited the project in October 1999 in association with an Intensive Learning ICR. William B. H-urlbut edited the report. Helen Phillip provided administrative support. 11 Principal Ratings ICR PAR Outcome Satisfactory Satisfactory Sustainability Likely Likely Institutional Development Substantial Substantial Borrower Performance Satisfactory Satisfactory Bank Performance Satisfactory Unsatisfactory Key Staff Responsible Department Director Division Chief Task Manager Appraisal Paul Isenman Ridwan Ali Guy Motha Midterm S. Ahmed Ridwan Ali Guy Motha Completion John Wall Ridwan Ali Rashed UI Qayyum 111 Preface This is a Performance Audit Report (PAR) for the Pakistan Northern Resource Management Project (NRMP, Cr. 25120-PAK) for which a credit in the amount of SDR 20.4 million (US$28.8 million equivalent) was approved on June 15, 1993. The project closed on June 30, 1999, six months behind schedule. The final total disbursed was SDR 18.3 million (US$25.5 million equivalent, 90% of the original SDR amount. An Implementation Completion Report (ICR) was submitted on 12/15/1999. The report was prepared by the Operations Evaluation Department (OED) based on the Implementation Completion Report, the Staff Appraisal Report, the Development Credit Agreement, and review of Bank files. The project was also discussed with Bank staff, beneficiaries, Village Development Committees (VDCs), donors, consultants, and government staff. This audit was carried out in association with an Intensive Learning ICR. However, OED was unable to join the stakeholder workshops as originally planned due to travel delay following the change in government in October 1999. The findings of the stakeholder workshops have been used in this report. The cooperation and assistance of all stakeholders and government officials is gratefully acknowledged as is the support of the staff of the World Bank Country Office in Pakistan. This audit set out to answer a number of questions of which the following were the most important: * Superficially this looks like an Integrated Rural Development Project (IRDP), but is it really? In what way do the findings modify the Bank's overall experience with IRDPs? Did project complexity negatively affect project performance? * What was the impact of the changed policies on the efficiency of resource use? In particular, what was the impact on levels of input use from the loss of input subsidies? * Should the Bank have been more proactive on the issue of inadequate local budgetary provisions? * How does the cost-effectiveness compare to the performance of other nongovernmental organization activities in AJK, for example, the similar National Rural Support Program (NRSP)? * How sustainable are the village institutions and the project? The ICR is clear, informative, and well presented. It rates the project satisfactory. Since this was an Intensive Learning ICR, OED was given the opportunity to comment on earlier drafts and a number of comments made have been incorporated. Following standard OED procedures, the draft PAR was sent to the borrower for comments before being finalized. Borrower comments have been taken into account, and are included as Attachment 2.  1 1. Background Azad Jammu and Kashmir (AJK), at the foothills of the Himalayas, covers an area of 13,300 square kilometers and has a population of about 2.6 million that is growing at 2.7 percent per annum. Per capita income is only about US$280 per annum compared to US$410 for Pakistan as a whole. AJK has a special status in Pakistan. It has its own president, prime minister, and legislative assembly and has a number of political, administrative, and developmental responsibilities, but matters relating to finance and defense are handled by the Government of Pakistan (GOP). Over 90 percent of the population of AJK live in rural areas and are engaged in subsistence agriculture on terraced land and steep hillsides. Forestland covers 42 percent of AJK; the remainder is divided between cultivated land, cultivable wasteland, and uncultivable wasteland. About 27 percent of the forestland is productive, 32 percent is potentially productive (but currently poorly stocked), 6 percent is protected forest, and 35 percent is permanent range. Timber on community- owned and state land is almost exhausted due to excessive exploitation. The main objective of the five-year Northern Resource Management Project (NRMP) was to provide for the sustainable and economically efficient use of land resources in one of the country's poorest, most environmentally vulnerable areas through: (a) improving the policy framework within AJK; (b) restructuring and strengthening the main sectoral institutions; and (c) testing, on a pilot basis, programs whereby communities took management and financial responsibility for planning and implementation of activities for sustainable resource management. The project components were as follows: * Policy actions, including reduction of subsidies (on potato seed, vegetables seed, fruit seedlings, and forest seedlings); transfer of a number of production and distribution activities to the private sector; and redeployment of staff arising from the privatization. * Institutional strengthening, including: reorganization and strengthening of the Department of Agriculture with a main objective of improving the efficiency of extension; seed quality-control; a community awareness program; reorganization and strengthening of the Department of Animal Husbandry; strengthening of the department of forestry including creation of a Social Forestry Wing; strengthening of the Planning and Development Department of AJK (US$13.3 million; actual $8.3 million). * Strengthening and operating support for basic technical services for some programs initiated under earlier projects covering extension, adaptive research, training, support for animal health campaigns, support for reforestation, etc. Farm activities mostly encompassed the districts of Kotli and Poonch (US$7.0 million; actual $4.9 million). * Support for community participatory activities in five pilot areas (Banguin, Hular, Baluch, Chhotagala, and Sensa); a Social Mobilization Unit (SMU) to interact with communities and assist group formation, programs of technology demonstrations, reforestation, provision of planting material, and support for women's group formation (US$1.9 million; actual $0.6 million). * Improvement of 165 kilometers of rural roads (US$8.4 million; actual $18.8 million). 2 2. Findings The main objective of the NRMP, providing for the sustainable and economically efficient use of land resources, was largely met. The policy framework was modified. Most planned actions to transfer input production and distribution to the private sector, to eliminate subsidies on inputs, and to re-deploy staff were completed. Institutional strengthening was carried out as planned. Communities were successfully organized in the five pilot areas, and a majority of the Village Development Committees (VDC), and the later-formed Women's Development Committees (WDC), were functioning satisfactorily. Most Project outcome was of the technical support services were provided and planned studiesache carried out. Road improvement work was substantially delayed, but was efficienty but with some largely completed in the end. However, the Bank failed to ensure the shortcomings - including application of the Involuntary Resettlement Operational Directive, which Implementation delays and a was applicable at the time (Lesson 5). While sustainability has been an theuaiaion f enhanced by the substantial policy and institutional reforms, the c n ty paiition sustainability of the community arrangements in the pilot villages without remains uncertain. further support is uncertain. LESSONS 1 . Multiple component rural development projects can work if they focus on efficiency at the village level through community participation backed by efficiency of services through institutional reform and privatization. Complementarity between policy reform, investments at village level, and community institutions and processes should be reflected in the logical framework. 2. With respect to sustainability there are four main lessons: (i) keep the community mobilization project element to a manageable scale - this was done in this case; (ii) at least indicatively plan for the post-project period at appraisal with respect to both funding and operational design and prepare early for that transition - this was not done1I; (iii) develop NGO capacity to play an increasing role - partly done outside the project under the National Rural Support Program (NRSP); and (iv) use the Bank's new project instruments with longer commitments -not available in this case at the time of appraisal. 2 3. Community-requested training programs on agricultural technologies, particularly those which enhance individuals skills to supplement household income and delivered within the village, can be a very effective technology transfer mechanism and may be appreciated more by beneficiaries than routine extension visits. For efficiency, a complementary combination should be sought. I. This reflects the borrowers comment no. 2, although it does not go as far as proposing a follow-on project which goes beyond OED's evaluative role. 2. The 1996 audit for the predecessor Integrated Hill Farming Development Project (IHFDP) concluded similarly: "some time is likely to be required to build up effective relationships between communities and implementing agencies .oa longer implementing periodl.. is desirable." 3 4. Appraisal reports should analyze project and post-project local funds demand, compare these with recent budget allocations, and explicitly size the project against these findings. Where financial sustainability still remains an issue, they should outline a phased program of response, for example cost recovery, phased reduction in services, privatization, etc. 5. Road construction, even road improvement, frequently involves some measure of land acquisition and therefore falls under the Involuntary Resettlement Operational Directive, calling for careful consideration of the compensation arrangements. From the inception of project processing managers should invoke the Involuntary Resettlement Operational Directive and assist borrowers in ensuring that the Bank's safeguard policies are met. 6. The absence of a suitable credit line limits the options in community driven development and has implications for sustainability. This raises the importance of addressing credit delivery issues in a pragmatic way. FUTURE DIRECTIONS "Where to go from here" is under active consideration by both the Bank and the borrower. The dilemma is that, on the one hand, the achievements of this project need to be consolidated, but on the other hand, the achievements need to be spread to other beneficiaries and the new approach institutionalized more widely. The following are some pointers: * It would be timely to shift focus more towards a sustainable rural livelihoods approach which acknowledges the reality that over 50% of household income is now from off-farm sources and focuses not simply on agriculture, livestock, and forestry but starts to introduce vocational training, employment information, further increases in attention to women, agro- processing and marketing support, and seeks out poverty alleviation benefits from rural- urban linkages. This would be associated with an explicit strategy to identify and reach the poorest groups, the poorest villages, and the poorest individuals and to ensure their inclusion in community deliberations. The Social Welfare Department could now be brought more into the picture with its comparative advantage in vocational training (and also in its role in facilitating registration of the apex bodies for VDCs). * There are pros and cons to handing over all or most community mobilization to NGOs. In favor of such an approach is the generally acknowledged skills of NGOs. Against it, however, is that public technology transfer will continue to be needed and, in order to play a key role, public services can benefit from the first-hand experience of participatory approaches. The aim should be to work out effective and efficient service delivery arrangements involving all actors: central government, NGOs, private sector, local governments, and local communities. * Given the role of small stock in milk production at the household level, their probable association with poverty, and their impact, through grazing, on fragile rangelands, more 4 attention should be given to sheep and goat breed improvement and work with communities on sustainable management of hill pasture grazing.3 * Financial sector policy weaknesses are holding back support for agricultural credit. This is a serious handicap to rural development programs such as NRMP. * A study is needed to address the problems involved in delivering inputs to remote areas. If subsidized support is considered, it should be looked at broadly as a question about how best to help poor remote areas not simply how to get fertilizer to them. Private sector participation would be important in such a study. The chances of future sustainability would be enhanced by: * The formation of the Village Development Committee (VDC) Apex body which has been under consideration. A few cluster VDCs are already showing signs of joining hands, but government support may be needed to establish this Apex body. * The expansion of the successful National Rural Support Program to the NRMP pilot areas, including, if possible, the credit component. * Agreement on a reduced and declining level of minimal social mobilization support for the VDCs established by the project. * Asking for a quid pro quo from the best of the VDC and WDC presidents, to assist, for a modest payment, in establishing new VDCs and WDCs in neighboring areas, but still with some support needed from SMU. * Charging for training, on a declining subsidy basis (NRSP is already charging for some of the training under that program). * Continuing, under the government programs, to give training priority to the newly established project VDCs and WDCs. * Putting input dealers more in touch with the VDCs and WDCs so that they start to take over more of the technology transfer load and training input dealers to better leverage the limited public technical resources. The project was largely relevant to the strategy of 3. Analysis GOAJK and to the problems of the rural areas of AJK, although a somewhat greater RELEVANCE focus in project design on poverty alleviation would The project addressed institutional reform and privatization of input have improved relevance. supply, which were the two most relevant needs in the government of AJK rural strategy at the time and were, and remain, consistent with the strategy. The project also addressed the redirection of extension and training and there is now a substantial focus on 3. The latter part of this recommendation is similar to one in the 1996 audit of IHFDP, which called for a more proactive stance by the implementing agency where community-based groups are important in the management of natural resources to foster longer-term conservation. That audit correctly noted that, as improved dairy animals increase, which are generally stall-fed, and as on-farm spring fodder cropping increases to feed those animals, the pressure to graze fragile spring pasture may reduce sufficiently to get farmer cooperation to improve upper range grazing, whereas, before these changes, improving communal grazing areas, if it involved any restrictions on access, would have been difficult. 5 community demand-based training programs, often delivered at facilities within the village itself. Project relevance would have been enhanced if there had been more explicit poverty alleviation design in the project. EFFICACY The first question on efficacy is whether the project funding was truly incremental for the line agencies concerned or whether amounts provided under the project were partly or wholly withdrawn from other programs. An analysis of the budget data done by the mission suggests it was largely incremental. The evaluation surveys by the Punjab Economic Research Institute were well done and credible, although barely an adequate statistical sample for The project was substantially the terminal survey. They found that, comparing the 1992 benchmark effective. Giving the greatest survey findings with the 1998 terminal study findings, yields of major weight to the policy and crops had increased by 18 percent for maize, 28 percent for wheat, and institutional reforms, most of 18 percent for apples. Field observations by the mission suggest that the objectives were met. vegetable production and poultry production, unrecorded in the surveys as enterprises, have experienced significant productivity gains. Whether milk production has gone up substantially as a result of the project is more uncertain. While significant numbers of newly bred improved dairy cattle were seen in the field, the survey itself shows some decline in numbers of adult cows although a significant increase in numbers of heifers. Since the decline in the number of cows could signal a decline in unimproved cows alongside an increase in improved cows, the true picture on milk production remains uncertain. Net farm income rose by 20 percent in real terms. Field visits found tremendous benefits from the road component which, due to delays, were mostly operative after the terminal survey. These were roads built on what had usually been unimproved tracks (mostly donkey tracks) along very steep hillsides. Benefits included reduced transport costs for fertilizer, wheat flour, and vegetables, both in and out, by around 80 percent (based on discussions with beneficiaries); greatly reduced travel times for all travel purposes including social purposes, which releases labor for productive use; better supply of social services; the saving of lives by enabling sick and injured people to get to hospital quicker; and increased land values on the order of 400 percent.4 Two stakeholder workshops held in association with the Intensive Learning ICR, found the following: * General agreement with the findings of the beneficiary surveys * Longer project periods were recommended to allow consolidation of the community participation, this plea was repeated in all the mission's village discussions (Lesson 2). * It was felt that if NRMP had had a credit component better results could have been achieved. * The problem with the flow of counterpart funds was noted. * More needed to be done to heighten awareness of natural resource concerns. 4. The use of changed land values may be an underused methodology for assessing rural roads benefits. Households near the project roads seemed to have a very clear idea about the increased land values due to the road, a value which represents a proxy for many individual benefit components which individually are difficult to measure and present double-counting challenges. 6 * The elimination of input subsidies and the privatization of supply was a good thing, but remote areas had suffered because the private sector concentrated on the easier areas. * The SMU should be retained for another 3 to 5 years, it had worked to effect the social mobilization due to a dedicated individual, but in future an SMU should be autonomous or it should be assigned to an NGO. * The project needed a better-designed exit transition when passing from Bank funding to government budget funding. * The most highly regarded project component was community skill training (Lesson 3). * Institutionalization of gender focus will require sustained efforts. * The government-supported follow-on operational project phase, currently approved for one year, should be extended to three years. * Future projects involving social organization should be not less than 10 years. Box 1. Stakeholder Comments "We were in a deep slumber and the project has helped us come out of it." "The project has instilled self-confidence in us." "We were not interested in the past in the free tree seedlings that were offered to us while now we demand these in much larger numbers at full price due to much higher survival rate of seedlings." "Social organization should be introduced as a subject in school curricula." Physical Targets. Physical targets were substantially achieved and are given in Table 1. In some cases targets were not fully met, or were delayed due to inadequate local counterpart fund provision and lack of timeliness. The problem was partly the financial constraints of the government of AJK, but there was also a problem with processes. For example, for forestry, late funds release and equal monthly releases made it very difficult to implement works in their narrow window of seasonal opportunity. Animal production and extension activities remained weak, and the artificial insemination program did not take off for want of operational funds. There was a considerable increase in orchard crop planting in the villages and an increase in the planting of fuelwood species. Forty Women's Development Committees (WDCs) with a total membership of 829 were established, which was not an original project target. Training to WDC members was provided in vegetable gardening, chick rearing, vocational training (e.g., sewing, soap making), poultry vaccination, forest nursery establishment, food preservation, and social mobilization. Some inputs were distributed to WDCs on a cost-sharing basis. The Land-Use Planning Unit has, so far, surveyed and mapped 45 percent of the total land area of AJK. A modest on-farm soil conservation demonstration program was successful, demonstrating high cost-benefit ratios, which would imply that the subsidy is no longer needed. Integrated Pest Management has been promoted. An Agricultural Research Master Plan was prepared covering a five-year research agenda. 7 Table 1. Physical Targets Indicator Target Actual Village Development Committees 20 60 attempted, 35 active Women Development Committees NA 40 Agric. Demo. Plots 2,886 2,150 Adaptive Trials 250 102 On-farm Soil Cons. Demos. 300 90 Vaccination - HS&BQ 510,000 570,000 Vaccination - Enterotox. 148,900 22,000 (low outbreaks) Plantations (ha) 10,000 9,625 Soil Cons Structures (cu. mt.) 247 113 Land Use Planning (ha) 22,467 22,467 Length of New Road (km) 202 189 Policy Reforms. The reforms were a major achievement of the project and, by and large, they achieved their objectives of increasing revenue and releasing staff resources to focus more on technology transfer. Over the period of the project, and in accordance with the project conditionality, subsidies on inputs such as fertilizer, potato and vegetable seed, orchard seedlings, poultry vaccination, and forestry tree seedlings were phased out. Input use does not appear to have fallen as a result of increased prices. During discussions with beneficiaries it was clear that availability was usually a bigger issue than price. At the same time, as planned at appraisal, institutional restructuring was carried out, a key element of which was getting the departments of agriculture, animal husbandry, and forestry out of input production and distribution activities. However, in one important respect, the policy reforms have left a problem that the government is well aware of. The privatization of input supply has resulted in the poor servicing of the more remote areas which are less profitable for the private sector due to low-volume and high transport costs. Since these are also the areas with many of the poorest households, a strategy is now needed to resolve this problem which should, perhaps, have been foreseen. Box 2. Arnila - A Typical Village Development Committee The only way to reach Amila is to ford a river that for part of the year cuts the village off from the world. The VDC currently has 14 members and has been functioning for about two years. The committee started late because its members were initially skeptical due to past unkept promises. The committee currently has Rs. 61,500 in its savings account; it had more but has used some. The committee members value the training they have received. They were assisted in the purchase of an improved bull and now have cross- bred cows that, though not yet in milk, are expected to produce 15 liters/day compared to 2 liters from the local breed. The VDC has planted 50,000 trees on the hillside, and the subsidy they received to do that was deposited in their account for further development activities. The committee has had assistance for four citrus orchards and some poultry enterprises and received poultry vaccination training. The committee is now self-sufficient in poultry and markets its surplus in the town just across the river. The VDC members are disappointed the project has closed just as they got started. Their priority for the future is continued training and, if possible, a bridge. Gender. The Women's Development Committees, commendably added to the project during implementation, have been very successful. Field observations by the mission suggested that there was even more commitment in these groups than in the men's groups. There appeared to be 8 greater planning and focus on the future and they were more inclined to formally record, in writing, their plans. EFFICIENCY Economic Rate of Return. The ICR estimates the ERR of the project to be 16%. The audit believes that this may be somewhat overestimated and would place the ERR at between 12% and 16%. There are arguments for both lowering and raising net benefit components of the ICR ERR estimate. Arguments for lowering the ICR ERR estimate include: * Benefits start at their full level in the final project year, which is optimistic for some components, particularly roads, where some of the construction was not even completed by the final year, however, some benefits would start to flow earlier. * A share of the incremental benefit stream in this project should be attributed to the investments under the previous Integrated Hills Development Project and, indeed, were claimed in the economic analysis of that project.5 Arguments for raising the ICR estimate include: Project efficiency is rated substantial. The rate of return * A number of additional benefits to the roads component, particularly is above 10%, however there education and health benefits, and enormously increased land values were serious implementation educaion nd halthdelays and there are (a proxy for many benefits). questions about whether the pilot program could have * The fact that the incremental agricultural benefits, based on the achieved similar impact at evaluation survey findings, remain static in the ICR analysis at their lower cost. 1998 level rather than rising somewhat over time as might be expected as privatized input supply improves, and as training benefits grow. On balance, it is the judgment of this audit, supported by some rough calculations, that the above would partly, but not entirely, cancel out, and given the difficulty of separating out attribution of costs and benefits between this project and earlier projects, the audit would place the ERR between 12% and 16% (for the sake of a data point, 14%). Cost-Effectiveness Relative to Other Programs. The costs of the pilot area component were high relative to the larger National Rural Support Program (NRSP), but the latter benefits from returns to scale. The number of VDCs covered per social mobilization specialist (about 12) was substantially lower under the project than under the NRSP (about 70). This would partly solve itself if the program were to grow. Efficiency of Project Management Structure. The project management arrangement is described in Attachment 1. The Secretary, Department of Agriculture (DOA), and Department of 5. The dilemma of what can be claimed for the project was addressed also in the audit of that project where the ICR had arbitrarily halved their initially calculated 2% ERR to 13% on the basis of benefit attribution but where the subsequent audit assessed that to be too conservative and considered an ERR closer to the initially calculated ERR to be reasonable. 9 Animal Husbandry (DOAH) had overall responsibility for the project but under him there was a Project Director with a small staff responsible for coordination. The current Project Director believes that efficiency could have been improved if, within his office, there had been a limited number of seconded departmental staff from the line agencies fully dedicated to the project work. It is possible that this arrangement would have improved efficiency during the project period. However, it would possibly have made the transition to the post-project period more difficult. Efficiency as an "Integrated Rural Development Project." This was not a traditional centrally planned Integrated Rural Development Project, so the findings do not suggest that, after all, with some minor adjustments, a traditional IRDP can work. However, this project does suggest that multiple-component rural projects can be quite successful provided there is a core element of community participation backed by institutional reforms in the public sector element and privatization of input supply (Lesson 1). There was substantial complementarity between the different project components and reforms. Whether the efficiency of project implementation would have been improved if the number of components had been reduced is difficult to judge. In a project that is based on meeting the demands of community organizations, the capacity to flexibly meet a wide range of demands would seem important. In future, one would anticipate an increasing share of the demands could be met by the private sector. Efficiency of Consultants. The borrower/implementing agency was particularly pleased with the quality of the consultants. Consultant reports reviewed by the mission appeared to be of high quality. This clearly seems to have been money well spent. INSTITUTIONAL DEVELOPMENT Institutional development impact was satisfactory. With respect to public institutions, it has resulted in the initiation of a paradigm shift in the line agency approach to communities, at least in the pilot areas, and, to a lesser extent, more broadly. The institutional reforms, which got the line agencies out of a substantial task in subsidized input distribution, appear to have been largely successful in improving effectiveness for Institutional development about the same overall public expenditure. The reform released staff who impact was satisfactory. A can now give greater attention to demand-based technology transfer. Also, paradigm shift towards some broadening of training areas is evident. It is still quite early to assess demand-based responses to some technology transfer was the overall impact of these reforms on rural welfare in AJK, and there will initiated. need to be a period of consolidation and adaptation, but, based on the evaluation surveys, the impact appears to be positive. With respect to institutional changes at the community level, the VDCs and V%DCs established under this project, and reflecting earlier NGO experiences, are significant institutional developments although the scale is still small. BANK PERFORMANCE Bank performance was satisfactory except for a clear violation of the Bank's Involuntary Resettlement Operational Directive in the roads component which overrides other considerations and results in an unsatisfactory rating. Preparation and appraisal drew a number of lessons from the Aga Khan Rural Support Program (evaluated in 1986, 1989, and 1995 by OED) and from the experience of the earlier IHFDP (audited by OED in 1996). However, the project appraisal report stated that "there are no resettlement issues involved" in the roads component. This was incorrect as defined in the Bank's safeguards. There were resettlement issues which should have 10 been appraised. The government of AJK policy at the time of appraisal, which the Bank does not seem to have been fully aware of, was that there would only be link road improvement where road users would not demand compensation and that affected persons were to give written consent to government and transfer the land to PWD. However, the project did not set up the process for this. The Bank operational directive, while not ruling out internal compensation within the community, would have required, at a minimum, a clear process that would have demonstrated individual household voluntariness, and a grievance mechanism (Lesson 5) 6.The audit, in its limited fieldwork, found enormous enthusiasm for the roads and did not, in fact, come across even one instance of a household that complained about loss of land, house, or assets. However, the audit did hear of at least one family on a road not visited who had lost their house and, it is believed, did not receive any compensation and it seems probable that a full survey would find others, although probably a small number, who have lost significantly. Shortage of local funding remained a problem for about three years and led to scaling down a few physical targets at midterm due to delayed implementation. Supervision missions noted this routinely and it was a regular feature of letters to the government. However, as the problem persisted there was no escalation of the issue from the Bank's side Bank performance was other than, at one point, to refuse an increase in the advances to the Special largely satisfactory but Account. The Bank should have been more proactive and should have due to the safeguard threatened suspension when the problem recurred. violation must be downgraded for the rating to unsatisfactory. There The appraisal mission should have ensured that procurement was much further was excellent rapport advanced by the time of project effectiveness, especially on roads. Substantial between government and delays were experienced, although some of these, involving re-bidding, were the Bank during not within the control of the borrower or the Bank. supervision. However there was a clear violation at appraisal of the The frequency of formal supervision was insufficient; in some cases 10 involuntary resettlement months passed between supervision missions. However, given that this project safeguard, and the Bank was being partly supervised from the field office with quite frequent informal should have taken firmer contacts, as is evidenced in the files, the impact on efficiency of the action on the local funding insufficient number of formal supervision's was probably not great. The issue. frequency of supervision improved to six months between missions when the project was managed from the field office. The borrower noted that cooperation with the Bank was excellent. Much of this was due to a very close relationship between government and the Bank field office. Borrower performance was BORROER PRFORANCElargely satisfactory except for the problems with local Borrower performance was satisfactory in most respects. The funding funding. There was a problem was serious and affected, in the end, the scale of the project. commendable and significant However, particularly in the roads component, considerable catching up was change of direction towards done in the final year. The borrower acknowledges the problems with ooapache woithhny funding and notes that now foreign-aided projects are being given greater transfer and village-level investment 6. The Task Manager comments: "The violation of the safeguard was mainly the Bank's failure to appraise the component from the compliance point of view. The absence of a formal grievance address mechanism at the PCU level was a notable omission to otherwise an internal conflict resolution and compensation mechanism which operated within the community and which had popular support in favor of building the roads." 11 priority for funding allocation - too late, of course, to have an impact on this project. The swing towards the community approach by the line departments has been a major achievement. Village communities are now starting to see line departments in a different light. Whereas in the past line departments, if they did anything, did things to people, now, increasingly, they are doing things with people. The challenge now will be to have the capacity to meet the demand being generated and to avoid failure to deliver and then to revert to the old cynical attitudes about government at the village level. There was a long delay in the announcement of the restructuring of DOAH, which was not announced until August 1995 although the groundwork was largely completed by September 1994. SUSTAINABELITY As is frequently the case in Bank projects, the capacity of the governments of Pakistan and Azad Jammu and Kashmir to carry a project of this scale with respect to annual government contribution during the project and post-project operation was not adequately analyzed. In this particular case, such an analysis would have been complicated by the special relationship between GOP and GOAJK. Nevertheless, it should have been attempted (Lesson 4). Sustainability in this project is difficult to assess. On the one hand the policy and institutional reforms are highly significant, sector- wide, and unlikely to be reversed. However, the community development activities, while very promising, are still fragile, but these cover a relatively small part of the project area. In particular the privatization of input supply and the redirection of the line departments' attention towards the servicing of villages on a demand basis with technological packages, bodes well for project sustainability. It would be difficult to argue that government transfers to the village or community level are sufficiently institutionalized to the point where one can be totally confident about sustainability, but they do seem likely to be sustained. In the pilot areas, and in other NRSP areas, the VDC savings and local contributions for services bode well for sustainability and for greater cost recovery, but again, there is further to go to achieve full institutionalization of these arrangements. The newly formed community groups have only been functioning, in most cases, for two or three years. While they have all had a taste of what can be done with community mobilization, as some of the beneficiary comments suggest, many of them have probably not reached the stage of full self-sustainability. The number of groups is small for AJK as a whole but this was a pilot. At the stakeholder workshops a number of beneficiaries and project staff said that they felt that a project snble prjet supporting the establishment of community organizations needed a good 10 although this Is a close call. years to become self-sustaining. The government is now addressing the issue Promising for sustainabilty is of how to keep the VDCs in the pilot areas going. A positive sign is that the important and far-reaching other community groups in AJK under other programs have sustained privatization, the public themelve resonaly wll.institutional reforms, and the themselves reasonably well.In ine department's community With respect to environmental sustainability, the main project objective was approach. Less promising for "to provide for the sustainable and economically efficient use of land sustainability Is the still fragile resources in one of the country's poorest and environmentally vulnerable condition of the newly formed areas." Did the project achieve this environmental sustainability? Yes, at the community groups. local level. No, at the regional level. In the foothills of the Himalayas, with such challenging and spectacular geology, and given the evidence that a very high percentage of erosion in such areas is geological in nature rather than man-made, it would have been optimistic to expect a project to have significant impact on the overall soil and water environment. The most that 12 could have been hoped for, and what appears to have been achieved, was to ensure sustainability on the land under project interventions with any small off-site regional benefits being a windfall. Road design appears to have accommodated environmental needs reasonably well, although the mission did not have a specialist in this field. One weakness is the lack of application of bio-engineering techniques for holding land in the proximity of new roads. With respect to ongoing financial sustainability, GOAJK is providing modest funding for the next year of operation. However, apart from the uncertainty of future operational funding availability because of the tight GOAJK financial situation, Bank staff have raised a valid budget procedural concern. During the project, funds to cover the increasing share of operational costs as the Bank share declines are drawn from the development budget not the operating budget, therefore, once the project closes and the development budget line item closes there is no pre-established line item to which to attach operational funds. This makes it procedurally more difficult to stake a claim on future operational funds. This needs attention in the design of future projects. On balance, given the substantial scale and significance of the policy and institutional reforms relative to the pilot scale of the community development component, and acknowledging that other community activities are functioning in AJK, we rate sustainability likely, although this is a close call. 13 Annex A Basic Data Sheet NORTHERN RESOURCE MANAGEMENT PROJECT (CR. 2512-PAK) Key Project Data (amounts in US$ million) Appraisal Actual or current Actual as percent of estimate estimate appraisal estimate Total project costs 36.0 32.6 91% Loan amount 28.8 25.5 89% Cofinancing 0 0 Cancellation (SDR) 2.1 Date physical components completed 12/31/1998 06/30/1999 Economic rate of return 13% 13% 100% Project Dates Original Actual Identification 11/90 11/90 Preparation 11/90 11/90 Appraisal 11/92 11/92 Negotiations 03/93 03/93 Board Presentation 06/15/93 06/15/93 Signing 10/15/93 10/15/93 Effectiveness 09/15/93 12/06/93 Project Completion 06/30/98 12/31/98 Project Closing 12/31/98 06/30/99 Staff Inputs (staff weeks) Total Up to Appraisal 160.0 Appraisal to Board 54.4 Board to Effectiveness NA Supervision 127.2 Completion NA Total 341.6 14 Mission Data Stage of Project Cycle No. of Persons and Specialty Performance Rating Month/Year Number Specialty Implementation Development Progress Objective Identification/Preparation: 11/90 7 Agr, Econ, For, Inst. Sp, Eng, WS Appraisal/Negotiations: 11/92 8 Agr, Econ, For, Inst. Sp, Eng, 2 Com Sp. Agr, Econ, For. 03/93 3 Supervision 01/94 3 Econ, For, Inst. Sp S S 09/94 3 Agr, Econ, For 08/95 3 Agr, Econ, For 07/96 5 Agr, Econ, For, Lsk Sp, WS 11/97 2 Agr, Econ 06/98* 2 Agr, For 12/98* 5 Agr, Econ, For, Eng, ACS 06/99* 3 Agr, Econ, For ICR 12/98 4 Agr, Econ, For, Eng, ACS 06/99 Agr, Econ, For Agr = Agriculturist, Econ = Economist, For = Forester, Inst. Sp = Institutional Specialist, Eng Engineer, WS = Women Sp., Com Sp = Community Development Sp, Lsk Sp = Livestock Specialist, ACS = Administrative & Client Support Staff. Mission dates with asterisk (*) indicate those supervision mission which were initiated from the Field Office Performance Rating of the 8 supervision missions has been consistently "satisfactory" for IP and DO indicators. Rating for the identification/preparation and appraisal/negotiation stages was not made. Other Project Data Borrower/Executing Agency: Follow-on Operations Operation Credit no. Amount Board date (US$ million) None yet, although follow-on under consideration 15 Attachment 1: Supplementary Background Information The Northern Resource Management Project (NRMP) was financed in response to the government's request to the International Development Association to support improved natural resources management in AJK. Management Structure. The project was implemented by the Department of Agriculture (DOA), Department of Animal Husbandry (DOAH), Department of Forestry (DOF), and Communication and Works Department, formerly Public Works Department (PWD). The secretary of the DOA and DOAH held overall responsibility for the project. The project provided for a senior officer as project director under the secretary. Coordination of the project was provided through the AJK P&DD, Azad Jammu and Kashmir Planning and Development Department which was strengthened. A Social Mobilization Unit (SMU) was established to assist with community participatory approaches. The PWD was strengthened to carry out the expanded road improvement program. Technical assistance was provided to enhance the capacities of implementing agencies. A Privatization and Reorganization Unit was established in P&DD to implement the phasing out of the public sector activities, the subsidy removal, and the associated staff redeployment. At the village level, the SMU and the community were to draw up a detailed management plan for each village. The SMU and technical staff were to identify suitable technical packages. The SMU was responsible for the formation of Village Development Committees in the five pilot areas. Monitoring and Evaluation. Monitoring of project activities was undertaken by the monitoring and evaluation staff in the planning and economics unit of the DOA. The inherent flexibility of the community plans was seen as making the role of monitoring more difficult but also more important. The project director was to coordinate the preparation of annual workplans and budgets and monitor project activities and prepare quarterly reports. A joint government/Bank midterm review was held to make any necessary changes. A benchmark survey was undertaken by the Punjab Economic Research Institute in 1992, prior to project effectiveness as a separate UNDP sponsored Socio-economic Survey of AJK. A terminal study - of 72 randomly selected respondents from 48 villages from the benchmark survey sample of 950 respondents from 96 villages- was carried out in December 1998. Average yields of major crops were found to have increased: 18 percent for maize, 28 percent for wheat, and 18 percent for apples. Financing. An IDA credit of SDR 20.4 million (US$28.8 million equivalent) was to cover 80% of total costs. There was no cofinancing. GOP's share was US$5.6 million. Procurement. Procurement procedures followed normal guidelines. All procurement was undertaken by the office of the project director, attached to the secretary of DOA and DOAH. This office was held to be sufficiently experienced in procurement as a result of two previous Bank projects. Environmental Aspects. The project was expected to be environmentally positive and to redress environmental problems. It was given a B classification. Senior staff were provided under the project to be located in the P&DD, and to have oversight over resource management and environmental matters in all development projects. Technical assistance was provided under the project. The project was expected to have a positive impact on tree crop cover and, by improving cereals and fodder production, to reduce pressure on hillside grazing. The feeder road improvement program was expected to have no major impacts on vegetation, wildlife, or other 16 land uses. They would be designed with provision for drainage and treatment of slopes using retention walls at weak locations. Indeed, a main focus was to correct poorly constructed roads from past work. 17 Attachment 2: Comments from Government COMMENTS OF AZAD GOVERNMENT OF THE STATE OF JAMMU AND KASHMIR PLANNING AND DEVELOPMENT DEPARTMENT Subject: PAKISTAN NORTHERN RESOURCE MANAGEMENT PROJECT (CREDIT 2512-PAK COMMENTS ON THE PROJECT PERFORMANCE AUDIT REPORT (PAR) I am directed to refer to your letter no. PC-3/21/KA&NA dated 24th April 2000. The comments on the Project Performance Audit report are given as under for further necessary action please. "The Operations Evaluation Department (OED), an independent organization of the World Bank, responsible for the preparation of PAR, has invited comments from the officials of the government (Borrower) through World Bank Resident Mission Islamabad, which would be reflected in the final PAR. We are in agreement with the findings/lessons learnt contained reflected in the PAR which is clear and well presented. However, following are the borrower comments on the captioned report". 1) PAR has rated the Bank performance as unsatisfactory as it failed to implement/invoke the Involuntary Resettlement Operational Directive (lesson 5), calling for careful consideration of the land compensation arrangement as road construction frequently involves land acquisition. This Comment of the OED in the PAR, appears to be very harsh on the following grounds: As 100% cost of the Land Compensation was to be borne by the Government (Borrower), hence against this backdrop, Government has categorically made it clear to the Bank's project preparation Mission through a Government notification that no compensation would be made for the link farm to Market roads and as such cost of land compensation was regarded as community share. It was again made clear to various communities at the time of road alignment survey and subsequent Land possession that no land compensation would be paid. However, there was only one instance, when one person got stay order from the court for want of Land compensation. But under the pressure of the communities members, the person withdrew the court stay order voluntarily without any financial demand. Due to above mentioned reasons, cost of land compensation was not reflected in the SAR by the project preparation Mission. 2) As it was mentioned in the paragraph under caption "Sustainability", in the PAR, the newly formed community groups started functioning only in the last two years, of the project. However, when the community participatory works were in full swing, the project was abruptly closed. It clearly suggested that proper exit mechanism was not designed/envisaged at the time of preparation of the project. It was mentioned at the stakeholders' workshop organized at the end of the project, that a project supporting the establishment of community organizations, which involves change in the human attitude and motivation for the better use of available resources for improvement in the socio-economic conditions of the farming community, needs 8-10 years of gestation period to become self-sustaining. 18 A lot of efforts, time and money were spent under the project to sensitize the communities about the philosophy and benefits of community organization at the grass roots level. However, when the communities just started taking benefits from the project, it was prematurely closed leaving behind frustration amongst farming communities. Therefore, it is suggested that under the caption "lessons" in the draft PAR a follow up project should have been suggested/proposed to sustain the activities of the community organizations initiated in the Northern Resource Management Project. 3) Due to the absence of budget credit line in the Project a lot of many productive infrastructure development schemes identified by the community groups, could not be initiated. It is therefore, suggested that under the caption "lessons" of the PAR, a recommendation should be made that under all the future community development projects, a budget credit line must be provided for the execution of productive infrastructure development schemes identified by the community on cost sharing basis. It is therefore requested that the above mentioned three comments, may please be communicated for onward transmission to OED/the World Bank Islamabad for incorporation of these comments in the draft PAR which is well conceived and entails comprehensive food for thought for the agencies/officials responsible for the preparation of the project (Muhammad Ejaz Khan) Assistant Director Development