Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) Report Number: ICRR0021496 1. Project Data Project ID Project Name P123820 BT: Remote Rural Communities Development Country Practice Area(Lead) Bhutan Agriculture L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-51640 31-May-2018 8,692,704.20 Bank Approval Date Closing Date (Actual) 02-Oct-2012 31-May-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 9,000,000.00 0.00 Revised Commitment 9,000,000.00 0.00 Actual 8,692,704.20 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hannah Mary Bleby John R. Eriksson Christopher David Nelson IEGSD (Unit 4) Orford 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The statements of the Project Development Objective (PDO) in Financial Agreement (FA), Project Appraisal Document (PAD) and Implementation Completion Report (ICR) are identical. The project development objective is to increase (i) agriculture productivity and (ii) access to community assets in remote rural areas (by improving access to markets, irrigation, agricultural technologies, and community infrastructure). (PAD p4, FA p4, ICR p1). Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) This ICR Review treats the following as the project objectives: PDO 1: increase agricultural productivity in remote rural areas PDO 2: increase access to community assets in remote rural areas b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project underwent a level 2 restructuring in August 2017 to reduce the scope of irrigation work (ICR p8), due to cost overruns (see section 2e below) although the process was initiated in November 2015. The PDO was unchanged from appraisal. The first three PDO indicator targets were maintained although proxies were adjusted, following consultations with the communities and to better capture results (see discussion in section 4 of this document). The fourth PDO indicator target was revised up to address underestimations at appraisal (ICR para 12). The project components were not revised. Component A. Rural Infrastructure (estimated cost of US$6.19 million; actual cost of US$5.33 million). This component focused on improving irrigation and connecting remote rural communities to markets through (a) constructing farm roads, landslide protection, and rehabilitation works; (b) rehabilitating existing irrigation schemes; (c) developing three high‐efficiency pilot irrigation systems; (d) constructing water storages to support paddy nurseries and high value crops; and (e) strengthening the capacity of Water User Associations (WUAs) to improve the management of operation and maintenance (O&M) schemes. This component focused at the dzongkhag (district) level, as it was designed to benefit multiple gewogs (local administrations) (PAD para 16). B. Community, Marketing, and Productive Infrastructure (estimated cost of US$2.33 million; actual cost of US$2.46 million excluding beneficiary contribution). This component was made up of two subcomponents: (B1) support in high-poverty areas for improving access to critical community infrastructure and community‐led marketing infrastructure. Subprojects were identified and implemented by communities, taking a participatory approach; and (B2) support to existing producers on a cost sharing basis, for improving productivity of key commodities and adoption of sustainable land management (SLM) and wildlife management technologies (WLM). This component was focused on the gewog (local) level. C. Project Management and Institutional Strengthening (estimated cost of US$0.48 and actual costs were US$0.41m) This component supported project management, monitoring and evaluation (M&E), and the strengthening of the Ministry of Agriculture and Forest’s (MoAF’s) capacity to support the implementation of community subprojects. It also financed the mid‐term review (MTR) and end‐line evaluation. This component focused at the national level. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) The ICR states that the scope under component 2 was expanded due to substantial cost savings from exchange rate gains (ICR p42). Saving from exchange rate gains (about US$800,000) plus US$200,000 from Component A1 (road construction) were allocated to Component A2 (irrigation) which faced cost overruns, due to a range of issues, including required changes in design, losses due to weather, transportation costs, and non-competitive procurement (ICR para 16). The expected borrower contribution at appraisal was US$500,000 in cash contributions from beneficiaries of cardamom, citrus and livestock activities. Around $143,000 was forthcoming (with higher in-kind contributions) but the requirement was eventually cancelled since it was causing significant delays, and was eventually recognized as unaffordable and inequitable, given the project was targeting the poorest areas (ICR p9-10, PAD para 6). Key dates were as follows: The project was approved on October 2, 2012 and became effective on November 15 2012. The mid term review was dated November 24 2015. Estimated original closure and actual closure were both May 31 2018. Restructuring Level 2 was dated August 24, 2017. 3. Relevance of Objectives Rationale The PAD and ICR showed clear links at appraisal and closing, between the project’s objectives and the development priorities of the Royal Government of Bhutan (RGoB) and the World Bank (the Bank). Country context: As set out in the PAD, Bhutan is highly dependent on the Renewable Natural Resources (RNR) sector, of which agriculture is a significant part, for employment (65% of total and 95% of Bhutan’s poor), food security and export earnings. Despite strong potential for high-value production, proximity to large, accessible markets and extensive government support, the sector was growing significantly slower at 2.2%, compared with annual GDP growth of 9% from about 2000-2008 (PAD para 1). The project objectives responded to the priorities in Bhutan’s Tenth Five-Year Plan (FYP) for National Development (2008-2013) and to the forward looking 11th FYP (2014-2019) which included: maintaining strong GDP growth and poverty reduction, emphasizing balanced regional development (including the poorest, most isolated areas, which had so far, benefitted least – PAD para 3); with particular emphasis on crop disease treatment; directly involving farmers in remote areas; strengthening supply chains, including internationally; improving public services such as infrastructure, irrigation, and social assistance (PAD para 10). The design also responded to fundamental constraints to growth and poverty reduction, such as poor connectivity between communities, markets and social services, due to mountainous terrain and limited and poor road infrastructure, especially in the south and south-west of the country, where poverty incidence is high (PAD p1). Sectoral and Institutional context: Improving productivity and incomes in the Renewable Natural Resources (RNR) sector has been key to the RGoB’s efforts over the past two decades, to decentralize and stimulate capacity for self-action and self-governance, and reduce reliance on central government as a means of poverty alleviation (PAD para 7). The project set out to address key constraints to productivity and Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) connectivity, such as: low levels of technology adoption; poor land, water and forest management; low capacity to deal with disease in important cash crops such as citrus and cardamom; and small dispersed rural population, which all compounded the challenges, especially in the poorest, more remote areas (PAD p2-3). The project design took detailed account of the multilayered governance arrangements between central and local government. It attempted to appropriately align implementation responsibilities with the respective administration levels (PAD p10-11 and Annex 3). To account for identified substantial risks in implementing agency capacity, Component C (Project management and institutional strengthening – PAD para 27) focused on strengthening MoAF to support local agencies in implementation. The Bank’s comparative advantage for this project lay in three decades of support to the rural development sector, and lessons from two ongoing operations in rural infrastructure and land management (PAD para 9). This project would bring about comprehensive livelihoods support, given its focus on remote and poor communities. Support to the RNR sector was agreed with the RGoB in the Country Partnership Strategy 2011-2014, in line with the Tenth FYP (2008-2013) and Eleventh FYP (2014-2019). Addressing low agricultural productivity was established as a priority, on the basis of a review undertaken by the Bank as part of the Country Program Strategy (CPS) (PAD para 10). The project also aligned with other IDA and IFAD and GEF projects in Bhutan (PAD para 11). However, a significant number of the project outputs were based on assumptions (set out in the Results Chain, ICR Figure 1) that did not hold, (see table below) particularly in relation to PDO1 (increased agricultural productivity). Consequently, the relevance of some planned outputs to achieving PDO1, was reduced (e.g. the amount of area replanted with certain crops was an inadequate output to achieve increased productivity, since in fact, disease persisted as a binding constraint to productivity). The table below sets out the discrepancies between the assumptions and the actual situation. Results Chain Key Assumptions (ICR Figure 1) Actual situation Project was restructured to change scope of Full rehabilitation of irrigation schemes without irrigation work in response to changed needs on the design change/modernization ground, cost overruns and delays. (ICR para 16); Local contractors with adequate capacity exist No significant change Cash contribution by beneficiaries proved Cash contribution is affordable to beneficiaries unaffordable and inequitable (ICR para 17); Replanting citrus/cardamom plants addresses Significant rates of disease persisted despite disease issues replanting. (ICR para 33-34); While productivity improved, there was a 40 percent decline in cow numbers, due to government policy Improved dairy cow breed will be maintained of eliminating ‘scrub animals’, including those cross- bred with the improved breed. (ICR para 35); Market for eggs existed but declined by 69 percent, Markets for eggs exist/existing groups with due to marketing difficulties, high feed costs and market linkages will be selected asset loss (ICR para 36); Unit prices for cardamom/citrus seedlings will not Unit rates for seedlings supplied by National Seed change Centre did increase (ICR para 19); Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) IA and field staff have capacity to implement Implementing agency and field staff had relatively capacity building works without TA low capacity and did require TA (ICR para 64, 67); Additional grant resources will be located for Additional grant resources were not found for institutional/capacity building. institutional/capacity building (ICR para 62). The ICR states that the original theory of change was not affected by the restructuring (para 23). This is apparent, even though assumptions related to implementation of certain activities were flawed (see table above as well as Section 9 for further discussion). The project’s PDOs themselves were realistic (increasing agricultural productivity (PDO1) and increasing access to community assets (PDO2)), taking into account the available financial, institutional and human resources. The PDOs targeted clearly established and agreed needs and drew on relevant Bank experience and resources. But the assumptions of what (activities) would be required to achieve them did not hold. Based on high relevance and reasonable ambition of objectives, but flawed assumptions regarding the relevance of some activities required for achieving PDO1, relevance overall is rated substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective PDO 1: increase agricultural productivity in remote rural areas Rationale Theory of Change (Results Chain) The theory of change for the project was logical. The project aimed to increase (1) agricultural productivity and (2) access to community assets in remote rural areas by improving access to markets, irrigation, agricultural technologies, and community infrastructure (ICR para 6). This would be achieved through providing comprehensive livelihoods support, making community-led investments, and through strengthening local, regional and national implementation capacity (see ICR Figure 1 p6 and PAD p2-3). The project would contribute to Bhutan’s higher-level objectives of balanced urban-rural development and poverty reduction by addressing key constraints to increased agricultural productivity and connectivity in some of the poorest, most remote areas of the country (PAD p3-4). Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) Against PDO1, the ICR reports that:  PDO Indicator target 1: increasing crop productivity by 20 percent (Paddy) was met, achieving 2.9 from a baseline of 2.2 tons per hectare. Although not in the revised results framework, the ICR notes the end-line survey also found that yields in high efficiency irrigation scheme areas (built by the project) increased by 33 percent for maize, 23 percent for vegetables, and 34 percent for cardamom (ICR para 29). In both high-efficiency and rehabilitated irrigation project areas, crop diversification was also observed in the end-line survey with notable increases in area planted with a range of other crops (ICR para 30).  PDO Indicator target 3: Marketed value of agricultural and livestock production was to increase by 20 percent in targeted areas through investments in rehabilitating selected productive assets. The target was not achieved in three of four targeted commodities where, in fact, market value declined. The result for cardamom was -9%, citrus -5% and poultry -47%. Only dairy increased, by 12%, which was less than the targeted increase of 20%. Baselines for each were given as 0% (ICR p38 and 45). This left some beneficiaries of these activities worse off than before the project began (ICR p46). To what extent this was the case, was not able to be assessed through the data collected by the project. Intermediate outcome results and targets (PDOI 1):  #13: Seedling planting target of 440ha was exceeded, with 594 ha planted (no baseline given).  #14: rehabilitation of 20% of targeted area were partially achieved: 61% for cardamom but 4% for citrus (no baseline given).  #15: (number of clients who adopted an improved agricultural and livestock technology promoted by the project) was 450, almost doubling the target of 284 (no baseline).  #16: target of 119 women from target #15 was exceeded, achieving 189 (ICR p38-39).  #18: Seedling survival rate targets of at least 70% were also achieved (no baseline given). According to the endline survey, there was 85.4 percent survival rate for forest and 69.8 percent for NFWP species (ICR p39).129 ha of 132.6 ha citrus area was productive. 66 ha of 82.7 ha of cardamom area was productive.  #19: Egg production fell significantly short of the target of 800,000 eggs, achieving 186,405.  #20: Milk production by those households who received cows from the project reached 386,400 liters against a target of 170,000 and a baseline of 0. Milk production from cows supplied by the project was substantially higher than that of traditional cows. Marketing of cheese and butter also increased substantially (by 44%). But overall, milk production declined due to a 40% reduction (mortality and culling) in cow numbers (ICR paras 13 and 35). The gains at the beneficiary household level were therefore offset by the overall decline in milk production. The Results Framework also refers to an intermediate target (#17) of at least 10 new technologies developed (from baseline 0), which was met. Evidence is not clear in the ICR, however, of what these 10 technologies were or by whom they were developed or used. Mention is made of fencing, training in farming practices and water saving techniques and general reference to ‘sustainable land management technologies’ and ‘wildlife management technologies’ (ICR para 40). But the ICR cites a lack of data to assess impacts of these activities (para 40). RGoB comments in response to the ICR note that the end-line survey had more detail on technology uptake rates. While included in Annex 5 of the ICR, it would have been appropriate to have included this government comment in the main text of the ICR. Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) The efficacy of PDO 1 overall was modest, based on having only partially met or not met at all a majority of targets, although in a few cases targets were exceeded. So while production increased in many cases, a number of targets specifying increasing marketed value of agricultural and livestock production were not achieved. The result was that some farmers were left worse off than before the project began (ICR p46. Specifics on the extent to which this was so are not provided). Rating Modest OBJECTIVE 2 Objective PDO 2: increase access to community assets in remote rural areas Rationale Against PDO2, the ICR reports that:  PDO Indicator target 2: (share of beneficiaries with access to critical community infrastructure was to increase by 20 percent in targeted area) was largely achieved (p38). From a baseline of 15% of beneficiaries with access to critical community infrastructure, at project closing 33% had access. This was an increase of 18 percentage points, largely achieving the target of 20 percentage points. This represented a combination of access to all kinds of infrastructure provided by the project (ICR paras 42-44 and clarification from project team).  PDO Indicator target 4: (25,000 direct project beneficiaries revised up at restructuring from 15,000) was exceeded, achieving 31,814 (127%), against a baseline of 0. This overachievement of the target was reported in the ICR as largely due to an underestimation of the original and revised targets during restructuring (ICR para 45). Of these beneficiaries, the target of 48% being female was met (15,304 (149% of target) also against a baseline of 0) (ICR p38). Intermediate outcome indicator targets (PDOI 2):  #6: 13 Chiwogs had improved road access to markets, exceeding the target of 11 (revised down from 40 at restructuring) over a 0 baseline  #7: 36.1 km of roads were constructed, exceeding target of 34 over a 0 baseline  #9: 688 ha were provided with improved irrigation, against a target of 845  #10: 162 ha were provided with high efficiency irrigation, exceeding target of 60 over 0 baseline  #11: 59 WUAs were created or strengthened, exceeding target of 49 over baseline of 33  #12: Chiwogs with improved access to critical marketing and community infrastructure met target of 47 over 0 baseline (ICR p38). There was, however, no further evidence of these activities presented in or annexed to the ICR. Access to community-led marketing infrastructure investments contributed to both parts of the PDO and increased by 9% overall (ICR para 43). The ICR also notes (para 39) that although the effects of these activities on production and market value were not tracked by the project, the end-line survey reported Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) substantial increases in maize productivity and market value, attributable to the maize processing units established by the project. The efficacy of PDO 2 overall was substantial, based on almost fully achieving targets in relation to increased access to critical community infrastructure and community-led marketing infrastructure, and exceeding the target numbers of beneficiaries. It should be noted though, that exceeding the latter target rested on the basis of an underestimation of this target at appraisal and again at restructuring. Rating Substantial OVERALL EFF TBL OLD Rationale Efficacy of the project outcomes was weakened by insufficient focus on and funding for strengthening local institutions to implement and monitor the project. This was true in both design and implementation (see ICR para 72 and section 9 of this document). Risks in this regard were identified in the PAD (p50) and mitigating measures were proposed through Component C (Project Management and Institutional Strengthening) but there were insufficient indicators against them and implementation was weak (e.g. ICR paras 31, 40, 56). RGoB reluctance to use credit/loan resources for such activities was a contributing factor (ICR para 57). This finding underscores the ICR’s observation that ‘hard’ investments must be supported by ‘soft’ investments in order to reach desired objectives (para 100). (Lack of adequate Bank supervision also contributed to this result (see section 8b of this document). Overall efficacy of the project was modest, based on only partly achieving its objectives. While indicator target 1 of PDO1 was met, target 3 was not only not achieved, importantly, some farmers were left worse off as a result of the project. The beneficiary targets (2 and 4) of PDO2 were largely achieved or exceeded (due in part to initial underestimation of targets and to no provided baseline), respectively, resulting in a substantial rating. Therefore, on balance, and given the lack of information regarding baselines from which to evaluate changes, the overall efficacy of the project is judged to be Modest. Overall Efficacy Rating Primary reason Modest Low achievement 5. Efficiency Economic efficiency: An economic analysis was undertaken at appraisal and at evaluation (ICR Annex 4), using the same methodology. It excluded returns to social and marketing infrastructures, community forests and SLM and WLM technologies due to unavailability of data. The ex-post economic rates of return and economic net present value estimates at 12% and BTN of 68 million were considerably lower than at appraisal at 23% and BTN 302 million, respectively. The ICR points to key contributing factors as: (i) considerable implementation delays which slowed the benefit realization; (ii) lower than expected benefit scopes for almost all activities; and (iii) poor performance of investments in cardamom, citrus, and poultry commodities (ICR para 49). Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) Return on investments estimates in irrigation infrastructure was EIRR 14%. Returns on agriculture were much lower at 5% for cardamom and citrus, and dairy and poultry at around 1% (ICR p47). The project beneficiaries reported efficiency gains resulting from: increased crop yields and diversification; improved water supply; increased irrigable land (ICR p44); higher milk production (from project cows); improved access to markets; goods and services; and the end-line survey reported a reduction in walking time by almost half (ICR para 41). Although not analyzed, due to lack of data, the ICR economic analysis also points out that it could reasonably be expected that investments in market and community infrastructure would improve access to income generating opportunities and wildlife technologies would reduce crop loss. Additionally, the project generated non-quantified benefits such as improved capacity, market information, agricultural and water saving techniques, and improved quality of agricultural products. These would overall, also likely increase returns on project investments, (assuming that these benefits exceeded their costs) (ICR p42). Administrative Efficiency: The project was completed without extension. As the ICR notes though (p16), the project faced substantial implementation delays primarily related to a lack of implementing agency capacity in engineering, procurement, financial management and M&E; lack of a dedicated team and staff shortages (e.g. no requirement to add the project to their own work plans); an unrealistic cash contribution requirement (see section 2e above). While causes of multiple delays are noted throughout the ICR, the length of delays against expected timeframes is not detailed. (Slow spending against disbursement is indicated at Figure 2, p24). Due to inadequate monitoring and evaluation (see section 9 below), issues arising were not addressed in a timely manner. There were also substantial cost overruns in irrigation works (36% more budget was required) but changes to the irrigation scope also brought efficiency gains in the form of more durable irrigation systems and lower maintenance costs. At the same time, implementation delays, reduced benefit scopes and poor investments in several key cash commodities resulted in an ERR at closure almost half that at appraisal. Saving from exchange rates (about US$ 800,000, ICR p42), however, were also instrumental in reaching irrigation targets (ICR para 52). The project was completed with a final 91.2% disbursement rate (final figure provided by project team post ICR). Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  23.00  Not Applicable 0 ICR Estimate  12.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) 6. Outcome The project generated a number of positive outcomes towards its objectives of 1) increasing agricultural productivity in remote rural areas and 2) increasing access to community assets for people in remote rural areas. Ultimately though, on the basis of only partially meeting its objectives (modest efficacy), its modest efficiency and substantial relevance, the overall outcome rating is moderately unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome The PAD recognized substantial capacity risk in the implementing agency (although its commitment was considered strong) but did not consider design, delivery or monitoring risks to be more than low-moderate (PAD p11).The ICR, however, pointed to likely high risks in relation to: maintaining infrastructure built by the project (due to insufficient capacity and support from the project for local groups responsible for ongoing maintenance and to Bhutan’s harsh conditions – para 95) and financial viability of crops (due to ongoing disease issues not resolved by the project – para 96). There were also concerns over the continuation of water and road user groups, critical to maintaining infrastructure, in the absence of support from the project and not being integrated into the MoAF structure. A lack of adequate technical support to beneficiaries after the project presented substantial risk (ICR para 98). These combined to make for a substantial to high overall risk to development outcome. 8. Assessment of Bank Performance a. Quality-at-Entry Alignment of the project with both the RGoB and World Bank’s priorities was clear. The Theory of Change (TOC) outlined in the PAD (para 12) made clear connections between objectives (increase agriculture productivity and access to community assets) and outputs/activities (improving access to markets, irrigation, agricultural technologies, and community infrastructure). These aligned well with the components of the project, and intended beneficiaries were also clear and appropriate to the government’s overall development objectives. In the ICR, the TOC is represented by the Results Chain diagram (Figure 1). However, the ICR highlights a number of significant weaknesses. Although reasonable actions were proposed in the PAD (p50) to achieve expected outcomes or mitigate risks, more emphasis could have been placed on critical parts of the project so as to increase the likelihood that outcomes would be realized, such as: maintaining the improved cow breeds, support for implementation capacity and institutional strengthening (the Bank did not identify expected grant resources for capacity and institutional strengthening – ICR para 86), and disease analysis and control measures (ICR para 85). Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) Additionally, closer analysis of the financial viability and perceived equity of the cash contribution by beneficiaries would have been useful in minimizing the significant disruptions caused by this requirement. Significantly, eight out of nine key assumptions listed in the Results Chain (ICR Figure 1, p6) did not hold (ICR para 61-63). Some were outside the project’s control, such as market rates for commodities, but the majority of assumptions were not grounded in evidence and would have benefited from deeper analysis and verification ahead of implementation. Based on a relatively clear theory of change and good alignment with Government and Bank priorities, but significant shortcomings in some aspects of analysis and emphasis that contributed to non- achievement of some objectives, Quality at Entry is rated moderately unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision There were serious shortcomings in the quality of supervision. The ICR cited infrequent field visits, inaccurate, untimely and insufficient identification and reporting of issues, for example: the unviability of the planned cash contribution by farmers; the state of the irrigation systems at project opening; and weak progress of capacity building programs. Neither was the beneficiary participatory monitoring implemented, as envisaged in the PAD, although it was described as ‘essential’ to measuring project specific indicators (PAD para 38). These resulted in poorer quality follow-up of identified issues than would likely have otherwise been possible (ICR para 92) and likely reductions in efficacy and efficiency. Three changes of Task Team Leader also contributed to these shortcomings. The quality of technical experts provided to the project was questioned by the implementing agency (ICR para 93). The ratings of Modest for M&E across the board also reflect the shortcomings in quality of supervision (see section 9 below). Another indicator of inadequate supervision was the large discrepancies between disbursement of funds and actual spending, particularly early in the project (due primarily to implementation delays, see ICR figure 2, p24), yet there were consistent implementation ratings of satisfactory, largely due to lack of accuracy and candor in the Bank’s reporting (ICR para 89). Bank supervision overall was inadequate and untimely, leading to major shortcomings in the quality of information and subsequent shortcomings in follow-up of identified issues. Quality of Supervision Rating Unsatisfactory Overall Bank Performance Rating Unsatisfactory Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) 9. M&E Design, Implementation, & Utilization a. M&E Design At appraisal, the M&E process design was satisfactory. It was intended to be based on existing MoAF practices and draw on its agricultural data, implemented by the Project Coordination Unit (PCU, located in the MoAF), and complemented by a beneficiary participatory monitoring system. It was intended to provide for ‘continuous learning and adjustment’ (PAD p 11, 54). The outcomes were aligned with outputs and reasonably informed by PDO level indicators (ICR para 69), although many of the targets were themselves, output not outcome based. The Results Framework at PDO indicator level remained largely relevant throughout project implementation. However, many of the intermediate outcome indicators were flawed, for example: having multiple targets, targets misaligned with plant maturity periods, inaccurate descriptions of baseline and target values (ICR p20). Neither Midterm Review (MTR) nor the restructuring addressed these issues and added unhelpful complexity (ICR para 70 and 75). Further, according to the ICR, the capacity of the PCU proved too stretched, due to an overload of other work demands and lack of incentives to prioritize the project (ICR para 72) and the participatory monitoring system was not implemented (ICR para 73). This affected data quality and availability for implementation as well as evaluation (ICR para 75). Given the importance of capacity building and institutional strengthening for both successful implementation and risk mitigation at several levels, identified in the PAD (e.g. p50), it was a deficiency in design not to have more intermediate indicators focused on these issues (see also section 4 above). b. M&E Implementation The ICR states that efforts by the PCU to establish M&E processes early in the project were positive although they were not sustained, due to capacity constraints of the PCU and MoAF more broadly, for M&E (ICR para 72). As mentioned above, the major shortcoming of M&E implementation was not undertaking the beneficiary participatory monitoring system. The ICR does not specify why this was the case but suggests that weak capacity and poor Bank supervision contributed to lack of implementation of the participatory monitoring system (see section 8b above). While the project budgeted for surveys, it did not include funds for training field officers (ICR para 71). This combination of factors left significant gaps and inaccuracies in data during the project and at the close of the project, particularly related to investments in critical social and marketing infrastructures, sustainable land management and wildlife management technologies, and community forests. These had to be excluded from the efficiency analysis due to the lack of reliable data (ICR p43). c. M&E Utilization While data collection and reporting from the field did occur, the ICR points out that the lack of accurate, timely data undermined supervision and evaluation of project results. This meant that issues facing beneficiaries as well as potential benefits, were not identified (ICR para 76), for example, disease in crops or lack of distribution pipes, resulting in water storages (ICR para 66). This undermined the Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) potential to make mid-course corrections in a timely way and negatively affected results, for example, lower yields in disease affected crops, which necessitated extra financing (allocated from within the project budget) to connect water storage and distributions systems late in the project (ICR para 28). Based on significant shortcomings of M&E design, implementation and utilization, which made it difficult to assess the results of the project overall, the rating is Modest. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project triggered the following safeguards: Environmental Assessment OP/BP 4.01, Pest Management OP 4.09, Involuntary Resettlement OP/BP 4.12, Forests OP/BP 4.36, Projects on International Waterways OP/BP7.50 (PAD p19). Social safeguards: According to the ICR (para 76), all activities complied with safeguard requirements, with consistent rating of satisfactory or moderately satisfactory. No negative impacts were encountered during implementation, and grievances were satisfactorily resolved at the community level through a mechanism established by the project. Several positive social impacts were observed by an independent assessment carried out in 2017, noting a reduction of women’s drudgery and ‘time poverty’ (i.e. not having enough time to meet the demands of daily life or to maintain their wellbeing), increased mobility and economic empowerment (ICR para 76). Further details of this assessment are not provided in the ICR. Environmental safeguards: while compliance was rated satisfactory or moderately satisfactory, and compliance reportedly improved as a result of training by the PCU, reporting on environmental safeguards was hampered by the lack of a full-time specialist for the full project in the PCU (ICR para 77). No other detail is provided in the ICR. b. Fiduciary Compliance Procurement compliance was mostly rated moderately satisfactory, with one instance of moderately unsatisfactory and some audit observations by the Royal Audit Authority (RAA), (ICR para 78). The ICR describes this as due to a lack of dedicated procurement capacity in the gewogs, unfamiliarity with Bank processes (some training was delivered by the Bank – ICR para 79), decentralized procurement processes and remote project sites. Goods and services procurement was satisfactorily addressed by assigning a procurement specialist for one year. But weak procurement capacity remained throughout the project. Works contracts faced significant issues such as rebidding and mis-awarding of contracts (although no actual mis-procurement cases), resulting in significant delays and underutilization of project resources. These were suitably addressed by project management (ICR para 81). Conduct of a survey to better understand the local Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) contractor market reduced these incidences and contributed to more efficient use of project resources (ICR para 80). The ICR notes that lessons from this project on use of a procurement specialist in the PCU influenced the Bank’s Food Security and Agricultural Productivity project (para 81). Financial management ratings were mostly satisfactory or moderately satisfactory, with one instance of moderately unsatisfactory in the 2015 Mid-Term Review (ICR para 82). According to the ICR, overall the financial management system worked well and reporting happened on time. Disbursement by project closing was 91.2% (final figure provided by project team post ICR). Challenges noted in the ICR were some slippages in awarding of works contracts, the departure of an experienced accountant part way through the project leaving pending work and causing some delays, and difficulty in collecting information from remote sites (ICR paras 82-83). Agreed internal auditing did not take place, primarily due to the still evolving capacity of the local audit authority. External audits were unqualified although they included some observations, which were regularly followed up by the Bank. Ratings by the RAA remained satisfactory (comment provided by project team post ICR). c. Unintended impacts (Positive or Negative) According to the ICR, the capacity of the National Seed Center and National Plant Protection Center were increased through the construction of 7 additional propagation houses and nurseries (in response to the NSC struggling to meet the project’s demand for seedlings), establishing a laboratory for disease detection unit, allowing for local testing of plants as opposed to overseas. This was and would continue to benefit communities beyond the project area (ICR para 60). d. Other Gender: The project did not explicitly consider gender in its design or implementation (other than PDO target of ensuring 48% female beneficiaries). The ICR indicates that there were some positive outcomes from the project on reducing women’s drudgery, ‘time poverty’, increased mobility, and economic empowerment were found by the independent results assessment, as well as ‘impacts on social dynamics (para 55). But there are no data presented from the assessment or elsewhere. Other: The ICR (para 58) points to modest outcomes in mobilizing private sector finance (US$51,000 as cash contributions from farmers, although the intended amount was US$500,000 in cash and in-kind contributions). Institutional strengthening, while insufficient for full realization of the project’s objectives, nevertheless did contribute some additional capability in road maintenance, water management, elements of crop production, bio-engineering, procurement and financial management and community infrastructure implementation (ICR paras 56, 57). Poverty reduction and shared prosperity: Since the beneficiaries of the project were some of the poorest, most remote communities in Bhutan, the benefits the project generated, such as improved access to community infrastructure, some strengthened implementation capacity, increased crop yields and diversification, can plausibly have contributed to more even development. This was supported by the RGoB’s response to the ICR (Annex 5), which emphasized positive outcomes of the project. No specific poverty analysis was undertaken after the project. Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Bank Performance Unsatisfactory Unsatisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons A lack of designated project, unit, and team management and implementation capacity can significantly affect project outcomes. The project implementation was affected by the lack of a dedicated project management team, frequent staff turnovers, and weak commitment by designated staff due to heavy work overload and lack of incentives. The implementing entity needed to ensure that there was a dedicated project management unit staffed with full‐time staff for key positions, including project management, procurement, financial management, and M&E specialists. The World Bank could help to ensure this through covenants and supervision. Supplementing ‘hard’ investments with ’soft’ investments is critical for achieving desired development outcomes, particularly where there is weak institutional capacity at the district and local levels. The project had limited scope for soft investments, as the government was unwilling to draw on credit/loan resources. The Bank’s expectation that additional grants could be mobilized for this purpose also failed to materialize. The lack of ‘soft’ investments hampered the development outcome and the efficiency of the project. Building a functional M&E system is critical to monitoring progress, identifying issues, and assessing outcomes and results and making mid-course modifications. A lack of availability of accurate M&E data significantly affected project implementation and constrained timely identification of the key issues faced by beneficiaries. Project evaluation was also hindered by the scattered, irregular, and incomplete M&E data. Selection of accurately defined outcome and results indicators, as well as relevant proxies, is critical for accurately measuring results and impacts, and for effective implementation of M&E activities. Close, frequent, and consistent Bank supervision can detect problems early on and work with the client to identify solutions. For example, close supervision would have helped identify semi-defunct irrigation systems early on, allowing for immediate repairs and avoiding unplanned but important changes in project activities (e.g. extra installation of high-efficiency irrigation), undertaken in order to maximize attainment of project objectives. Ex‐ante market assessment must underpin Bank‐financed operations that seek to increase market access or market values. Market issues, as evidenced by the project, cannot be addressed through a selection criterion that requires having existing market linkages. That is, selection of Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) groups with existing market access does not guarantee that they will not (or do not) also face market access issues. Farmers should also have ‘skin in the game’ for viability and sustainability of investments in productive assets. Additional lesson from IEG: Assumptions in the Results Chain warrant very close attention by local implementing partners, prior to implementation, to minimize design and implementation issues. The assumptions set out in the results chain were critical to the success of the project, yet almost all did not hold, and underlay many of the challenges the project faced. Given the high percentage of failed assumptions, some mitigation of project challenges may have been possible upon closer and early examination of these assumptions. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is of substantial quality. It is well written, and consistently provides evidence of claims or observations and paints a clear picture of the project. It provides insightful analysis on the background to issues that arose and uses sparse data well. The Efficiency Analysis (ICR Annex 4) is strong, with substantial detail provided and explanations of method and results, including where few project data were available. While being strongly critical of the project in places, the ICR acknowledges achievements and provides solid reasoning for its conclusions. It is frank about the Bank’s poor performance. There were, however, a moderate number of inconsistencies or gaps in figures between different parts of the ICR (often resulting from complexities of the restructuring), which required clarification in order to make proper evaluations. This compromised the quality of the ICR somewhat. The end-line survey and an ‘independent assessment’ are referred to several times but specific data from these are not presented, nor are references given to the documents. This leaves a couple of small gaps noted above, and raised questions from the borrower regarding interpretation of the end-line survey data, in an otherwise tight presentation of the evidence. Inclusion of the RGoB’s response to the ICR and the Bank’s rejoinder (ICR Annex 5) provides important context to how the project was viewed by local stakeholders (which was more positively than the overall ratings suggest, noting results in the end-line survey in particular), as well as underscoring the reasoning presented in the ICR. a. Quality of ICR Rating Substantial Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BT: Remote Rural Communities Development (P123820) Page 17 of 17