SOLOMON ISLANDS Rural Development Program Report No. 110914 – SB DECEMBER 14, 2016 © 2016 International Bank for Reconstruction This work is a product of the staff of The World RIGHTS AND PERMISSIONS and Development / The World Bank Bank with external contributions. The findings, The material in this work is subject to copyright. 1818 H Street NW interpretations, and conclusions expressed in Because The World Bank encourages Washington DC 20433 this work do not necessarily reflect the views of dissemination of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they represent. noncommercial purposes as long as full attribution to this work is given. The World Bank does not guarantee the accuracy of the data included in this work. The Any queries on rights and licenses, including boundaries, colors, denominations, and other subsidiary rights, should be addressed to information shown on any map in this work do World Bank Publications, The World Bank not imply any judgment on the part of The Group, 1818 H Street NW, Washington, DC World Bank concerning the legal status of any 20433, USA; fax: 202-522-2625; e-mail: territory or the endorsement or acceptance of pubrights@worldbank.org. such boundaries. Report No.: 110914 – SB PROJECT PERFORMANCE ASSESSMENT REPORT SOLOMON ISLANDS RURAL DEVELOPMENT PROGRAM (COFN-C1280 IDA-H3320 IDA-H6150 IDA-H8860 TF-90651 TF-97737) December 14, 2016 Financial, Private Sector, and Sustainable Development Independent Evaluation Group This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without authorization. ii Currency Equivalents (annual averages) Currency Unit = Solomon Dollars 2007 US$1.00 $7.23 2008 US$1.00 $7.23 2009 US$1.00 $7.60 2010 US$1.00 $7.80 2011 US$1.00 $7.77 2012 US$1.00 $7.25 2013 US$1.00 $7.34 2014 US $1.00 $7.19 2015 US $1.00 $7.64 Abbreviations and Acronyms ARDS Agriculture and Rural Development Strategy AusAID Australian Agency for International Development CDD Community Driven Development CDF Constituency Development Funds EU European Union ICR Implementation Completion Report IEG Independent Evaluation Group IFAD International Fund for Agricultural Development MAL Ministry of Agriculture and Livestock MDPAC Ministry of Development Planning and Aid Coordination MIS Management Information System MOFT Ministry of Finance and Treasury PCDF Provincial Constituency Development Funds PCU Program Coordination Unit PPAR Project Performance Assessment Report RDP Rural Development Program SEF Supplementary Equity Financing SIG Solomon Islands Government Fiscal Year Government: January 1 – December 31 Director-General, Independent Evaluation : Ms. Caroline Heider Acting Director, IEG Financial, Private Sector & Sustainable Development : Mr. Stoyan Tenev Manager, IEG Sustainable Development : Ms. Midori Makino Task Manager : Mr. Christopher Nelson iii Contents Principal Ratings ................................................................................................................. v Key Staff Responsible......................................................................................................... v Preface............................................................................................................................... vii Summary ............................................................................................................................ ix Lessons ............................................................................................................................... xi 1. Background and Context................................................................................................. 1 2. Objectives, Design, and their Relevance ........................................................................ 1 Relevance of Objective ................................................................................................... 2 Relevance of Design ....................................................................................................... 3 Lack of Alignment between Components....................................................................... 4 3. Implementation ............................................................................................................... 5 Quality and Selection of CDD Investments .................................................................... 7 Fiduciary aspects ............................................................................................................. 7 Environmental and social safeguards.............................................................................. 8 Poor Facilitation and Management ................................................................................. 8 Incomplete Subprojects ................................................................................................... 8 Monitoring and Evaluation ............................................................................................. 9 4. Achievement of the Objectives ....................................................................................... 9 5. Efficiency ...................................................................................................................... 17 6. Ratings .......................................................................................................................... 19 Outcome ........................................................................................................................ 19 Risk to Development Outcome ..................................................................................... 20 Bank Performance ......................................................................................................... 21 Borrower Performance .................................................................................................. 24 Monitoring and Evaluation ........................................................................................... 25 7. Lessons .......................................................................................................................... 26 References ......................................................................................................................... 29 Appendix A. Basic Data Sheet.......................................................................................... 31 Appendix B. List of Persons Met ...................................................................................... 35 This report was prepared by Christopher Nelson and Daniel Evans who assessed the project in March 2016. The report was peer reviewed by Lauren Kelly and panel reviewed by Jack W. van Holst Pellekaan. Vibhuti Khanna provided administrative support. v Principal Ratings ICR* ICR Review* PPAR Outcome Moderately Satisfactory Moderately Satisfactory Moderately Satisfactory Risk to Substantial Substantial Moderate Development Outcome Bank Performance Moderately Satisfactory Moderately Satisfactory Moderately Satisfactory Borrower Moderately Satisfactory Moderately Satisfactory Moderately Satisfactory Performance * The Implementation Completion Report (ICR) is a self-evaluation by the responsible Bank department. The ICR Review is an intermediate IEGWB product that seeks to independently verify the findings of the ICR. Key Staff Responsible Division Chief/ Project Task Manager/Leader Sector Director Country Director Appraisal Marianne Grosclaude Christian Delvoie Nigel Roberts Completion Erik Caldwell Johnson Maninder Gill Franz Dress-Gross vi IEG Mission: Improving World Bank Group development results through excellence in evaluation. About this Report The Independent Evaluation Group assesses the programs and activities of the World Bank for two purposes: first, to ensure the integrity of the Bank’s self-evaluation process and to verify that the Bank’s work is producing the expected results, and second, to help develop improved directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, IEG annually assesses 20-25 percent of the Bank’s lending operations through fieldwork. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which Executive Directors or Bank management have requested assessments; and those that are likely to generate important lessons. To prepare a Project Performance Assessment Report (PPAR), IEG staff examine project files and other documents; visit the borrowing country to discuss the operation with the government, and other in-country stakeholders, and interview Bank staff and other donor agency staff both at headquarters and in local offices as appropriate. Each PPAR is subject to internal IEG peer review, Panel review, and management approval. Once cleared internally, the PPAR is commented on by the responsible Bank department. The PPAR is also sent to the borrower for review. IEG incorporates both Bank and borrower comments as appropriate, and the borrowers' comments are attached to the document that is sent to the Bank's Board of Executive Directors. After an assessment report has been sent to the Board, it is disclosed to the public. About the IEG Rating System for Public Sector Evaluations IEG’s use of multiple evaluation methods offers both rigor and a necessary level of flexibility to adapt to lending instrument, project design, or sectoral approach. IEG evaluators all apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (additional information is available on the IEG website: http://worldbank.org/ieg). Outcome: The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently. The rating has three dimensions: relevance, efficacy, and efficiency. Relevance includes relevance of objectives and relevance of design. Relevance of objectives is the extent to which the project’s objectives are consistent with the country’s current development priorities and with current Bank country and sectoral assistance strategies and corporate goals (expressed in Poverty Reduction Strategy Papers, Country Assistance Strategies, Sector Strategy Papers, Operational Policies). Relevance of design is the extent to which the project’s design is consistent with the stat ed objectives. Efficacy is the extent to which the project’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Efficiency is the extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared to alternatives. The efficiency dimension generally is not applied to adjustment operations. Possible ratings for Outcome: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Risk to Development Outcome: The risk, at the time of evaluation, that development outcomes (or expected outcomes) will not be maintained (or realized). Possible ratings for Risk to Development Outcome: High, Significant, Moderate, Negligible to Low, Not Evaluable. Bank Performance: The extent to which services provided by the Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of supported activities after loan/credit closing, toward the achievement of development outcomes. The rating has two dimensions: quality at entry and quality of supervision. Possible ratings for Bank Performance: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Borrower Performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development outcomes. The rating has two dimensions: government performance and implementing agency(ies) performance. Possible ratings for Borrower Performance: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. vii Preface This is a Project Performance Assessment Report by the Independent Evaluation Group (IEG) of the World Bank Group on the Rural Development Program (RDP I) of Solomon Islands. In addition to reviewing the project, the mission sought to investigate the links between this project and evaluative questions being raised in IEG’s Rural Nonfarm Economy Evaluation. IEG met with a range of different stakeholders linked to the program including project coordination unit staff, project beneficiaries, government counterparts and partners, World Bank staff, other key donors and NGOs. While the focus of these discussions was the RDP, an additional set of questions was directed towards identifying important aspects of the rural nonfarm sector to inform IEG’s Rural Non-Farm Economy. Following standard IEG procedures, a copy of the draft report was sent to the relevant government officials and agencies for their review and feedback. No comments were received. ix Summary This is a Project Performance Assessment Report (PPAR) of Solomon Islands Rural Development Program (Phase I). The project was approved by the World Bank’s Executive Board on September 11, 2007. Solomon Islands is into its second decade of peaceful post conflict settlement since the Tensions of 1998-2003. The conflict resulted in significant internal displacement, a 24 percent drop in GDP over five years and the cessation of basic state functions. The causes of the conflict in Solomon Islands were complex. Inequitable distribution of natural resources, land rights disputes and frustration with local-center relations coupled with poor social service delivery undermined national cohesion. Economic recovery followed the peacekeeping and reconstruction efforts. Log exports resumed, the palm oil plantations reopened and the Gold Ridge mine restarted production, all of which stimulated economic growth. However, GNI per capita rates are yet to return to the pre-tension levels and the factors contributing to growth are tenuous. Rural development in Solomon Islands has performed poorly since the peace settlement of 2003. Social services have been unavailable, rural infrastructure has been limited and no commercial opportunities have been available to those in more isolated regions. These factors sat at the core of the demand for a rural development program that could enable recipients to engage in opportunities to improve services and to develop and exploit potential commercial opportunities. The Solomon Islands Rural Development Program (Phase I) was the Bank’s first major intervention in Solomon Islands following the Tensions of 1998-2003. The project envisaged making the government more visible in rural areas of the archipelago and sought to re- energize commercial activity and opportunities for agricultural producers. The project’s objective was to take a community driven development (CDD) approach to supporting rural infrastructure and to put in place the necessary structures for CDD to be a useful mechanism for future development. This was a substantially relevant objective in terms of the World Bank’s assistance strategy and the Government’s development strategy. However, the project’s design was overly complex and ill suited to the challenges of this diverse country in geographic, social and economic terms and therefore rated as modestly relevant to the project’s objectives. The project was always intended to progress to additional phases, both to broaden the reach of the CDD component in areas not covered by Phase I and to develop future commercial opportunities. The RDP’s objective, “to raise the living standards of rural households by establishing improved mechanisms for the delivery of priority economic and social infrastructure and services” was substantially achieved. Despite shortcomings in the preparation and design of the project, the design team undertook an extensive participatory planning exercise to determine community level infrastructure investments. These investments were then rolled out to the designated communities in an organized fashion with a robust system for quality control and the allocation of materials. In addition, the Ministry of Agriculture and Livestock was re-energized by the institutional assessment and subsequent training and support x provided by RDP. In particular, nascent extension activities were directed at regions with potential agricultural markets and the capacity of staff was developed with targeted technical assistance for popular commercial crops to improve productivity and yields. The Supplementary Equity Fund also contributed to emerging opportunities by providing expansionary financing opportunities to companies that were prepared to engage in commercial activity in various isolated regions. The efficiency with which the project was implemented was assessed as modest given significant shortcomings. No overall economic rate of return meant relying on infrastructure comparisons to arrive at a judgment on the CDD element’s cost effectiveness. The cost effectiveness calculations for capacity building and extension activities for component two had methodological limitations and limited information on component three also made an accurate efficiency assessment problematic. The overall outcome rating for RDP Phase I is moderately satisfactory. World Bank performance is rated moderately satisfactory. RDP was built on a strong analytical base that responded to the needs of post-conflict environment. It was supported by a range of donors and drew on the lessons from coming out of the World Bank’s previous project, the Community Development Scheme which also funded small-scale infrastructure. The project was responsive to the nuances of working in a difficult geographic location and the participatory approach was consistent with effective programs delivered in other post- conflict environments. In addition, the operational decisions were complementary to the challenges of working in the constrained capacity of a small state. Overall borrower performance is rated moderately satisfactory with a range of shortcomings. Government performance was moderately unsatisfactory with delays in the provision of funds for the project which had consequences for the poor completion of infrastructure projects. In addition, while the Ministry of Finance was supportive of the World Bank’s role, they were ‘hands off’ during preparation and implementation and there was only limited effort to take ownership of RDP which had consequences for the sustainability of the project in the longer term. The performance of the implementing agency is rated moderately satisfactory. There was a sensible management structure put in place and regular, intense and thoughtful responses to improving performance were made as the project progressed. Given the moderately satisfactory outcome rating, the overall rating for borrower performance is rounded up to moderately satisfactory. The risk to development outcome for RDP is moderate. While the country continues to be constrained by limited capacity and geographic challenges, the political situation is relatively stable and the fiscal environment has improved. Importantly, a successor project is underway. It has drawn on many of the lessons from RDP I, particularly around the important link with the development of markets and the commercial sector. The combination of continued support to the CDD model and the enhancement of commercial opportunities in rural areas should ensure the benefits of RDP are sustained. xi Lessons CDD projects in fragile and conflict states face difficulties when they try to be ‘all things to all people’. RDP was a particularly complex project given the split between the infrastructure and service delivery support, the capacity enhancement of the Ministry of Agriculture and Livestock and the supplementary equity facility. The acknowledged temptation with RDP was to design and institute a project that sought to ‘fix everything’ in the rural space. This limited the impact as the implementation team struggled to keep the various aspects on track while identifying the necessary points of integration to ensure maximum benefit to those living in rural areas. CDD projects in island states are more effective when they recognize the additional support required and costs involved in working in a limited capacity environment. The experience of RDP showed that small island nations need to be treated differently. The RDP required a project design that recognized the complexity of not only a post conflict, low income, low capacity country, but more importantly, a country composed of isolated islands with high costs and complex logistics. Lessons from traditional CDDs, even those of archipelagic countries such as Indonesia are difficult to transfer to a context such as Solomon Islands. The World Bank has few TTLs that have worked in an island context and they need to be thoroughly aware of the constraints imposed on them when working in poorly serviced islands with extremely limited development opportunities. Appropriate project design is challenging in diverse and difficult geographic locations and is more effective when there is sufficient recognition of the different needs of commercial hubs versus the needs of isolated rural areas. RDP was delivered in hugely varied environments, some with good access to markets and others that were isolated with few options for commercial enterprise. In addition, there were semi-urban areas made up of heterogeneous, itinerant communities where the participatory planning approach could not gain traction. Having a uniform CDD model is necessary to ensure consistency and fairness in how projects are delivered, but a more bespoke approach to how the infrastructure and service opportunities are applied will enhance the effectiveness of the CDD model in geographically diverse locations. In addition, the slow maturation of RDP highlights the importance of having phased project based engagements in fragile states given the weak capacity and considerable time required to set up the necessary systems and support structures. It would appear clear that without a follow up project to RDP, the risk to Development Outcome would have been high. Capacity building and institutional reform in weak ministerial environments is more effective when it is appropriately targeted to the change expected within the partner organization. Much of the success of the institutional aspect of RDP can be linked to the acknowledgement of the limitations of the Ministry of Agriculture and Livestock (MAL) and the strategic decision not to use this ministry as the implementing partner. RDP’s sole support to the ministry was through extension support and technical assistance which ensured better outcomes for farmers. This left oversight for the project to the Program Coordination Unit and Ministry of Development Planning and Aid Coordination, who were better able to cope with the operational realities of a complex CDD project. xii Rural enterprise development is difficult and is more effective when it uses a combination of approaches. RDP spent considerable time and effort on improving extension activities, particularly on products that were cash convertible for rural producers. However sufficient effort was not placed on understanding the nature of rural development in determining the approaches. A better understanding of the nature of commercial opportunity and the potential growth opportunities for certain crops will be an essential corollary to enterprise development in RDP II. Women tend to be included overall in CDD projects, but they are more successful when sufficient emphasis is placed on women gaining access to the same level of productive activities as men. Information is lacking on whether female rural enterprise owners were identified, supported or targeted in RDP activities as they should have been given the commitment to advancing the interests and livelihoods of women. Therefore, while RDP projects were saving women time and they were involved in various trainings and activities, the question remains as to whether the project was contributing to transforming their roles in a sustainable and substantial manner. CDD projects have a long track record of making valuable contributions to the lives of women and this was certainly intended for RDP. However, because of limited disaggregated information on the role of women both in terms of traditional and non-traditional roles, it is difficult to get a sense of how women have fared in Solomon Islands because of RDP. Therefore, greater emphasis on how women can and do contribute to productive activities is necessary to ensure that projects reflect the diversity of its participants. The monitoring and evaluation of phased programs is more effective when it reflects, and is responsive to the capacity of the implementing agency. RDP spent considerable time, budget and energy in formulating a management information system that was not suitable for what they needed to track, nor helpful in supporting the participatory planning work that was essential to the infrastructure component. RDP needed M&E that was pragmatic in scope, scale and usability, but ended up with an overly technical platform that did not help in the tracking of essential information. As a result, much of the important work done in the communities was either lost or poorly archived, preventing the second phase from building on the lessons of various metrics recorded in the first phase. Stoyan Tenev Acting Director Financial, Private Sector & Sustainable Development 1 1. Background and Context 1.1 Solomon Islands is into its second decade of peaceful post conflict settlement since the Tensions of 1998-2003. The conflict resulted in significant internal displacement, a 24 percent drop in GDP over five years and the cessation of basic state functions. Since 2003, the country has remained largely peaceful aside from a period of unrest in Honiara’s Chinatown that followed the 2006 elections. The causes of the conflict in Solomon Islands were complex. Inequitable distribution of natural resources, land rights disputes and frustration with local-center relations coupled with poor social service delivery undermined national cohesion. Economic recovery followed the peacekeeping and reconstruction efforts. Log exports resumed, the palm oil plantations reopened and the Gold Ridge mine restarted production, all of which stimulated economic growth. However, gross national income per capita rates are yet to return to the pre-tension levels and the factors contributing to growth are tenuous. 1.2 Despite the improvement in economic conditions following the conflict, development in the rural economy was disappointing. For the population in more isolated regions, social services were lacking, rural infrastructure was limited and no commercial opportunities were available. The gap in agricultural support services, low productivity, inadequate access to credit, and weak transport facilities and infrastructure were all factors that stimulated demand for a rural development program that could enable recipients to engage in opportunities both with regard to improved services and potential commercial opportunities. The rationale for assistance from the International Development Association (IDA) was outlined by the Ministry for Development Planning and Aid Coordination (MDPAC) based on the country’s national strategy for long term agricultural and rural development (the Agriculture and Rural Development Strategy ARDS - https://openknowledge.worldbank.org/handle/10986/7757?show=full ). The World Bank was asked to support the government in the preparation and implementation arrangements for the Rural Development Program (RDP) to respond to aspects of the ARDS. The program built on commitments outlined in the World Bank Pacific Regional Engagement Framework 2006- 2009, which sought to address root causes of conflict in Solomon Islands and in particular, to “provide for rural communities in different provinces that require better access to social, economic infrastructure and services, and the promotion of income-generating activities”. Importantly, the project also sought to utilize donor funds (AusAID, the EU and IFAD) in a coordinated fashion to respond to the needs of rural communities. The RDP was always intended to be a multi-phase project with a firm commitment to a second phase should the preliminary work be judged successful in establishing the foundation for longer term support to rural populations. 2. Objectives, Design, and their Relevance 2.1 The Objective of RDP in the Financing Agreement (September 24, 2007) was: “To raise the living standards of rural households by establishing improved mechanisms for the delivery of priority economic and social infrastructure and services” through: 2 (i) increased, cost effective and sustained provision of local services and basic infrastructure determined by participatory planning prioritized by villagers; (ii) increased capacity of agricultural institutions to provide demand-driven agricultural services at the local level; and (iii) support for rural business development. 2.2 In the Second Additional Financing Project Paper (August 30, 2013), the wording of the Development Objective was changed. In place of: “To raise the living standards of rural households by establishing improved mechanisms for the delivery of priority economic and social infrastructure and services", the phrase “To increase access to high priority, small- scale economic and social infrastructure, agriculture and financial services in rural areas” was substituted. This left the project with a less specific measure of the outcome, but this was justified by the in-country team on the basis that the collection of baseline data on living standards was not possible and measuring it effectively over time was going to be too difficult given the logistical and budgetary constraints of tracking the project’s performance. Certainly, the intention of the project was to see a positive impact on welfare, and thus we continue to see this as the reach objective throughout this PPAR. 2.3 The project had four components: local infrastructure and service delivery (actual $19.7 million); improved agricultural services (actual $9.1 million); rural business development (actual $1.2 million); and program management (actual $7.5 million). As it turns out, the first three of these components mirror the three sub-objectives that are investigated and evaluated in the efficacy section (section 4) of this review. Relevance of Objective 2.4 The RDP project aligned both historically with the World Bank’s Regional Development Strategy at implementation and with the World Bank’s Country Program Strategy for FY2013-17. The project directly supported the two strategic areas of the CPS, namely ‘strengthening economic resilience’ and ‘improving public service provision’ as well as contributing to the expected CPS outcome of increasing, “capacity of collective action” and “access to services for rural communities”. Similarly, there was an evident fit with Solomon Island’s inaugural Interim Strategy Note (FY 2010-11), contributing to addressing the World Bank’s stated medium term development objective – outlined under strategic area three of the ISN (‘supporting improved public administration and management’) – of assisting service delivery, specifically rural service delivery. The project was explicitly referenced in both the ISN and the CPS. 2.5 The project’s objective also responded to a number of pre-existing and contemporary government policy objectives relating to rural development. Since the independence of Solomon Islands in 1978, successive governments have prioritized polices of decentralization, pursuing a variety of sub-national growth strategies that have been articulated in numerous policy documents. However, considerable constraints of a structural, institutional and geographic nature, combined with limited capacity, have typically meant that governments have been unable to realize their stated goals in this area to date. 3 2.6 Most recently, the incumbent government (the Democratic Coalition for Change) has continued the path established by its predecessors. Its policy statement indicates it will, among other things, facilitate “... development of rural infrastructures and utilities”, will “[e]ncourage indigenous entrepreneurs to participate in income generating activities in… rural areas”, and will see the establishment of micro-financing initiatives across Solomon Islands (Office of the Prime Minister and Cabinet). 2.7 The government’s objectives are in addition to the goals detailed in the country’s ‘National Development Strategy 2011 to 2020 (Ministry of Development Planning and Aid Coordination 2011). The strategy deals extensively with rural development. The project supported several country development objectives, including improved rural infrastructure (with a particular focus on water and sanitation), increased production in agriculture, and strengthening agricultural support services (objective 7, 1a.1-1a.5). Initiatives are to be carried out in a manner so as to, “[b]uild community development capacity and encourage communities to take ownership in rural development programs and projects” (objective 7, 1a.6). 2.8 Sector specific policies to which the project has contributed include the ‘National Agriculture and Livestock Sector Policy 2009-2014’ implemented by the MAL (Ministry of Agriculture and Livestock 2009). In particular, its’ emphasis on strengthening rural extension services through converting, “extension services into an instrument of agricultural growth” and advising, “farmers on improved technology to stimulate sustainable agricultural growth”. Similarly, its successor policy (spanning 2015-2019) is equally directed towards improving demand-driven and sustainable agriculture extension services, a key objective of component two of the RDP project. Given this consistent alignment with both the country’s own development priorities and the World Bank’s strategies, the relevance of RDP’s objective is rated High. Relevance of Design 2.9 According to the project appraisal document (PAD), the project was designed to be implemented in two phases. The first five-year phase was expected to test and gradually scale-up the project approaches. The second phase of the project (years 5 to 10) was designed to, “focus on further scaling up, consolidation and institutionalization of … innovative approaches”. RDP II is now underway. Provinces were gradually added during phase one, with the addition of the final province, Renbel in 2015. This completes the rollout across the country. 2.10 Component one of the project was designed as a community-driven development approach. According to the PAD, the local infrastructure and service delivery component of the project sought to implement, “participatory planning, budgeting and execution mechanisms at community and provincial level” (World Bank. 2007. Solomon Islands - Rural Development Program). The PAD indicates that, “[c]apacity building activities would target key stakeholders including provincial administration staff, facilitators, ward councilors and other community representatives, provincial assembly members, SICS [sub-implementation committees] and contractors and service providers.” (World Bank. 2007. Solomon Islands - Rural Development Program) The PAD explains that existing local governance structures, 4 such as tribal councils and churches, “would be utilized for planning and management of RDP activities within the community and for arranging implementation of small works, where suitable” (World Bank. 2007. Solomon Islands - Rural Development Program, 28). The PAD also indicated that, “improved accountability and responsiveness of the elected government to community needs is at the heart of the governance objective of the RDP” (World Bank. 2007. Solomon Islands - Rural Development Program, 28). To further this aim, the PAD stated that it would include existing local representatives (elected ward councilors) in project decision-making processes. This consideration for the contextual issues central to Solomon Islands, in addition to the understanding and need for community outreach was a sensible and timely approach for RDP. The community engagement was building on existing communal structures and was closely aligned to the way many villages in rural areas were already functioning either through church or community councils. 2.11 Component two of the project sought to improve agricultural extension services in rural areas. The specific objective detailed in the PAD was to improve the access of, “smallholder households to quality agricultural services to support rural income growth.” This was principally to be achieved through building the capacity of service providers, including the MAL. The PDO envisaged a role for service providers such as nongovernmental organizations (NGOs), churches, rural training centers and the private sector. Under this component, annual allocations would be assigned at the provincial level to finance agreements with service providers to deliver agriculture and related services to smallholders. This component would also build the capacity of MAL to facilitate and supervise provincial service delivery, provincial service providers and rural households. Sensibly, the component built on existing structures and sought to stimulate and support the status and needs of rural areas. Component 3 was designed to complement this support to farmers by enhancing the opportunities for businesses operating in this space through financing for expansion. This aspect of the project was designed to support longer-term sustainable aspects of grower opportunities in respective sectors and ensure there was demand for any increased output building on the extension activity. Component four was simply a project managerial function. Lack of Alignment between Components 2.12 No evidence shows that the separate project components were in any way aligned, or well integrated in the original design. The components of the project have largely operated in silos with staff working on the different components being unaware of each other’s roles and activities. This situation, in part, stems from the project design documentation which did not articulate how, if at all, the various components of the project were to link together. Each of the components were essential to the project’s intended outcomes, but the lack of design protocols and approaches articulating how, where and why they should be linked made integration difficult and resulted in the project team referring to ‘two projects in one’. While in principle, there was nothing wrong with the various component’s contribution to the outcome, the lack of sufficient detail and direction on how the different parts should be integrated made it difficult to see how the links were being made. Integration is a fundamental aspect of CDD programs developing and maturing to ensure greater engagement with government services and to enhance income generation opportunities. While perhaps not essential in the early stages, the lack of foresight on how the various parts of RDP fit 5 together is likely to have ramifications for the program in the longer term. Thus, while the over-arching premise behind RDP was sound and the theory of change is consistent with the trajectory of a CDD across multiple phases, there were moderate shortcomings in the design resulting in a rating of Moderate for the relevance of design. 3. Implementation 3.1 As a new CDD program in a difficult environment recovering from internal tensions, the RDP approach was successful in getting services into many regions. In particular, support was broad for the small scale infrastructure investments that utilized participatory approaches and garnered community support which increased government visibility in targeted areas and improved village level infrastructure. Similarly, there was significant uptake of the agricultural extension activities and MAL benefitted considerably from the injection of funds and support in a previously problematic ministry struggling to get its officers into the field. IEG’s conversations with MAL officials and staff highlighted the open and transparent nature of Bank support, in addition to the strategic arrangements around how capacity would be improved through the project. In this way, both project components had traction with development partners from the beginning and concerns about competing interests between the MAL and MDPAC did not emerge. 3.2 The operational structures for the program took considerable time to set-up and operationalize, but project documentation and discussions with project staff indicated that they proved robust. The Project Coordinating Unity (PCU) had a solid foundation on which further program support could be built, which was evident in the adjustments and reforms instituted in RDP II which is now underway. However, there is little evidence of sustainable structures being embedded in the responsible government ministry (MDPAC). Evidence of cohesion in the operational teams was found. They have learned about the delivery of services throughout RDP I and are continuing to improve in the early stages of RDP II. But in the interviews conducted by IEG with MDPAC staff, there was still limited understanding of how the project could operate effectively in the future both with regards to the activities themselves and in terms of tracking achievements and impacts. There was little evidence that either MDPAC or MAL had developed sufficient competency to oversee procurement arrangements, contracting, staff development or strategic oversight of the CDD process without the ‘hands-on’ involvement of a designated PCU. 3.3 Component one which focused on local infrastructure and service delivery was the strongest of the various components and took up the majority of time and resources. Components two and three, which were more ambitious given the status of the country’s agricultural sector and limited business investment, were less successful in delivering development benefits. In addition, both of these components were somewhat divorced from the other parts of the project due to their specific needs. Many of the respondents interviewed for this review referred to RDP as ‘two or three projects in one’ – the first focused on a classic CDD approach of community level service delivery and construction; the second and third entailed a rebuilding of MAL’s extension services with an outreach aspect focused on business level support. While there was good justification for all three aspects of the 6 program, the mandate’s ambition stretched both the project team and the government’s absorptive capacity. 3.4 Other implementation issues arose during conversations with stakeholders and during the focus group discussions that illustrate the project’s performance. First, with RDP II underway a greater focus on commercial opportunities and business engagement, it is worth reflecting on the limitations of the Supplemental Equity Facility (SEF) in RDP I. The IEG review team encountered skepticism amongst many respondents with regards to the economic returns of the enterprises that were equity partners through the SEF. While some partners were considered good prospects, conversations with business investors indicated that the developmental side of these investments will be slow in the current environment. These investors acknowledged that donors and the government were not cognizant of the commercial risks of these groups failing nor of the very limited opportunities for growth. Greater emphasis on developing both the analytical base and the mechanisms to accurately track its’ impact would have been useful in responding to the project’s needs and ensuring greater awareness of what was working and what was not. 3.5 Second, the project did little to appropriately track the range of benefits of the participatory CDD approach and the additionality that came with using it for community cohesion. Feedback from the focus group discussions indicated that there was strong support for the participatory decision making process, but respondents were not sure how this could be enhanced or utilized in the future. Thus, while the approach resulted in project goodwill, participants did not feel there would be sufficient on-going support to warrant the maintenance of the community groups over time. These groups did not associate the work with the provision of regular services from either the government or the project, nor with productive opportunities and this appears to have been a missed opportunity during the implementation phase. 3.6 Third, there are considerable differences in the needs and requirements of the different regions in Solomon Islands. The generic nature of the RDP project might not have been appropriate given the variety of needs and opportunities. Considerable differences arose in how component two was rolled out across the islands, in particular, the number of attendees and the volume of training sessions offered were of some concern given the limited effectiveness of agriculture services in many places. A more targeted strategy for different islands may have been more useful for agriculture and extension services and certainly for the development and distribution of viable business funding. 3.7 Fourth and finally, RDP appears to have been well run on the World Bank side with regular and informed adjustments in response to the limited capacity in Solomon Islands. The World Bank team and the PCU sought to respond to procurement challenges by finding more efficient ways to get goods to project sites and finding work-arounds on logistics. Most importantly, the team learnt from the limitations of the procurement tranche release in RDP I which caused logistical and timing problems and has arranged either a two tranche or one tranche total release policy for RDP II. This is much more appropriate for the dispersed island project sites where transport is expensive and is a good example of the team solving and responding to project issues. 7 Quality and Selection of CDD Investments 3.8 The subprojects IEG visited during the mission were well-constructed with beneficiary communities were positive about the final product. Most projects were matched to the community’s priorities and expectations, indicating that the subproject selection process did respond to the community dialogue. However, in a number of instances, the choice of subprojects was questionable with seemingly insufficient regard to the acquisition of economically productive assets, infrastructure that would be irregularly used, and recurrent financing and/or the placement of national-level staff issues being ill considered (e.g. school buildings, clinic staff housing). This suggests potential flaws with the subproject facilitation process, particularly in light of the emphasis on improving living standards of recipient groups. 3.9 In discussions with the PCU, there was little evidence of adequate critical reflection on the approach to facilitation. This was reinforced by discussions with Solomon Islands Development Trust (SIDT) which was procured in the early stages to train the community facilitators. They acknowledged that the initial workshops were a “one-off” intervention and did not adequately prepare inexperienced field officers for the complex task of taking the communities through the decision making process. Thus, while the process was participative, it was not necessarily successful in developing an iterative cycle of decision making that would help the communities make effective and informed livelihood decisions. The process was more reflective of communities choosing from a designated list of options, rather than illustrating an iterative discussion of contesting and discussing various options and arriving at a consensus agreement. This was perhaps unsurprising given the ‘newness’ of the approach, but seeing a more comprehensive plan for developing the facilitators’ skillset over time would have given IEG confidence that sustainability had been appropriately addressed. Fiduciary aspects 3.10 Procurement related problems occurred during implementation. In numerous instances, delays in the delivery of goods caused significant delays in subproject completion. This situation was linked to an iterative procurement and delivery process that sought to purchase materials in bulk lots and minimize transport costs by clustering shipments together. Various communities were critical of this aspect of the subprojects arguing they had no role to play in procurement and the responsible community helpers often took too long to act on specific needs. At one subproject visited by IEG, this lack of involvement meant purchased materials were not fit for their intended purpose and a substantial portion of the project budget was wasted. In three of the other subprojects we visited there were leftover materials and/or budget shortfalls due to additional expenses related to the procurement process. Much of this was not surprising given the challenges of logistics in Solomon Islands, particularly for a new program engaging in a range of different investments across the archipelago. However, it would have been good to observe greater utilization of the lessons and the recording of procurement problems to enable more informed managerial responses. 3.11 In IEG’s discussions with the PCU, staff were aware of various issues regarding procurement, but there was no consistency amongst the Bank supervision team as to what responses worked, how they were applied and why they were necessary. In addition, the 8 observations had not been recorded formally in the monitoring and evaluation (M&E) system or oversight reports. It was impossible to adequately discern the various steps taken to tackle intractable procurement issues. With regards to financial management, annual audits were completed (though late for the first half of the project) and outsourced to a private firm in the latter stages given capacity constraints in the Attorney General’s Office. Environmental and social safeguards 3.12 The project at appraisal was assessed as a category “B” project with the following four safeguard policies triggered: Environmental Assessment (OP/BP 4.01), Forests (OP/BP 4.36), Natural Habitats (OP/BP 4.04) and Indigenous Peoples (OP/BP 4.10). A fifth safeguard, the Pest Management (OP 4.09) policy, was triggered due to the use of pesticides for farming under Component 2 following the restructure. An Environmental and Social Management Framework (ESMF) and a Pest Management Plan (PMP) were successfully adopted in 2011 following the triggering of OP 4.09 and were complied with. Other minor social safeguard issues that came up in the regular assessments, including disputes over the supply of inputs and inter-ward differences, and this led to the cancellation of a number of small subprojects. In addition, it was challenging to ensure that all subprojects were ESMF compliant which caused delays due to the dispersed nature of projects and the considerable expense in getting specialized staff to do the work. A spot assessment approach has been adopted as a more appropriate means of reviewing projects in RDP II. Poor Facilitation and Management 3.13 The project had difficulties sourcing and retaining qualified staff. While this challenge was also unsurprising given the capacity constraints in Solomon Islands, this aspect of the project should have been more carefully appraised with training and support being integrated from the beginning. Based on the visits to the field and discussions with the PCU, the quality of the community helper facilitation was variable. IEG encountered instances where there was poor communication with sub-implementation committees (SICs) by Community Helpers, especially around the transfer of project funds and their explanations of the process of facilitation was often out of line with traditional CDD practices. Again, the World Bank team was responsive to these issues and additional support to the Community Helpers was provided at various times. However, no coordinated strategy was developed for building their capacity. Moreover, the initial training was not supported by regular professional development opportunities that looked at how their skills were being improved through their work and how the facilitation role could be adjusted and improved. One of the reasons cited for this shortcoming was the expense of getting the Community Helpers into Honiara for training and support. A centralized approach had been tried at different times, but the cost became an intractable barrier to this model. A more decentralized system was being looked at for RDP II, but it would have been a useful element in RDP I. Incomplete Subprojects 3.14 As of March 2016 various subprojects visited, including several from phase I, had remained incomplete. In some instances, the subprojects were close to completion but were sitting dormant, requiring only a relatively small amount of money to finish the necessary 9 work. However, in a number of instances IEG found that neither the project team nor the community were able to do anything to rectify this situation given the various procurement rules relating to final budgets. Delays were a consequence of various implementation factors. First, the budgetary ceiling meant insufficient matching funds and final inputs were often missing and could not be covered by the program. Second, increased costs and project adjustments during construction also put pressure on budgets which had a negative effect on finalizing work. Thus, the effectiveness of various investments was limited due to operational constraints that should have been overcome with a ‘work-around’. Monitoring and Evaluation 3.15 The implementation of the M&E system was slow and incomplete, particularly with regards to the undertaking of baseline assessments, the setting up of the management information system (MIS) and the utilization of data. As explained in the M&E section, the arrangements for M&E were overly ambitious and this affected the implementation arrangements. The scoping and preparation of the baseline work took too long and was not appropriately adjusted to reflect the emphasis on pragmatic systems suited to those in the PCU and in the Ministries. The MIS system developed was also inappropriate for its intended purpose with shortcomings in terms of its utility and its management of the project data sets. It did not adequately capture and track the kind of results needed for evaluating a CDD project and has had to be completely redesigned for RDP II. The IEG team learned from the project’s first phase and the new RDP II that the interface is much improved, but the lack of urgent adjustment to the M&E system during RDP I was disappointing. It resulted in a poorly integrated and costly system that did not give the program management what it needed to make implementation improvements. 4. Achievement of the Objectives 4.1 RDP was designed in a period of post-conflict recovery to help achieve improved livelihoods for rural households. About 80 percent of the population in Solomon Islands’ reside in rural areas, according to the 2009 census. The original project development objective was to, “raise the living standards of rural households by establishing improved mechanisms for the delivery of priority economic and social infrastructure and services through: (i) increased cost effective and sustained service provision of local services and basic infrastructure determined by participatory planning prioritized by villagers; (ii) increased capacity of agricultural institutions to provide demand-driven agricultural services at the local level; and (iii) support for rural business development ”. This was revised following the second additional financing in 2013 as follows: “to increase access to high priority, small-scale economic and social infrastructure, agriculture and financial services” with the same three identified interventions. 4.2 This review has chosen to treat the notion of ‘raised living standards’ as a high level over-arching objective beyond the scope of the project. This leaves the emphasis, as an objective, on “establishment of improved mechanisms for the delivery of priority economic and social infrastructure and services” as the primary focus which for the purposes of this review has been parsed out into the three parts that align to the three sub-objectives identified 10 in the original objective. The revision of the objective in 2013 effectively lowered the ambition of the original objective replacing improved ‘living standards’ with ‘access to services’. Rather than undertaking a split evaluation, this review has chosen to prioritize the “establishment of improved mechanisms for the delivery of priority small-scale economic and social infrastructure and services” as measured against the three sub-objectives across the whole time period. The rationale being that this best reflects the intention of the project and ensures emphasis on the importance of establishing participatory mechanisms for getting better infrastructure and services to villagers in rural areas. Sub-Objective 1: Increased, cost effective and sustained provision of local services and basic infrastructure determined by participatory planning prioritized by villagers. 4.3 Two key PDO indicators were identified in the PAD (“number of people using agricultural and infrastructure services provided” and, “percentage of rural households satisfied with the quality of agriculture and infrastructure services provided”). These were subsequently modified given the lack of appropriate baselines at the household level and to better align with the infrastructure focus and construction at the community level. The changes to the indicators were made when the project received additional financing and the PDOs were changed (“percentage of villages with satisfactory access to functioning infrastructure” and “percentage of villages with improved access to effective agriculture services”). The first of the revised indicators covers the CDD aspect of RDP and is used as the measure of efficacy of this sub-objective. The second indicator is aligned to the second sub-objective on support to MAL. 4.4 While prior to the restructuring the focus was on the utility of the infrastructure and services, the revised indicator focused on access to the infrastructure and services because the emphasis on construction in the first phase and its slow roll out. While the revised indicator was more realistic given the early stage of the project cycle and the limited time available for communities to utilize the improved services, the indicator only provided a shallow view of the mechanisms by which the infrastructure and services were developed and instituted. It is hoped that RDP II’s emphasis on community use of the infrastructure and services will provide more detail on how the CDD approach might support a move to the provision of productive investments in the future. The CDD Approach 4.5 Based on the focus group sessions with the targeted communities, IEG determined the groups involved in RDP successfully grasped the CDD approach. IEG’s mission revealed an in-depth understanding of the participatory elements of the project and communities were proud of the facilities that had been built. Most respondents were able to articulate the process of decision making, the approach to procuring inputs and the way the community had been involved (including the provision of in-kind support). The discussions highlighted ways in which the approach was aligned to community governance and organization, and there was a legitimate consensus amongst groups that decision-making on service provision and infrastructure was well received. 11 4.6 However, the success of the CDD approach was not universal and was geographically variable. In semi-urban areas which are marked by heterogeneous and transient communities, the SIC/CDD process did not work. In these instances, the communities acknowledged the need for community cohesion, but the SICs rarely met or met only once or twice and there was no community contribution was made towards subprojects (including the provision of in- kind labor). This was evident in Noro and Western Province, where the SICs had not functioned and no community contribution to projects had occurred (adding to project expense and seeing projects not finished on time). Infrastructure 4.7 The PAD set an ambitious objective of 300,000 beneficiaries for, the “number of people using agriculture and infrastructure services provided” (equivalent to over half of the country’s population on official 2009 data). While there was some movement during the project as to how this figure was to be calculated (ie. a combination of components one and two, or component one alone), the final outcome based on data collected by the project showed a shortfall. At project closure, 198,340 people were benefiting from completed infrastructure subprojects. This achievement was undermined by a sizeable number of subprojects being incomplete at closure (34 percent) and a significant number remaining so as at March 2016 when the IEG field team reviewed the works. Of the 370 planned subprojects, 283 were completed and 77 were identified in the ICR as ‘nearing completion’. During IEG field visits, IEG observed a number of ‘near complete’ projects, often with only minor issues that needed to be addressed. However, when IEG approached the PCU to follow up on the status of the remaining projects post-ICR, appropriate documentation outlining the remaining works was unavailable. The PCU had not put together an appropriate audit of remaining works and with the activity associated with RDP II, the evaluation team is concerned that these incomplete projects are not getting the focus they deserve. In follow-up conversations with the Program Manager, he acknowledged discussions were on-going with regards to funding outstanding works and dealing with project ceiling caps. Thus given the short gap between the PPAR and project completion, the sustainability of investments is difficult to gauge. 4.8 Evidence central to measuring the progress of RDP against this objective relies heavily on the impact evaluation1 completed in 2013. The evaluation found there were considerable improvements in accessing services for communities involved in RDP. Travel time to improved water sources was reduced by an average of 13.5% following RDP investments. Similarly, 71% of households reported having satisfactory access to water against a baseline figure of 33%. Similar results are reported against other infrastructure investments, with notable improvements in access to roads where 49.4% of villages indicated satisfactory access compared to 32% in the baseline survey. On access to markets, there was a 40% increase in the proportion of villages with satisfactory access to markets and this result was replicated in IEG’s field visits where focus group discussions highlighted enhanced viability of certain cash crops. Most importantly, the evaluation found that where villages were most dissatisfied with a particular service the likelihood of investing in this area was 1 http://documents.worldbank.org/curated/en/926411468294672780/Evaluation-of-the-Solomon- Islands-rural-development-program 12 high. For example, villages choosing education investments had almost 27 % lower satisfaction with the level of service versus those villages choosing alternative investments. 4.9 Both the responses in the project’s beneficiary survey and in IEG’s own field visits indicate that the quality of the infrastructure investments was high. In community level discussions with IEGs evaluation team, recipients expressed their satisfaction with the investments and IEG’s field observations confirmed that projects were completed either at or above the specifications outlined in the community level plans. In addition, the quality of the investments was considerably above comparable village level facilities funded through government funds or similar church-based facilities in the same villages or in regions outside the scope of RDP I interventions (the notable differences included fitted windows, insect wire, appropriate ventilation and acceptable elevation in flood prone areas). At two of the sites visited, the school construction had enabled the communities to meet Ministry of Education specifications for an upgrading of the school’s year level certification (meaning they could increase the year-levels attending). At the visited sites, water tanks and piping was still in fully functional condition and sanitation facilities had been attached to both existing and new infrastructure. Housing for health workers had led to increases in the number of staff posted to certain designated villages, and this had dramatically increased the provision of health services in those places. Importantly, in one village the arrival of a female maternal health care nurse had resulted in almost 100% coverage of health care assisted births, from rates previously below 20%2. 4.10 Participation in the community decision making process was another priority of the CDD model of RDP. The baseline household surveys for the project found that 48 percent of households in targeted planning units participated in sub-project decision making (ICR p30) which was well below the target of 80%. However, in both the project impact evaluation and in the focus group discussions held by IEG, this target was viewed as unrealistic. The structure of community inter-relations and the considerable commitments of community members to their own agricultural activities made it impossible to gather members even for important village functions. Given these issues, the almost 50% participation rate did not look quite so disappointing. In field discussions with four heads of the SICs, they explained that families/households would often have informal proxy representation at the participatory sessions or alternatively ensure that a trusted representative was presenting their views. This was reinforced in a number of the villages where we spoke with community members that did not take part in the discussions, but felt that the investments were representative of the majority view. 4.11 With regards to other community decision making data, the target for participants was achieved just exceeding 30,000, but the proportion of female participation was lower than expected at just under 10,000 with a conservative target of 12,000 not met. This was a disappointing result, but again reflects the considerable commitments women have in farming and other household duties. The project will need to look at ways to respond to this under-representation in the future. The target of 15% for community contribution was exceeded with a 36% figure, but this was largely the result of labor and in-kind support to project construction activities. In the majority of cases where projects remained unfinished 2 Central Province Village II visit 13 the community contribution was small, the task overly technical or funds allocated were too small for the scope of the project. Cash contributions were rarely available except in the few instances where the project was a jointly funded undertaking (with other government or donor funding) or where local politicians provided a goodwill contribution. The inference in the ICR (p30) that the financing was indicative of strong links to provincial government funding, ward member investments or contributions from national Members of Parliament was not confirmed in the discussions IEG had with SICs and communities. Gender 4.12 The broad involvement of women in the CDD process was low, as only 33% of reported beneficiaries were female. However, feedback from the focus group discussions indicated that many of the infrastructure investments provided considerable direct benefit for women. The 13.5% decrease in travel time to access an improved water source identified in the impact evaluation was of primary benefit to women who are generally responsible for water collection. Likewise, women in the focus groups reported particular satisfaction with the maternal focus of improved health services. Both of these services were the dominant infrastructure investment in communities and therefore there was some legitimacy to the claim in the ICR that female beneficiaries were underreported. This does not negate the need for the project to look carefully at the low figures for women’s participation in the planning process and to emphasize greater participation in the SICs in RDP II where only 29% of representatives were female in the first phase. 4.13 In summary, while evidence regarding the achievement of the first objective was limited by the lack of any kind of household data, there was evidence in both the beneficiary surveys carried out by the project and from the focus group discussions undertaken for this review that the lives of recipient communities were positively affected by the infrastructure and service investments. As outlined above, travel times improved for water collection, government staff were now available as a consequence of investments in housing and improved education and health facilities were introduced in communities. There was reliable potable water available to communities, an expressed pride in the infrastructure facilities built and stakeholders enjoyed having the responsibility of in-kind support to contribute to the investments. Community discussions also highlighted ongoing concern among beneficiaries about having similar opportunities in the future, although if funds were to be available, they were confident in being able to institute effectively other priorities in their village-level plans. Targeted discussions with SIC members revealed an ability to see how to build on the experience and an understanding of how they could improve procurement and future funding opportunities. Thus, while it is difficult to pin-point precisely how effective the mechanisms would be in delivering future benefits, there was evidence of a sustained transformation amongst recipient communities that had undertaken the participatory planning and construction process. The feedback was positive on why the investments were needed, how the process was run, the interaction and activity around the construction, and the response to allowing communities to be involved in decision making to improve services. Thus, the achievement of this objective is rated Substantial. 14 Sub-Objective 2: Increased capacity of agricultural institutions to provide demand-driven agricultural services at the local level. 4.14 The second and third objectives of RDP are not typical of classic CDD projects. In the case of Solomon Islands, RDP was seen more broadly as a vehicle to enhance service delivery in areas outside the capital, Honiara. As a country still emerging from an extended period of unrest, RDP could in many ways be seen as two projects in one. This was acknowledged in discussions with Bank staff, the government and with other donors. The relatively disconnected nature of the second and third objectives must be seen in light of RDP as a delivery vehicle for improving outcomes in the rural areas through a range of approaches. While the intention to include this element was sound, the way in which the parts functioned made integration difficult. 4.15 The Ministry of Agriculture and Livestock had been underfunded and dysfunctional for some time prior to the institution of RDP. The intention of the second objective was to provide an injection of funds to MAL, ensure extension services were able to support rural areas and to broaden the range of products and commercial opportunities for the population in rural areas. The approach was to develop a series of demand driven extension activities and work with MAL on formulating a sustainable model for how these could be delivered in the future. A separate team linked to the PCU was housed in MAL and worked closely with Ministry staff in formulating an operational model. 4.16 The activities covered under objective two had two elements: an increase in the capacity of MAL staff and extension officers to deliver support to rural communities; and the provision of support services and extension programs to farmers in targeted regions. The first of these activities focused on providing various support options to MAL and its affiliated staff and extension officers. The breadth of activities and training for this component was staggering – it included everything from English language programs, to participatory methods, to computer training and various things in between. Both the PCU coordinator of components two and three and senior officers in MAL mentioned that this support had emerged following an institutional needs assessment of MAL. However, during IEG’s discussions with the team, it was unclear how strategic this review was. The gaps identified were not linked to a designated strategy, and there was very limited information on who received the training, why they received the training and how it was being applied to the long term benefit of MAL. Records were kept on the participants, the number and type of training conducted, but no tracking data was available on who received the support and how they responded once back in the workplace. Thus, lots of resources were expended on various training activities without adequately tracking the impact on the Ministry or on their capacity to improve services in the future. 4.17 Staff interviewed in MAL expressed enthusiasm for the renewed interest and efforts in supporting extension activities and reactivating work that had been unfunded. In addition, MAL officers explained that some of the courses they did were useful in providing technical information that had since been shared with farmers, particularly with regard to oil palm farming and animal husbandry. But when asked how the support was being tracked and prioritized, the officers could not give a definitive answer. They put in requests for training that related to their work, some of which occurred and some did not. Thus, while there seems 15 to have been benefit accruing from the capacity support activities, it remains difficult to assess the degree of impact and the scope of influence that the funding has had on future MAL extension work. 4.18 In summary, the project also fell short in the specified indicators for this sub- objective: “percentage of villages with satisfactory access to infrastructure and/or services” and, “percentage of households who have changed agricultural practices, including varieties, as a result of agricultural advice.” A target of 35 percent was set in relation to the latter. When using project data including households which both received advice and those that did not, some 11.2 percent of households had changed agricultural practices, including varieties. This was a significant shortfall on the objective. However, as pointed out in the ICR, the result looks better if we only include households that received advice, with the outcome being 48 percent. While useful, the differential highlights the limitation of not adequately scoping where and what are the most useful extension activities and how communities will access markets and commercialize their outputs. There was considerable activity and training under this objective, but the strategic oversight was missing and this limited the impact of the work. The ICR (p.31) indicates that 49 percent of villages in participating provinces were provided with agricultural services, just below the target of 50 percent, but again this is largely focused on the provision of outputs rather than understanding the impact of the intervention. Given these shortcomings, the lack of substantial evidence on the transformation, and the ongoing concerns regarding the sustainability of the gains, the achievement of the second objective is rated as Modest. Sub-Objective 3: Support for rural business development 4.19 The third objective of RDP sought to expand the scope and reach of the CDD model to the provision of financial support to businesses operating in the rural sector. Again the tracking and data on progress for this objective was extremely poor. This was partly due to the arrangements that were made with the banks providing the credit for this activity and partly due to difficulties in getting businesses to provide data on their returns. This made the assessment of the objective difficult, particularly with regard to the sustainability and transformative effect of RDP activity. 4.20 The premise for this objective was limited access to credit for viable rural businesses. The rural sector in Solomon Islands was characterized by many smaller scale trading operators and logistics companies responding to surplus production of agricultural commodities. There was some low-tech processing, but the emphasis was on the buying and first stage processing of cash crops across the archipelago. Demand for the Supplementary Equity Fund was intended to encourage these businesses to expand. In particular, the support sought to enable rural industries to meet the requirements of the banking sector for loan eligibility with support grants and equity financing. 4.21 SEF support was eventually provided to 58 businesses (in excess of the target) with 65 loans worth SI$31 million (US$3.98 million) of which RDP provided SBD7.7 million (US$1.00 million) in supplemental equity. The results of the SEF were impressive in terms of core project indicators. According to the independent evaluation report commissioned by IFAD in preparation for RDP II and included in the ICR, the SEF led to 350-400 new jobs 16 being created worth approximately SI$4.5 million annually. In addition, the report identified flow-on effects from increased economic activity, particularly in areas where copra and cocoa cash crops were popular as these were identified as viable scale up ventures. The regional application of SEF was also noteworthy, with 36 of the total of 58 businesses based outside Honiara and Guadalcanal and 11 of those in Honiara involved in agricultural products. Thus, while the evidence base for economic impact was limited due to poor information from recipients of the credit assistance, the results included in the IFAD report indicate that there was economic impact from the additional funds flowing into many of the isolated communities. Based on IEG’s mission observations, this had positive effects on rural communities assuming the money was being spent locally. There was only one loan default and of the few people we were able to speak with regarding the SEF component, all argued that SEF had been an important stimulus for the commercial bank’s to loosen their restrictions and their perceptions with regard to lending to small agricultural businesses. 4.22 Notwithstanding the achievements, the evaluation identified shortcomings with regards to the SEF objective. First, evidence was weak around the notion that flow-on effects from these businesses was likely to have direct effects on the community component, and in particular on improving the supply of goods and services in these communities is based on weak evidence. Increased economic activity is likely to have been stimulated by the credit injection given the very limited cash economy active in outer-lying areas, but the ICR’s claims that businesses would donate funds to local youth groups and women’s organizations is unlikely. Second, the highly varied results from the indicator on business turnover is unhelpful both with regards to the -78% to +115% figures which reveal little about what worked and what didn’t, but also due to the limited detail that comes with tracking turnover rather than net income. In discussions with SEF stakeholders, the evaluation team could not get an accurate overview of why some businesses were very successful, while others struggled to get traction. The only issue consistently referred to amongst stakeholders was the influence of commodity price variations. But these were of minor importance over the duration of SEF support. Thus, the nuances of equity support remain elusive due to information and data constraints that could well have been better addressed in design and implementation of the M&E system. 4.23 The World Bank acknowledged that mistakes were made with regards to the data on the SEF. Recipients were not prepared to share financial details with the project team and the banks provided only general detail on repayment rates rather than on business specifics which the project should have understood at the outset. Of note, only one of the 65 loans defaulted. However, without a more granular overview of where the money went and how it impacted the relevant businesses there is concern that appropriate strategic oversight was missed as we cannot be sure what sectors performed well, what constraints there were to business prosperity, nor an awareness of whether additional technical support may have helped some of these groups. The credit was certainly needed, preparation was well-handled and the businesses responded well to the injection of funds, but the project team must in future develop a more effective protocol to track, record and explain the way the SEF was used if it is to positively inform further growth in RDP II. Given the considerable impact of the equity model on the relevant communities and notwithstanding the shortcomings on tracking and illustrating the transformative effect of the SEF, the achievement of this objective is rated as Substantial. 17 5. Efficiency 5.1 Given the considerable variation in the project’s scope and scale of the project as was outlined in the previous section on efficacy, this review covers a range of efficiency issues relating to RDP, both from a macro perspective looking across the full range of components, and from the CDD perspective both of which deal primarily with component one. 5.2 First, no overall economic rate of return was calculated. The ICR argued that this was due to the lack of appropriate ex-ante calculations and information on infrastructure costing, agricultural yields, and the early nature of the business investments. It goes on to point out that infrastructure comparisons in Solomon Islands are difficult and that geographical variation makes uniform cost comparisons unhelpful. While it is true that ex-ante calculations were not prepared and infrastructure comparisons do not make sense due to the dispersed nature of populations and the size and scale of the various islands, there were ways and means that a cost-benefit analysis could have been undertaken. Considerable work was done in Indonesia as part of the PNPM/KDP CDD program to establish useful protocols for economic analysis in CDD projects. This material was readily available in the design stages of RDP and it is disappointing it was not used in preparing the project for calculating the ERR, nor for an ex-poste estimate of the ERR. 5.3 Second, the efficiency calculations for components two and three are problematic. Crude average cost per beneficiary calculations were made for the extension services provided through MAL, but these have considerable methodological limitations, particularly given the clustering of workshops (where different training was provided on the same visit and to the same beneficiaries) and the multiple activities many farmers were being trained for. Thus, it is highly likely that considerable double counting occurred in the assessment of the training benefits and that attendees were receiving support for skills that may or not be required or applied in their work (see below in 5.7). The estimates for returns on component three are equally suspect with the assumptions made on returns based on the leverage factor and the administrative cost component of the project. Finally, there is no attempt to calculate the likely returns on productive assets funded by the project. The original objective sought to improve the livelihoods of the rural population in Solomon Islands. Sampling and conducting a cost benefit analysis on a range of the infrastructure investments would have been a useful way to better understand both the economic returns for different assets and as an important baseline for facilitators working with communities in RDP II. The team did institute an EIRR calculation in preparation for RDP II, but unfortunately this does not provide adequate cost- benefit calculations with regards to what infrastructures work well in different settings. The composition of project sites in Solomon Islands is so different that any economic analysis needs to treat infrastructure differently based on the location and production expectations. 5.4 Whilst acknowledging the limitations of these efforts, RDP did formulate an efficiency measure from data collected on a bundle of actual investments and surrogate measures, in addition to limited ex-post analysis of RDP I data carried out during the design stage of RDP II. While this limits the robustness of the information available to measure the project’s efficiency, the ICR’s efforts to triangulate data to provide some insight with regards 18 to the cost effectiveness of RDP is helpful in getting a limited take on the project’s efficiency. 5.5 The ICR project team used a World Bank background paper on rural investment in Solomon Islands3 to formulate a set of comparators for RDP. This report drew on data from some 1449 sub-projects undertaken between 2008 and 2012 financed by the Provincial Capacity Development Fund and the Rural Advancement Micro-projects Program. The report found that while there was considerable variation in infrastructure costs from one region to another, it could provide a useful benchmark for RDP subprojects. The analysis found that construction costs were ‘broadly consistent with international practice for low cost permanent structures’. But the report does not adequately explore the considerable variance in the cost comparisons and while there is a reference to the geographical/location variation, this does not adequately explain the huge cost differences between some of the water projects. Having said that, the average costing for the various water projects indicates that RDP had considerably lower average costs (when taking geographical isolation into account) than the comparator subprojects for other programs. More detail on this variation would have been helpful in identifying the factors driving up costs. There was no cost breakdown for RDP projects in the report and this could not be provided by the PCU during IEG’s country visit. 5.6 The economic internal rate of return (EIRR) for selected subprojects carried out as part of the preparatory work on RDP II range between 12 percent for health projects to 39 percent for solar power projects. The assumptions for these calculations are rudimentary at best given the table provided in the ICR Annex 3, but the analysis did provide some useful comparators that were not previously available. What IEG found in discussions with the project team was that the enormous variation in type, location and utility of the infrastructure investments made uniform comparisons difficult. In addition, the sampling approach needed more careful consideration for the location of investments to be useful for efficiency analysis in RDP II. 5.7 The component two efficiency calculation was based on a crude calculation of dividing the number of beneficiaries by the total cost of the component to get an average cost per beneficiary of USD$76. This calculation is made using the figures included in Annex 2 of the ICR where the 1084 extension and training activities are matched to the number of farmers in the various provinces (36954). While these figures look impressive, the IEG evaluation team is concerned that there is extensive double counting in the figures with regards to farmers involved in multiple training exercises and thus the ICR has inflated beneficiary numbers. For example, in Temotu (one of the smallest provinces in Solomon Islands) the listed beneficiaries receiving extension and training support totaled 12,500 farmers throughout the project period. Given the estimated total population for Temotu is approximately 22,000 it is highly unlikely these figures are accurate. The number of training sessions provided in Temotu is also extremely high (up to 78 sessions in 2010), which suggests there is considerable bundling of training and attendance for the various sessions. This does not detract from possible benefits, but it does make the beneficiary numbers somewhat misleading. When the evaluation team discussed this with the PCU, they could not 3 World Bank, Towards Better Rural Investment in the Solomon Islands 19 account for the exact numbers of clustered training sessions, nor could they account for farmers who may or may not have been attending multiple sessions. This is disappointing as it makes the tracking of impact difficult with regards to the component two activities and the claimed unit cost for the sessions is also variable based on how and where sessions were combined and who attended. In addition, the spread of activities across the various provinces is also devoid of strategy with the largest number of programs being delivered to one of the smallest provinces with extremely limited commercial opportunities. 5.8 With regards to the SEF, the only attempt at measuring efficiency was to highlight the leveraging that resulted from the supplementary equity. This came in at around 75%, given the $0.985 spent on the component and the total equity pool of $4.3 million. In addition, the ICR points to the low administrative costs of this component (pg56 of ICR) given the work done by the finance institutions. But actual costs for this were not disaggregated and were included in the overall project costs for all components making it difficult to value the estimate of 0.9% for administrative costs alone. Thus, the evidence base for measuring the SEF efficiency is weak even with follow-up conversations with PCU staff responsible for the monitoring and delivery of the activity. In discussions with both the head of the PCU and the task team leader, they acknowledged that inadequate tracking of the SEF component considerably limited the insights that could have been drawn from this aspect of RDP, suggesting that increased effort will need to be addressed towards similar aspects of RDP II. 5.9 It should also be mentioned that RDP had three operational components for a total of USD30 million and project management which provided “simply a managerial function” – had a cost of $7.5 million. For every dollar spent in an operational component 25 cents was spent on project management. By traditional Bank project standards this is unacceptably high, but the circumstances in Solomon Islands are such that considerable additional expenses are required. The factors informing this additional expense include the following: limited capacity in both the government and the private sector; the difficulty of attracting appropriately qualified staff to an isolated region; the extraordinary logistical constraints necessitating transport costs for staff to get to the isolated project regions; and finally, the inflated overheads of running an office in Honiara. Each of these issues were explored in discussions with the PCU, but they do highlight the need to be attuned to the significant additional costs of operating Bank projects in fragile environments. 5.10 Thus, while the project has enabled a wide range of infrastructure to be built and has appropriately sought to undertake these works at least cost, there were moderate shortcomings both in the project’s approach and in the lack of detail with which efficiency has been measured. Therefore, the rating for efficiency is Modest. 6. Ratings Outcome 6.1 The Rural Development Program was an important project for Solomon Islands. The approach was well thought through, built on previous experience of both the World Bank and other donors working in the country and it filled an immediate need to respond to marginalized poorer populations in rural areas. RDP was consistent and complementary to 20 both the World Bank country strategy for Solomon Islands and to the government’s rural development priorities. Therefore, notwithstanding the overly ambitious early commitment that was made, RDP’s project objective remains Highly relevant. While this review found RDP to be the right approach given current development circumstances in Solomon Islands, there were shortcomings in the design logic and the structure of RDP was conceived and put together. Thus, the relevance of design for RDP is rated as Modest. 6.2 Breaking down the achievements (efficacy) of the sub-objectives revealed a project focused on the delivery of services and infrastructure through a CDD approach, a capacity and extension training function to turnaround MAL and a supplementary equity support program to stimulate small businesses. The CDD and equity objectives (one and three) were both Substantially met, but there were moderate shortcomings with both the MAL capacity building approach and the delivery of extension services across the islands. Thus, the efficacy of objective two is rated as Modest. Given the limited information available, the relatively slow roll-out of infrastructure and services, the remaining cohort of unfinished projects and the relatively modest gains that can be attributed to various aspects of RDP, efficiency is rated Modest. The combination of the ratings for relevance, efficacy and efficiency results in an overall outcome rating for RDP of Moderately Satisfactory. Risk to Development Outcome 6.3 Solomon Islands continues to be an environment limited by low capacity and burdened by geographical constraints. In the medium term the country is likely to remain on the World Bank’s Fragile and Conflict Situations list. There are a range of factors driving this categorization, some of which relate to political instability and the potential for further conflict, but these factors have dissipated somewhat over the last five years. The binding constraint now relates to governance and the slow progress on institutional reform and improvement. Thus, the political dimension regarding the risk to RDPs development outcome has become more predictable and less likely to threaten the progress that has already been made. Similarly, while politics is still volatile, the turnover in leadership has diminished and the fiscal environment has stabilized making RDP’s future more secure. 6.4 The successor project to RDP is now underway. This second phase was always envisioned given the complexity inherent in setting up CDD systems and the need to ensure roll-out across all the islands. The new project has drawn on many of the lessons from RDP I and has sought to respond to the importance of productive investments for long term rural benefit. RDP II will continue to provide facilitation to the investment process, and will develop systems for operation and maintenance of commitments already made. In addition, the second phase will focus on the development of private sector partnerships designed to enhance the supply chain aspect of rural markets. There will be less focused support to MAL and less direct funding of extension activities. But the emphasis on the development of markets is intended to encourage private companies to fill this void. 6.5 As the ICR explains, the main concern with regards to the risks of RDP has been the limited integration of approaches into national and provincial plans and priorities. Of particular concern to this review was the many recipients that saw no potential for future funding or support. In our focus group discussions, respondents weren’t confident that either 21 the existing commitments or future investments would receive sufficient financial support. The distrust in government and the lack of essential services made many communities believe that RDP funding would only provide a ‘one-off’ investment. They also explained that village level committees put together for the decision making process would not survive without a tangible set of tasks to undertake. 6.6 Thus, the RDP model is threatened both by a skeptical community still questioning of a government commitment to community development and by money being diverted to ‘constituency funds’ (CDFs) and ‘provincial funds’ (PCDFs) that do not have the same rules and structures as those flowing through RDP. While RDP II does have government support, the integration of various funding streams and services is necessary if the community is to believe there is a serious commitment to appropriate funding and support in the longer term for essential services that follow a community model. For the CDD model to survive and thrive in Solomon Islands, it will require much greater support from the government to ensure it its provincial financing is transparent and that essential services are committed based upon community discussions and identified needs. 6.7 With regards to the sustainability of the investments, there is some concern that a number of the projects have no means of future financial support. This was particularly true of the village halls and communal facilities. Those linked directly to government ministries had been given assurances with regards to maintenance (including schools and health clinics), but without on-going provincial funds many of the non-productive assets will degenerate over time. So while the quality of investment works was good, the review found that there would be significant variability in the available resources for maintenance. 6.8 Finally, there is a risk that the marginal gains with regards to MAL are not adequately supported by future financial allocations to ensure extension activities can continue and be incorporated into recurrent budgets. RDP had re-energized much of the extension program of MAL, particularly after many years of neglect, but for the training and support to lead to productive outcomes, further logistical support and funding will be required. Despite the range of risks identified, there is evident momentum in RDP II and the review team is confident that the lessons of RDP I have been incorporated in to the new design. The consistency of the PCU should enable the program to gradually build an institutional home and the appetite for rural programming does mean there is a responsive audience and a solid model for future funding to the regions. Thus, the risk to development outcome is rated Moderate. Bank Performance 6.9 The World Bank took a pro-active and conservative approach to design and preparations for RDP. The project built on work that was part of Solomon Islands Smallholder Agriculture study funded by AusAID in 2006 and on the lessons coming out of the World Bank’s earlier project, the Community Development Scheme which also funded small-scale infrastructure. The intention of the project was to consolidate the various investments made by donors in the rural space and deliver support and services through a cohesive CDD model. The World Bank incorporated community procurement arrangements and financial management into the design to ensure sustainable elements were built into 22 longer term planning. Likewise, the protocols for community facilitation and support to community workers was carefully considered in addition to the importance of a PCU given the limited capacity in the recipient ministry. These factors ensured that while slow in building momentum, the project had the necessary oversight and guidance, particularly with regards to technical support on setting up the facilitation for the participatory CDD model. The World Bank also sensibly chose to recruit an internationally recruited staff member to be based in Solomon Islands to manage the project and ensure there was responsive support and management to guide and mentor the PCU in the rollout of the various components. 6.10 RDP was also designed to accommodate and build relations with other donors supporting rural development. Relationships with AusAID and IFAD were well established and exploited both funding and complimentary expertise. In the IEG team’s discussion with Department of Foreign Affairs (previously AusAID), participants strongly agreed that the project was well designed for the existing conditions within the sector and responded to a defined strategic need. Use of Country Systems 6.11 Similar to other World Bank supported projects in Solomon Islands, the use of country systems with the initial project design proved to be overly optimistic. In the face of weak and insufficient capacity of country systems, the project used its own financial management, procurement and monitoring systems. The need for this should have been foreseen given previous project experience, and would have justified the institutional costs of this approach. Significant effort was expended under component two to rebuild service provision in MAL, which was essentially dysfunctional following the civil conflict of 1998- 2003. This arrangement showed greater foresight than the engagement with the main partner, MDPAC, where there were serious shortcomings with regards to internal capacity. The support to MAL was a useful design feature in a two phase project expected to continue for over ten years and could have been effectively replicated in some way with building the capacity of MDPAC to deliver on projects in the future. Overestimation of Private Sector Presence and Capacity 6.12 As concerns component two, a flaw in the initial project design was a failure to sufficiently assess the capacity of and the necessity of utilizing secondary project beneficiaries, namely NGOs, churches, rural training centers and the private sector to assist with rural service delivery. These groups play a central role in much of the community decision making and provide significant service delivery capacity through the various activities they undertake. Thus while some effort was made to incorporate them into activities designed to improve agricultural services these were largely unsuccessful. The ability to encourage an interest amongst mainly Honiara-based NGOs in the project proved difficult and the project design possibly misunderstood the potential role of churches and rural training centers in Solomon Islands. A very narrow private sector (which engages selectively across the country and in select agricultural products) similarly hampered the planned approach. More detailed project-related empirical work during the design phase would have been beneficial in this regard. This failure placed an additional burden on MAL and its agricultural extension officers who had been inactive for considerable periods of time 23 and did not have the capacity to engage, liaise and work with the external groups capable of supporting the extension component. 6.13 Thus, while carefully thought through and responsive to the prevailing conditions, there were minor shortcomings. First, the project was designed in tandem with MDPAC which in hindsight was ill equipped to take on the lead role in coordinating a very complex CDD project even with support from the World Bank and the PCU. The Ministry was certainly the obvious partner given its lead role in developing the agriculture and rural development strategy, but the capacity limitations should have been better incorporated into the project design. Second, the M&E arrangements and results framework were far too ambitious for the project. There was little detail around how the ambitious base line work would be carried out and the overly ambitious objective was a reflection of this lack of realism regarding circumstances in Solomon Islands. Not only were there too many indicators to adequately track without necessary support structures in place, there were also indicators that did not align well to the activities being undertaken. Finally, the design did not adequately articulate a longer term vision for how the CDD model would mature over time and incorporate government systems into its operations in the medium to long term. This led IEG to rate quality at entry as Moderately Satisfactory. 6.14 With regards to the supervision during implementation, the World Bank team was responsive both to the issues requiring attention and to regular iterative innovation. In three of the focus group discussions, respondents explained how the approach evolved over time and how grievances or problems were dealt with by the community workers. In one case, a committee member explained that after concerns regarding cost over-runs, the PCU manager visited the project site and sat down with the community to explain the procurement approach and to explore possible solutions. The implementation status reports and aide memoires provide detailed insight on progress and the midterm review was a comprehensive a reflective take on how the project was progressing. Most importantly, the partners also acknowledged the detail with which the Bank engaged in issues and explored various options when problems emerged. IFAD noted the ‘highly effective’ approach taken to supervision and the various people IEG spoke with at the Australian Department of Foreign Affairs and Trade were pleased that RDP ensured an on-going dialogue with the Bank regarding rural development issues. The project restructuring and additional financing was sensibly handled given the initial progress made and in light of the ongoing government support for an additional phase. 6.15 However, there were shortcomings. The M&E design, system and utility was poor and the Bank should have done a better job at rectifying this issue, particularly given the intention to have subsequent CDD projects and to develop appropriate models for service delivery. Even pragmatic oversight of the participation process was poor in documenting the decision making activities and what communities were identifying as their priorities. The inability to institute a workable and effective M&E system put the project back and this should have been rectified earlier. Thus, the quality of supervision rating is Moderately Satisfactory, and the Bank’s overall performance is rated as Moderately Satisfactory. 24 Borrower Performance 6.16 As outlined earlier, the relationship between the World Bank and Solomon Islands government was effective both in terms of planning for RDP and having an ongoing dialogue, particularly as the team prepared for RDP II. However, there were significant shortcomings. First, the provision of funds for the project was delayed and this led to problems with completion of various sub-projects. Second, while the Ministry of Finance (MoFT) was supportive they were not proactive with regards to preparation and implementation. Third, the Auditor General’s office was late in the provision of audits and while they were completed this was only after instituting an arrangement for independent auditors to submit audits after required deadlines. Finally, there was no real effort to have government representatives participate in supervision missions and thus RDP remained largely disconnected from the government, even while increasing its’ visibility in the provinces. Most importantly, the shortcomings make it difficult to see what progress, if any, the project can make in complementing future government allocations to the provinces and how the distribution channels established through RDP can better serve the efficiency of funding to rural communities. Given these considerable shortcomings, even in an environment of limited capacity, government performance is rated Moderately Unsatisfactory. 6.17 The Project Coordination Unit (at the national level) and Project Support Units (at the provincial level) in the main operated independently of government. Planning was similarly unaligned (across all rural development initiatives in Solomon Islands, not solely RDP). As detailed in a 2013 Bank produced sector report relating to rural development investment in Solomon Islands, “the [RDP] planning process sits outside provincial or sector plans. The development priorities identified by communities are not formally fed into the development of provincial or national plans, but rather used only to identify projects for RDP funding.” In addition, recurrent government financing for the project was not established. 6.18 Little evidence of project ownership exists at either the national or sub-national levels of government, particularly for component one. Based on the evaluation team’s discussions with beneficiaries, a little more success in this regard has been achieved at the community (or village) level where the sub-implementation committees (SICs) enthusiastically embraced the approach to prioritization of investments and the planning associated with raising funds and preparing for projects. While the project overestimated capacity, the extent of community consultation around project implementation was significant. However, there was no evidence of improved accountability amongst locally elected ward councilors as was hoped, and their involvement in the project was minimal. There is also no evidence that churches and existing “tribal councils” were used for RDP activities, despite separate project structures at the community level (SICs) being established. The SICs almost universally existed for the duration of the sub-project period and disbanded thereafter. One of the perceived weaknesses with the SIC structure experienced in some communities visited during the IEG mission was its failure to adequately incorporate, extant indigenous governance mechanisms. 6.19 MDPAC was the key government partner. While their limited capacity made it difficult to provide strategic oversight, they were responsive to the direction of the World Bank and supported a high performing PCU. The very large increase in the management 25 costs do raise questions around the efficiency of the PCU, but these can be explained by the considerable additional financing and expansion in scope of the project. The borrower, bilateral partner and the IDA envelope were all increased substantially to increase the reach and scale of infrastructure investment and this resulted in substantial project management cost increases. In the review team’s discussions with the PCU, there was an excellent grasp of the issues, a clear set of project insights, an awareness of areas for improvement and the ability to see the longer term objective of RDP. Finances were generally well managed and the PCU understood where there were potential flaws in the RDP model. There was certainly an unrealistic optimism amongst PCU staff with regards to provincial capacity and the M&E arrangements, but they showed and expressed an ability to respond to the contracting and procurement demands and were given the necessary latitude from MDPAC to adjust as required. The PCU team acknowledged that it was difficult to know where best RDP should be housed, but there were benefits with MDPAC oversight as they understood donor requirements and had better checks and balances than MAL, the other potential partner. They also had more interest in the project than the larger MoFT who had the broader oversight. Thus, the rating for implementing agency performance is moderately satisfactory, and the rating for overall borrower performance is moderately satisfactory given the outcome rating of moderately satisfactory which necessitates rounding up according to the IEG and Operations Policy and Country Services harmonized review guidelines. Monitoring and Evaluation 6.20 There were a number of shortcomings in the design of the M&E system for the project. First, there was little clarity around the nature, scope and realism of the planned household surveys that were to inform the baseline data. In addition, the design details and procurement arrangements with regards to the Management Information System were vague preventing adequate scrutiny of the approach during the design review process. Second, there was limited detail on the timing, costing and approach with regards to the village needs assessment. The intended commitments with regards to M&E were laudable, but the reality of limited capacity, poor technology, logistical issues and under investment in the approach meant these commitments were unlikely to be met. Previous experience internationally in countries such as Indonesia, Tanzania, Morocco and Timor Leste had shown how difficult it is to build pragmatic M&E systems for CDD programs. These lessons should have been more carefully applied to the M&E design approach in RDP to avoid the unreasonable expectations that were presented in the PAD. 6.21 The management and utility of data and information was disappointing. An enormously complex and detailed decision making approach was undertaken with project villages. This involved a series of participatory workshops where important information on investment priorities, village needs, the recording of existing services and an outline of what villages wanted from the project were captured. This material was manually recorded and used in the formulating of the infrastructure project scoping and costing. However, this manually captured data and information has not been processed, analyzed, nor built into a functioning MIS of any kind, thus missing out on crucial data that could provide information at the village level. 26 6.22 While attempts were made to track program performance, including third party procurement of an automated system, the hiring of a high level M&E expert and a default to collecting information manually on field visits, there were considerable shortcomings with building a pragmatic and simple system that could provide for some of the basic needs. Following lengthy discussions with the new RDP II M&E officer, the review team learned that much of the confusion and problems associated with M&E was driven by domestic capacity constraints in bringing the right team on board and the lack of direction from the PCU and the World Bank in formulating an approach built on information utility and the capabilities of the community officers. IEG’s evaluation team remains concerned that the team is still being far too ambitious with the M&E approach in RDP II rather than thinking about what is sensible and realistic to track in the medium to longer term. 6.23 The PCU was disciplined in formulating a mechanism to capture and record outputs including infrastructure progress and costing, training and extension activities and then apply these to provincial reporting. In addition, a set of baseline data was eventually put together and an ‘impact’ (not an RCT) evaluation was procured from an outside third party which collected a set of useful lessons. However, there were considerable shortcomings with regards to tracking the variable costing of similar infrastructure investments across regions, the appropriate tracking of gender and other beneficiary information, the recording of decision making protocols, the measurement and development of facilitation approaches, and in formulating any cost-benefit work on productive benefits of different investments. This is all extremely basic information that has been collected pragmatically and sensibly in similar programs in the Bank’s CDD portfolio, but remains absent from RDP’s records. 6.24 From a purely operational perspective, RDPs inability to put together a viable baseline, track this effectively through the life of the project and draw on the findings to inform managerial responses meant that many of the tacit insights RDP staff and beneficiaries had developed could not be captured and utilized to improve the structure and systems for RDP II. Given these considerable shortcomings, the rating for M&E is Modest. 7. Lessons 7.1 CDD projects in fragile and conflict states face difficulties when they try to be ‘all things to all people’. RDP was a particularly complex project given the split between the infrastructure and service delivery support, the capacity enhancement of the Ministry of Agriculture and Livestock and the supplementary equity facility. The acknowledged temptation with RDP was to design and institute a project that sought to ‘fix everything’ in the rural space. This limited the impact as the implementation team struggled to keep the various aspects on track while identifying the necessary points of integration to ensure maximum benefit to those living in rural areas. 7.2 CDD projects in island states are more effective when they recognize the additional support required and costs involved in working in a limited capacity environment. The experience of RDP showed that small island nations need to be treated differently. The RDP required a project design that recognized the complexity of not only a post conflict, low income, low capacity country, but more importantly, a country composed of isolated islands with high costs and complex logistics. Lessons from traditional CDDs, 27 even those of archipelagic countries such as Indonesia are difficult to transfer to a context such as Solomon Islands. The World Bank has few TTLs that have worked in an island context and they need to be thoroughly aware of the constraints imposed on them when working in poorly serviced islands with extremely limited development opportunities. 7.3 Appropriate project design is challenging in diverse and difficult geographic locations and is more effective when there is sufficient recognition of the different needs of commercial hubs versus the needs of isolated rural areas. RDP was delivered in hugely varied environments, some with good access to markets and others that were isolated with few options for commercial enterprise. In addition, there were semi-urban areas made up of heterogeneous, itinerant communities where the participatory planning approach could not gain traction. Having a uniform CDD model is necessary to ensure consistency and fairness in how projects are delivered, but a more bespoke approach to how the infrastructure and service opportunities are applied will enhance the effectiveness of the CDD model in geographically diverse locations. In addition, the slow maturation of RDP highlights the importance of having phased project based engagements in fragile states given the weak capacity and considerable time required to set up the necessary systems and support structures. It would appear clear that without a follow up project to RDP, the risk to Development Outcome would have been high. 7.4 Capacity building and institutional reform in weak ministerial environments is more effective when it is appropriately targeted to the change expected within the partner organization. Much of the success of the institutional aspect of RDP can be linked to the acknowledgement of the limitations of the Ministry of Agriculture and Livestock (MAL) and the strategic decision not to use this ministry as the implementing partner. RDP’s sole support to the ministry was through extension support and technical assistance which ensured better outcomes for farmers. This left oversight for the project to the Program Coordination Unit and Ministry of Development Planning and Aid Coordination, who were better able to cope with the operational realities of a complex CDD project. 7.5 Rural enterprise development is difficult and is more effective when it uses a combination of approaches. RDP spent considerable time and effort on improving extension activities, particularly on products that were cash convertible for rural producers. However sufficient effort was not placed on understanding the nature of rural development in determining the approaches. A better understanding of the nature of commercial opportunity and the potential growth opportunities for certain crops will be an essential corollary to enterprise development in RDP II. 7.6 Women tend to be included overall in CDD projects, but they are more successful when sufficient emphasis is placed on women gaining access to the same level of productive activities as men. Information is lacking on whether female rural enterprise owners were identified, supported or targeted in RDP activities as they should have been given the commitment to advancing the interests and livelihoods of women. Therefore, while RDP projects were saving women time and they were involved in various trainings and activities, the question remains as to whether the project was contributing to transforming their roles in a sustainable and substantial manner. CDD projects have a long track record of making valuable contributions to the lives of women and this was certainly intended for 28 RDP. However, because of limited disaggregated information on the role of women both in terms of traditional and non-traditional roles, it is difficult to get a sense of how women have fared in Solomon Islands because of RDP. Therefore, greater emphasis on how women can and do contribute to productive activities is necessary to ensure that projects reflect the diversity of its participants. 7.7 The monitoring and evaluation of phased programs is more effective when it reflects, and is responsive to the capacity of the implementing agency. RDP spent considerable time, budget and energy in formulating a management information system that was not suitable for what they needed to track, nor helpful in supporting the participatory planning work that was essential to the infrastructure component. RDP needed M&E that was pragmatic in scope, scale and usability, but ended up with an overly technical platform that did not help in the tracking of essential information. As a result, much of the important work done in the communities was either lost or poorly archived, preventing the second phase from building on the lessons of various metrics recorded in the first phase. 29 References Barron, Patrick, Rachael Diprose, and Michael Woolcock. 2011. Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia. New Haven, CT: Yale University Press. http://www.jstor.org/stable/j.ctt5vkw76. Ministry of Agriculture and Livestock. 2009. Solomon Islands National Agriculture and Livestock Sector Policy 2009-2014. Honiara, Solomon Islands: Ministry of Agriculture and Livestock. http://pafpnet.spc.int/pafpnet/attachments/article/solomon- islands/National-Agriculture-and-Livestock-Sector-Policy.pdf Ministry of Development Planning and Aid Coordination. 2011. National Development Strategy 2011 to 2020—Vision: A United and Vibrant Solomon Islands. Honiara, Solomon Islands: Ministry of Development Planning and Aid Coordination. http://www.nationalplanningcycles.org/sites/default/files/planning_cycle_repository/solo mon_islands/cobp-sol-2015-2017-sd.pdf Office of the Prime Minister and Cabinet. 2015. Democratic Coalition for Change Government Policy Statement. Honiara, Solomon Islands: Office of the Prime Minister and Cabinet. http://pafpnet.spc.int/images/articles/policy- bank/solomon/Demo_Coa_Policy_Statement.pdf Olken, Benjamin A., Junko Onishi, and Susan Wong. 2011. Indonesia's PNPM Generasi Program: Final Impact Evaluation Report. Washington, D.C.: World Bank. http://documents.worldbank.org/curated/en/476001468269431928/Indonesias-PNPM- Generasi-Program-final-impact-evaluation-report. World Bank. 2005. Regional Engagement Framework FY2006-2009 for Pacific Islands. Washington, D.C.: World Bank. http://documents.worldbank.org/curated/en/514351468096563700/Regional-engagement- framework-FY2006-2009-for-Pacific-Islands ———. 2007. Solomon Islands—Agriculture and Rural Development Strategy: Building Local Foundations for Rural Development. Washington, D.C.: World Bank. https://openknowledge.worldbank.org/handle/10986/7757 License: CC BY 3.0 Unported. ———. 2013. Evaluation of the Solomon Islands Rural Development Program. Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/926411468294672780/Evaluation-of-the- Solomon-Islands-rural-development-program World Bank Group. 2014. Solomon Islands: Towards Better Investment in Rural Communities. Washington, D.C.: World Bank. https://openknowledge.worldbank.org/handle/10986/21530 License: CC BY 3.0 IGO. 31 Appendix A Appendix A. Basic Data Sheet SB-RURAL DEVELOPMENT PROGRAM (LOAN COFN-C1280, IDA-H3320, IDA-H6150, IDA-H8860, TF-90651, TF-90652, TF-97737) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs USD21.80M USD37.40M 173.0 Loan amount USD 3.20M USD 9.50M 287.5 Cofinancing USD23.10M USD27.9M 120.0 Cancellation Cumulative Estimated and Actual Disbursements 2008 2009 2010 2011 2012 2013 Appraisal estimate (US$M) 0.70 1.18 1.87 2.55 3.05 3.20 * Actual (US$M) 0 820 4.18 8.18 13.98 22.33 Actual as % of appraisal 0 694 223.5 320.7 458.3 697.8 Date of final disbursement: 12/2/2013 *Actual cumulative disbursement is reported in the Portal for each fiscal year for Q1. Project Dates Original Actual Initiating memorandum 12/12/2005 9/7/2005 Negotiations 7/24/2007 7/24/2007 Board approval 12/15/2006 6/11/2007 Signing 6/24/2007 Effectiveness 12/21/2007 Closing date 11/30/2015 Staff Time and Cost Staff Time and Cost (Bank Budget Only) USD Thousands Stage of Project Cycle * (including travel and No. of staff weeks consultant costs) Lending FY07 233.44 Total: 233.44 Supervision/ICR Appendix A 32 Staff Time and Cost (Bank Budget Only) USD Thousands Stage of Project Cycle * (including travel and No. of staff weeks consultant costs) FY08 43.13 FY09 84.08 FY10 49.85 FY11 74.30 FY12 140.18 FY13 96.35 FY14 115.98 FY15 31.40 Total: 635.30 *ICR does not reflect the number of staff weeks. Task Team Members Responsibility/ Specialty Names Title Unit Lending Marianne Senior Agriculture Economist Task Team Leader GFAR Grosclaude Natural Resources Oliver Braedt Management LCC6C Program Leader Specialist William Cuddihy Economist Economist Consultant Consultant Christophe Ribes Local Governance and CDD Specialist Consultant GSPGL Ros Michelle L. Chen Program Assistant GSURR Program Assistant David Michael Sr. Financial Sr. Financial Chandler Management GGODR Management Specialist Specialist Financial Financial Management David Whitehead Management GGODR Specialist Specialist Melinda Good Senior Counsel LEGES Senior Counsel Mazhar Farid Legal Analyst LEGEA Legal Analyst 33 Appendix A Responsibility/ Specialty Names Title Unit Lending Sheila Braka Musiime Counsel LEGES Counsel Crstiano Costae Procurement Silva Nunes GGODR Procurement Specialist Specialist Jean-Claude Sr. Agriculture Sr. Agriculture Economist Economist GENDR Balcet Rural Infrastructure Rural Infrastructure Specialist Julian Abrams Specialist Gitanjali Country Program Country Program Assistant Ponnambalam Assistant EACNF Carmenchu D. Austriaco Finance Analyst WFALN Finance Analyst Lovella Senior Finance Tolentino Assistant WFALN Senior Finance Assistant Morada Supervision/ICR Erik Caldwell Senior Operations GSURR Task Team Leader Johnson Officer Stephen Paul Financial GGODR Financial Management Hartung Management Specialist Specialist Bruce M. Harris Safeguard EASTE Safeguards Specialist Specialist Susan S. Shen Manager LLIOP Safeguards Specialist Operations Ly Thi Dieu Vu Consultant GEN02 Safeguards Specialist Knut Opsal Lead Social GSURR Safeguards Specialist Development Specialist Marjorie Mpundu Senior LEGES Lawyer Counsel Jinan Shi Senior GGODR Senior Procurement Specialist Procurement Specialist Appendix A 34 Responsibility/ Specialty Names Title Unit Lending Gitanjali Country Program EACNF Country Program Assistant Ponnambalam Assistant R. Cynthia Program Assistant GFADR Program Assistant Dharmajaya Janet Funa Team Member EACSB Team Member Ross James Butler ET Consultant GSURR Safeguards Specialist Dan Vadnjal Consultant Agriculture Specialist Allan Oliver Operations GFADR Agriculture Specialist Officer John Victor Bottini CDD Advisor Consultant Annette Leith Operations officer Operations Officer William Cuddihy Consultant Agriculture Economist Ruth Alsop Consultant Sociologist David Leeming Consultant ICT specialist Faustinus Ravindra Consultant GSURR M & E Specialist Corea Daisy Lopez Zita Finance Analyst WFALN Finance Analyst Haiyan Wang Senior Finance WFALN Senior Finance Officer Officer 35 Appendix B. List of Persons Met Name Title Erik Caldwell Johnson Task Team Leader RDP Shadrach Fenega Permanent Secretary, Ministry of Development Planning and Aid Coordination Jimi Saelea Permanent Secretary of the Ministry of Agriculture and Livestock Daley Tozaka-Illala Director Finance and Economics Development Unit, Ministry of Finance and Treasury Silke Speier Department of Foreign Affairs and Trade (Australia) Longden Manedika Chief Executive Officer of SIDT John Westland Support Officer of SIDT Daniel Vadjanal IFAD Consultant Mark Johnson RDP Agricultural Consultant Lottie Vaesikavea Project Manager, Rural Development Program Patricia Sese Ministry for Women’s Affairs Hon Sam Iduri MP Member of parliament Hon Batholomew Parapolo Member of parliament Kosuke Anan World Bank Task Team Leader RDP II Salvador Jiao Rapid Employment Project Manager Jeremiah Manele Leader of the Official Opposition and former Permanent Secretary, Ministry of Development Planning and Aid Coordination Annette Leith Senior Operations Advisor – World Bank Collin Potakana Department of Foreign Affairs and Trade (Australia)