Hanks. Document of The World Bank Report No: ICR0000763 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-71480) ON A LOAN IN THE AMOUNT OF US$ 15 MILLION TO JAMAICA FOR THE NATIONAL COMMUNITY DEVELOPMENT PROJECT November 24, 2008 Finance, Private Sector and Infrastructure, Sector Management Unit Caribbean Country Management Unit Latin America and the Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective December 3, 2008) Currency Unit = Jamaica Dollars (J$) 1.00 = US$ 0.0128 US$ 1.00 = 77.75 FISCAL YEAR January 1 - December 31 ABBREVIATIONS AND ACRONYMS CAS Country Assistance Strategy CBC Community Based Contracting CBO Community Based Organization CDC Community Development Committee CDD Community Driven Development CDP Community Development Plan DFID British Department of International Development ERL Emergency Recovery Loan EU European Union GDP Gross Domestic Product GOJ Government of Jamaica ICR Implementation Completion Report IERR Internal Economic Rate of Return JSIF Jamaica Social Investment Fund KRC Kingston Restoration Company NCDP National Community Development Project NGO Non Governmental Organization NPEP National Poverty Eradication Program OPCPR Procurement Policy and Services Group PIOJ Planning Institute of Jamaica PMC Project Maintenance Committee SDC Sustainable Development Commission TA Technical Assistance Vice President: Pamela Cox Country Director: Yvonne M. Tsikata Sector Manager: Guang Zhe Chen Project Team Leader: Taimur Samad ICR Team Leader: Taimur Samad Jamaica National Community Development Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 4 3. Assessment of Outcomes............................................................................................ 9 4. Assessment of Risk to Development Outcome......................................................... 15 5. Assessment of Bank and Borrower Performance ..................................................... 16 6. Lessons Learned ....................................................................................................... 18 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 19 Annex 1. Project Costs and Financing.......................................................................... 21 Annex 2. Outputs by Component ................................................................................. 23 Annex 3. Economic and Financial Analysis................................................................. 26 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 28 Annex 5. Beneficiary Survey Results........................................................................... 30 Annex 6. Stakeholder Workshop Report and Results................................................... 32 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR..................... 33 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders....................... 43 Annex 9. List of Supporting Documents ...................................................................... 44 MAP A. Basic Information National Community Country: Jamaica Project Name: Development Project Project ID: P076837 L/C/TF Number(s): IBRD-71480 ICR Date: 12/21/2008 ICR Type: Core ICR Lending Instrument: SIL Borrower: JAMAICA Original Total USD 15.0M Disbursed Amount: USD 14.7M Commitment: Environmental Category: B Implementing Agencies: Jamaica Social Investment Fund Cofinanciers and Other External Partners: OPEC Fund for International Development B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 01/24/2002 Effectiveness: 04/08/2003 04/08/2003 Appraisal: 03/25/2002 Restructuring(s): Approval: 05/30/2002 Mid-term Review: 12/05/2005 01/09/2006 Closing: 12/31/2007 06/30/2008 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Moderate Bank Performance: Satisfactory Borrower Performance: Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Satisfactory Quality of Supervision: Satisfactory Implementing Agency/Agencies: Satisfactory Overall Bank Overall Borrower Performance: Satisfactory Performance: Satisfactory C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Performance Indicators (if any) Rating Potential Problem Project No Quality at Entry None i at any time (Yes/No): (QEA): Problem Project at any Quality of Moderately No time (Yes/No): Supervision (QSA): Unsatisfactory DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) General education sector 20 76 General water, sanitation and flood protection sector 20 Health 20 5 Other social services 30 2 Roads and highways 10 17 Theme Code (Primary/Secondary) Access to urban services and housing Primary Primary Participation and civic engagement Primary Primary Rural services and infrastructure Primary Primary Social safety nets Primary Primary E. Bank Staff Positions At ICR At Approval Vice President: Pamela Cox David de Ferranti Country Director: Yvonne M. Tsikata Orsalia Kalantzopoulos Sector Manager: Guang Zhe Chen Danny M. Leipziger Project Team Leader: Taimur Samad Thakoor Persaud ICR Team Leader: Taimur Samad ICR Primary Author: Julia Van Domelen Jessica Wurwarg F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) Project DO are: (a) to help communities in areas affected by economic decline and lack of basic services and (b) to assist the Government of Jamaica to promote greater social and community development, especially among the poor. ii Revised Project Development Objectives (as approved by original approving authority) PDOs not formally revised. (a) PDO Indicator(s) Original Target Formally Actual Value Indicator Baseline Value Values (from Revised Achieved at approval Target Completion or documents) Values Target Years Indicator 1 : Number of incremental temporary jobs created by sub-projects -15,420 person/months -79% of households in NCDP communities with Value road projects cited quantitative or the creation of Qualitative) many temporary jobs as a result of NCDP, 61% for health projects and 39% for education projects Date achieved 06/30/2008 Comments (incl. % No targets set# local employment effects widely perceived achievement) Indicator 2 : Increase in number of subprojects financed by JSIF after being identified in Community Development Plans -173 CDPs Value delivered by SDC quantitative or 0 135 CDPs -15 approved Qualitative) projects had CDPs from SDC. Date achieved 09/11/2002 12/31/2007 06/30/2008 Comments (incl. % 128% for CDPs; no quantitative targets set for # of projects (only 11% of CDPs achievement) had subprojects approved by JSIF) Indicator 3 : Increase in access to and quality of basic services Overall, 62% of NCDP community Value Positive members felt that quantitative or improvements by the community was Qualitative) type of a better place to intervention live, compared with 45% in comparator communities iii Date achieved 12/31/2007 06/30/2008 Comments (incl. % No quantitative targets set # significant positive impact in all main sectors. achievement) Indicator 4 : Improved operation and maintenance of project facilities -Schools-64% noted improvements -Health cntrs-70% noted Value improvements quantitative or -67% of subprjects Qualitative) had good to avg maintenance for infrastructure -71% of households stated infrstr is maintained Date achieved 06/30/2008 Comments Positive operations and maintenance on all indicators versus comparison groups. (incl. % -Community interest in maintenance was higher in NCDP communities (at 86%) achievement) than comparators (at 60%). Increase in the number of communities which develop the capacity to select their Indicator 5 : own priority projects and to manage project contracting and project implementation effectively - 187 communities undertake prioritization processes (all but Value Hurricane Ivan quantitative or 0 30 CBC subprojects) Qualitative) communities -52 communities trained and managing Community-Based Contracting method Date achieved 09/11/2002 12/31/2007 06/30/2008 Comments (incl. % 173% of CBC target met achievement) Evidence of increased complementary investment generated (from residents, Indicator 6 : other public sector entities and the private sector) through JSIF and Social Development Commission (SDC) assistance to community groups. Value -33 new Benevolent quantitative or Societies created Qualitative) -15 out of 52 CBC iv projects successfully received additional complementary funding (29%) from other agencies or locally -JSIF-directly leveraged private sector investments (by Red Stripe and Digicel). Date achieved 06/30/2008 Comments (incl. % achievement) Indicator 7 : Number of subprojects with active community maintenance plans and clear indication of agreed support from line ministries and/or local authorities. - 31 subprojects prepared and implemented maintenance plans Value 100% of - 100% of health, quantitative or traditional and education, road Qualitative) CBC infrastructure subprojects subprojects received no- objection from sector ministry and parish council Date achieved 12/31/2007 06/30/2008 20% of applicable subprojects (all economic and social infrastructure Comments subprojects, less Hurricane portfolio) (incl. % 100% of subprojects have indication of agreed support from ministries/local achievement) authorities Indicator 8 : Level of community satisfaction with subproject and performance of sponsoring entities, JSIF, SDC, and Community Development Committees (CDCs). Over 90% of NCDP Value beneficiaries quantitative or satisfied with the Qualitative) subproject and NCDP performance Date achieved 06/30/2008 Comments (incl. % No targets set. achievement) v (b) Intermediate Outcome Indicator(s) Original Target Formally Actual Value Indicator Baseline Value Values (from Achieved at approval Revised Completion or documents) Target Values Target Years Indicator 1 : Number and type of subprojects Total NCDP: 293 Of which: Social Value Infrastructure: 227 (quantitative Economic or Qualitative) Infrastructure: 29 Social Services: 33 Date achieved 06/30/2008 Comments (incl. % achievement) Indicator 2 : Number and type of subprojects financed by CBC Total 52, of which: 4 roads and drains 8 community spaces 16 schools 5 life coping skill facilities Value 1 sanitation (quantitative 0 30 5 agroprocessing or Qualitative) facilities 2 counseling facilities 4 health centers 2 homes/shelters 4 skills training facilities 1 water supply Date achieved 09/11/2002 12/31/2007 06/30/2008 Comments (incl. % 173% achievement) Indicator 3 : Facility utilization rate over time Value 96% of 297 (quantitative completed or Qualitative) subprojects Date achieved 06/30/2008 Comments vi (incl. % achievement) Indicator 4 : Number of people trained to support community development 640 members of Value project (quantitative management or Qualitative) committees/ Benevolent Societies. Date achieved 06/30/2008 Comments (incl. % achievement) Indicator 5 : Increase in average percent of community contribution in subprojects Overall NCDP: Value 5% 12.3% (quantitative (JSIF 1) 10% Of which: or Qualitative) Traditional: 11.7% CBC: 24.9% Ivan: 0% Date achieved 09/11/2002 12/31/2007 06/30/2008 Comments (incl. % 123% of target for all NCDP portfolio; 150% subtracting out Hurricane Ivan achievement) projects which did not require a counterpart Indicator 6 : Reduced time taken to prepare and implement subprojects 80% of All NCDP subprojects subprojects: reaching targets: 53% of NCDP 525 days subprojects met Value 855 days application to processing targets (quantitative (JSIF I ICR) practical to practical or Qualitative) completion completion within 735 days 525 days application to final 21% reached goal completion of 735 days to final (NCDP OpMan) completion Date achieved 09/11/2002 12/31/2007 06/30/2008 Comments Subprojects may stay in registration for long periods a result of gov't delays. If (incl. % adjustments are made to exclude projects which spent an inordinate time in achievement) registration, then the % of projects completed w/in the specified time target increases to 60% Indicator 7 : Improved quality in the design and preparation of sub-projects -Quality of work Value done by contractor (quantitative ranked as high in or Qualitative) 66% of NCDP communities vs. 46% for comparator vii communities Date achieved 06/30/2008 Comments (incl. % achievement) G. Ratings of Project Performance in ISRs Actual No. Date ISR Archived DO IP Disbursements (USD millions) 1 12/06/2002 Satisfactory Satisfactory 0.00 2 05/06/2003 Satisfactory Satisfactory 0.15 3 11/05/2003 Satisfactory Satisfactory 0.65 4 05/04/2004 Satisfactory Satisfactory 1.20 5 06/24/2004 Satisfactory Satisfactory 1.20 6 11/29/2004 Satisfactory Satisfactory 2.59 7 05/10/2005 Satisfactory Satisfactory 4.22 8 06/09/2005 Satisfactory Satisfactory 4.61 9 10/18/2005 Satisfactory Satisfactory 7.05 10 01/27/2006 Satisfactory Satisfactory 8.29 11 10/10/2006 Satisfactory Satisfactory 11.72 12 05/03/2007 Satisfactory Satisfactory 13.00 13 12/13/2007 Satisfactory Satisfactory 14.42 14 06/20/2008 Satisfactory Satisfactory 14.57 H. Restructuring (if any) Not Applicable viii I. Disbursement Profile ix 1. Project Context, Development Objectives and Design At the time of project preparation, Jamaica had suffered several years of low or negative growth following external shocks, including the post-September 11 decline in tourism earnings and low demand for Jamaica's exports. Internally, devastating floods, high unemployment, limited access to basic services, as well as crime and violence continued to hamper the ability of poor communities to rise out of poverty. Inequality remained a key social and economic issue, with the poorest quintile households accounting for only 6 percent of national consumption. Unemployment remained high at 15% of labor-force age Jamaicans, with greater concentration among youth and women. For years, the need to attend to financial sector crises and to contain public spending had constrained much- needed public sector investments in basic services, particularly those used by the poor. Poverty, unemployment, and lack of services were intertwined and led to rising social tensions, especially in poor urban neighborhoods. Jamaica suffers from very high levels of violent crime. Exacerbated by inequality in wealth, domestic violence, youth unemployment, political tribalism and drugs, murder had escalated over the last two decades to one of the highest per capita homicide rates in the hemisphere. The Government's strategy, articulated in its Medium-Term Economic and Social Policy Framework 1999/2000 - 2002/03, was to protect the poor and vulnerable while improving quality of life through increased access to developmental programs, social and economic opportunities and enhanced social harmony. Poverty reduction was to be achieved through a combination of measures to improve incomes, self reliance, and the quality of life of the poorest groups and communities. As a core component of the Government's strategy, the National Community Development Project (NCDP) supported the World Bank's November 30, 2000 Country Assistance Strategy objectives of restoring economic growth, promoting inclusion and protection of the poor, and supporting sustainable development. The NCDP sought to enhance sustainable development of low-income communities by promoting greater community selection of development projects and strengthening local capacity to implement such projects, with the active participation of the population. The Government of Jamaica (GoJ) chose to channel this support through the Jamaica Social Investment Fund (JSIF). The Government had established the Jamaica Social Investment Fund (JSIF) in early 1996 as a core component of the GoJ's national poverty reduction strategy. JSIF was designed to assist in responding to the needs of the most vulnerable groups and communities. Several studies and evaluations confirmed that JSIF had considerable success in reaching a large number of the poor in a manner that was quick, efficient, transparent and non-partisan. JSIF had a proven track record and had built strong capacity in participatory approaches to community development among its implementing partners, including parishes, ministries, NGOs, CBOs and contractors using community driven development (CDD) principles. 1 1.1 Original Project Development Objectives (PDO) and Key Indicators (as approved) Project development objectives were: (a) to help communities in areas affected by economic decline and lack of basic services and (b) to assist the Government of Jamaica to promote greater social and community development, especially among the poor. Key performance indicators for NCDP's basic service and employment objectives: 1. Increase in access to and quality of basic services 2. Number of incremental temporary jobs created 3. Improved operation and maintenance of project facilities NCDP's social development objectives were to be met through two approaches: (a) embedding community subprojects in participatory community development planning; and (b) transferring greater control over resources and decisions to community groups (i.e. adopt a CDD approach), mainly through developing community-based contracting, a first for Jamaica. Key social development performance indicators include: 1. Increase in the number of projects financed by JSIF after being identified in Community Development Plans (CDPs). 2. Increase in the number of communities which develop the capacity to select their own priority projects and to manage project contracting and project implementation effectively to ensure work quality, reasonable costs, and timeliness. 3. Evidence of increased complementary investment generated (from residents, other public sector entities and the private sector) through JSIF and Social Development Commission (SDC) assistance to community groups. 4. Number of subprojects with active community maintenance plans and clear indication of agreed support from line ministries and/or local authorities. 5. Level of community satisfaction with subproject and performance of sponsoring entities, JSIF, SDC, and Community Development Committees (CDCs). 1.2 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification No formal revisions were made to the NCDP. Response to Hurricane Ivan in 2004 prompted an adjustment in activities within prevailing project development objectives. 1.3 Main Beneficiaries (original and revised, briefly describe the "primary target group" identified in the PAD and as captured in the PDO, as well as any other individuals and organizations expected to benefit from the project) The NCDP's main beneficiaries are poor communities in Jamaica using a targeting system based on the nation's poverty map developed by the Planning Institute of Jamaica (PIOJ). The PIOJ map identified poverty zones based on a weighted composite poverty indicator using youth unemployment, primary school education, and exclusive use of toilets and piped drinking water. NCDP prioritizes Quartile 4 (poorest) communities in 2 the poverty map as well as pockets of poverty in other quartiles identified through local indicators and qualitative assessments. Secondary beneficiaries include community groups through learning-by-doing of preparing plans and implementing small-scale investments as well as support to them through technical assistance and training. Community capacity effects would also be transmitted through the experience of formulating Community Development Plans. 1.4 Original Components (as approved) The NCDP has three components: (a) subprojects, consisting of demand-driven community investments to provide basic services while helping to create temporary employment. Community proposals would be generated by a participatory process. In the past, JSIF had directly facilitated these community consultations. Under the NCDP, this task was to be delegated to the Social Development Commission (SDC), tasked with facilitating multi-sectoral Community Development Plans. NCDP set forth two types of sub-project implementation arrangements: (i)"traditional" subprojects where fiduciary arrangements would be handled directly by JSIF, particularly for larger and/or complex works or for situations where community contracting would not be feasible; and (ii) community-based contracting (CBC) with procurement and financial management carried out directly by community organizations (US$100,000 maximum). This was Jamaica's first experience with direct community procurement and financial management. (b) technical assistance and institutional strengthening, including support to the operational capacity of JSIF as well as technical assistance to the SDC and community sponsors in the identification, preparation, implementation and maintenance of subprojects, including training to support the CBC system. (c) administration to enable JSIF to carry out NCDP activities, including monitoring and evaluation. The bulk of JSIF operating costs were to be funded by the GoJ. 1.5 Revised Components The components were not formally revised. 1.6 Other significant changes (in design, scope and scale, implementation arrangements and schedule, and funding allocations) In September 2004, Hurricane Ivan caused widespread damage in Jamaica, including about 5,000 homes and 340 schools in need of reconstruction. In response, the Bank's portfolio was refocused to address reconstruction needs. The NCDP was tasked with rebuilding 100 schools in target areas. Since this could be accomplished within the existing project development objectives and components, no formal revisions were made. 3 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry (including whether lessons of earlier operations were taken into account, risks and their mitigations identified, and adequacy of participatory processes, as applicable) Project design built on the experience of JSIF, analysis of the needs of poor communities in Jamaica and the evolving international best practice in community-driven development (CDD). The previous Bank-funded Jamaica Social Investment Fund Project was rated by the ICR and IEG project evaluation reports as having satisfactorily met its development objectives with a modest institutional impact. Concerns were raised about sustainability issues including uncertainties about JSIF's future since follow-on funding was not sure. Design of the NCDP specifically addressed the key lessons learned, identified as: Improved community-level planning. Rather than support one-off projects developed in the context of a JSIF-managed identification process, the NCDP sought to embed its activities within a broader participatory community planning process, in keeping with the evolving international best practice and guidance on these types of operations. While this is often done in LAC countries through the decentralization agenda, with participatory municipal planning playing a role, however, in Jamaica the parish councils did not yet play this role and decentralized investment planning was very weakly focused around a narrow set of parish responsibilities. On the other hand, the Social Development Commission (SDC), a public entity transitioning from a social welfare-type focus to a community mobilization and facilitation agency, was expanding its mandate to play a more active role in community-level participatory planning, but had little track record in this. The NDCP preparation process determined that the SDC would help communities establish their Community Development Plans, develop funding proposals for JSIF and other entities, which would also help SDC fulfill its mandate in this area. The need for reinforced attention to operations and maintenance. SDC would also be tasked with reinforcing operations and maintenance training to community entities, under a service contract with the NCDP. Although most beneficiary communities are capable of carrying out minor maintenance activities, they often lack financial resources and/or technical expertise for larger tasks. The project sought to help communities to improve their own maintenance efforts as well as improve linkages with line ministries and parishes. Deepening participation and community control. The NCDP reformed JSIF's identification and selection process at the community level, focusing on mobilizing participation of the poor. The transfer of procurement and financial management responsibilities to the community through the use of community-based contracting was central to NCDP's social development objectives. Proper role for JSIF vis a vis other agencies. Project design streamlined roles and responsibilities between JSIF, parishes and sector ministries. Emphasis was placed on JSIF's role as a complement to infrastructure investment and operation and maintenance programs implemented by line ministries, parishes and others. 4 During project preparation, JSIF's Operational Manual was revised to address shortcomings identified in previous evaluations and to guide CBC arrangements. DFID and the EU provided technical assistance to improve JSIFs operations and strengthen the capacity of the SDC to create Community Development Plans. Project preparation identified several key risks. The highest risks were seen as a lack of contractor commitment to labor intensive techniques, mitigated by weighting job creation more heavily in selection criteria; and difficulties in working with communities with high social tensions, addressed by phased payment tranches based on community performance. Other risks, including local coordination and implementation issues, were addressed through adjustments to procedures and criteria in the Operational Manual. In terms of implementation readiness, JSIF had a strong track record in managing Bank loans. The decision to phase in the CBC approach based on an initial EU pilot allowed the NCDP to move cautiously with this innovation. General working principals were agreed with the SDC during preparation. However, a full-blown institutional analysis of SDC was not done during preparation and inter-agency agreement between JSIF and SDC was not signed until well after project effectiveness. 2.2 Implementation (including any project changes/restructuring, mid-term review, Project at Risk status, and actions taken, as applicable) The NCDP was approved in October 2002 and declared effective in April 2003, with a five year implementation horizon. Prior to this, JSIF had accumulated a backlog of applications awaiting funding. This pipeline of subprojects was reviewed against NCDP eligibility criteria which allowed the NCDP to move quickly into executing investments, albeit with this first batch not generated through SDC community development plans. The NCDP phased in its community-based contracting module with the launch of a pilot of 16 CBC-managed subprojects, funded by the EU, starting late 2003. This pilot gave extremely useful input to the NCDP's CBC component, which commenced in mid-2004. NCDP's approach gave communities choice as to three different types of CBC, from a model where the community enters a contract with a contractor who then hires labor and purchases materials, to the more commonly-chosen option of the community directly hiring labor, buying materials and responsible for all phases of the work. This allowed for greater tailoring to community circumstance and preference. Implementation details between JSIF and SDC were not finalized until January 2004. By mid-2004 SDC had delivered 58 project proposals including Community Development Plans to JSIF. However, a large number of SDC-generated subproject proposals were rejected due to poor technical quality, ineligibility, land tenure issues, and a concentration in community centers that did not fit with agreed criteria of sustainability and poverty impact. By mid-2005, the Bank supervision team concluded that the SDC would not be able to deliver sufficient "ready to bid" projects in time to ensure execution before project closing and agreed that the NCDP would have to seek other alternatives. 5 As the NCDP was scaling up the CBC module and working with SDC to iron out implementation issues, Hurricane Ivan struck in September 2004. Streamlined appraisal and contracting procedures were adopted to respond to the emergency, including waiving community counterpart contributions. Fiscal and market conditions caused implementation challenges as well. In response to fiscal shocks in 2004, the OPEC Special Fund allowed JSIF to utilize its US$5 million NCDP co-financing to fund part of the government counterpart requirements, allowing NCDP to continue operating during this critical period. A Mid-Term Review in January 2006 reported substantial meeting of the project's development objectives. Specific issues identified were the need to hire consultants to reinforce operations and maintenance training given the limitations of SDC performance in this area and reinforced CBC training and site visits to address some delays. Implementation Status Reports rated achievement of development objective and implementation progress as satisfactory throughout the lifetime of the project. The Project received a six-month extension and closed on June 30 2008, mainly to allow for completion of the Summative Evaluation, which was delayed in contracting. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization The NCDP built on the experience and capacity of JSIF in monitoring and evaluation. The MIS from JSIF's first phase had a sophisticated design but poor linkages between certain modules and lagging data input. The system was redesigned in 2000, with further adjustments made in 2002. In terms of evaluation, a robust quasi-experimental impact evaluation had been carried out at the end of JSIF's first phase providing substantial information on successful outcomes. The evaluation approach of the NCDP was to build on this experience. Adequate indicators were identified to monitor progress toward NCDP's development objective, but no quantitative targets were established at appraisal. In terms of actual performance in M & E, several iterations of MIS redesign and changes in key MIS staff during implementation affected progress on internal consolidation of the system. At the end of the project, the MIS was functioning as a subproject record, particularly in terms of registration, appraisal, contracting and execution, but most reports were not being used for decision-making and the modules on performance monitoring were not fully operational. In terms of evaluation of NCDP outcomes, outcome indicators were agreed with the Bank in mid-2004 and although baseline and follow-up surveys were anticipated, the consultancy to conduct the baseline survey was delayed due to a number of factors. An initial baseline was developed incorporating ex-ante appraisal information and Census data. In the end, the baseline did not provide sufficiently robust information so the final impact evaluation adopted the use of comparator communities to create the counterfactual. The final summative evaluation was conducted by an international research group using household surveys, community score cards, a process evaluation and an infrastructure evaluation to determine NCDP's development outcomes. 2.4 Safeguard and Fiduciary Compliance 6 (focusing on issues and their resolution, as applicable) Fiduciary Management JSIF has a strong track record of successfully implementing the Bank's FM requirements. JSIF has an internal auditor and an Audit Committee of its Board of Directors. External audit reports were delivered on time and were not qualified. Some challenges were encountered in upgrading systems, like AccPac and Fund Manager, but streamlined financial reporting through the use of FMRs was satisfactorily achieved. The FM aspects of the project were temporarily downgraded to marginally satisfactory for a time in 2006 based on deterioration in accounting procedures and controls, largely due to ongoing staff changes. Nonetheless, the Bank found that none of these shortcomings posed a significant risk to JSIF's financial integrity. Moreover, JSIF's Financial Management team successfully implemented the innovative use of community financing agreements. To implement the new CBC approach to procurement, JSIF carried out extensive training on CBC to community representatives. JSIF ensured that procurement procedures on CBC subprojects were held to an equally high standard of transparency by engaging the National Contracts Commission Office of the Contractor General in monitoring. Ex-post procurement reviews were systematically carried out during implementation. The only significant issue raised was the concentration of contracts among a limited number of contractors largely due to the emergency packaging done after Hurricane Ivan, with prior approval from the Bank, the types of communities NCDP reached (poor, high conflict) as well as its strict unit cost controls. JSIF furthered transparency by being the only public agency to publish awarded contract amounts in the newspapers. In 2004, JSIF received an award from the Master Builders of Jamaica for transparency in awarding contracts. Environmental safeguards NCDP preparation included a detailed environmental management plan and revised subproject appraisal criteria. An external evaluation of subprojects under implementation confirmed that guidelines were being followed and found that less than 20% showed minor potential environmental impacts that were easily mitigated. JSIF also developed an Environmental Management Framework (EMF) to comply with the requirements of Jamaican environmental laws and regulations and the relevant policies of its funding agencies, covering all projects financed by JSIF, not just NCDP. In order to ensure the effectiveness of the EMF, JSIF implemented an Environmental Management System to identify, monitor and control environmental aspects. JSIF is currently in the process of international certification for its environmental management through ISO 14001, which is the most stringent of international environmental standards. 2.5 Post-completion Operation/Next Phase (including transition arrangement to post-completion operation of investments financed by present operation, Operation & Maintenance arrangements, sustaining reforms and institutional capacity, and next phase/follow-up operation, if applicable) Sustainability at the community level 7 At the community level, there were shortcomings with some of the intended mechanisms to promote operations and maintenance. The expected support from SDC in O&M training was more limited than anticipated and the Summative Evaluation found maintenance planning to be lacking. JSIF is completing the community maintenance training through the on-going Bank-funded Hurricane Dean Emergency Recovery Loan. However, the data on actual community performance and engagement in O&M paints a more positive picture. The Summative Evaluation found that community interest in maintenance was much higher in NCDP communities (at 86%) than comparators (at 60%). 71% of NCDP households stated that infrastructure/equipment is being maintained by the community versus 42% in comparator communities. This improved outcome was also reflected in site visits by engineers who found indicative results that 67% of NCDP subprojects had good to average maintenance versus 50% for comparator infrastructure. The increased sense of community ownership and engagement in the development process may have driven these more positive outcomes. JSIF itself sought to aggressively change the maintenance culture by establishing clear community maintenance responsibilities and expectations. However, it is worth pointing out that maintenance is considered a shared responsibility, with less than half of respondents in all communities stating that the community alone is responsible for maintaining basic services. In terms of provision of staff and equipment to schools and health centers, coordination with the line ministries was supported by generally good performance nationwide in Jamaica in staffing health and education facilities. In terms of the post-completion phase, the Ministry of Education is making internal arrangements to pay for the repair and construction of schools through the establishment of a National Educational Trust Fund (NETF) under the forthcoming Education Transformation Project. Nonetheless, facility maintenance remains a systemic concern in Jamaica. As a result, under NCDP, JSIF modified its technical designs to reduce future maintenance needs. At the national level, JSIF spearheaded the formation of a multi-agency National Maintenance Committee in 2006 to develop national policies and attracted the support of the Caribbean Development Bank to develop maintenance systems. In terms of sustainability of community institutions, NCDP reinforced existing community based organizations (CBOs). In order to receive NCDP funds using CBC they had to become legal entities with a reporting and maintenance structure. This was accomplished by establishing Benevolent Societies which report to the Ministry of Industry and Commerce's Department of Cooperatives and Friendly Societies, which requires annual audits and provides ongoing technical assistance. This support system increases the ability of the CBOs to maintain the projects and even develop others. At Project closing, about one-quarter of these Benevolent Societies had been successful at securing additional funding from other sources for community development efforts. Sustainability at the national level In terms of the sustainability of JSIF's community development efforts, JSIF has ensured its continuation in the medium-term and consolidated its position as the key community development agency of the Government, with a funding base of almost US$90 million for 8 the current second phase, with multiple donors, Government, community and private sector support (Annex table c). Three subsequent Bank operations managed by JSIF help sustain the gains made under NCDP, including: The Inner Cities Basic Services Project, approved in March 2006, to improve quality of life in Jamaican inner-city areas and poor urban informal settlements through improved access to basic urban infrastructure, financial services, land tenure regularization, enhanced community capacity and improvements in public safety. CBC is included in the ICBS Project to further community empowerment and savings. The Hurricane Dean Emergency Recovery Loan, approved in January 2008 after a devastating hurricane struck in August 2007, to restore community infrastructure and strengthen the GOJ's ability to respond to future natural disasters. Any NCDP infrastructure damaged by Dean has been prioritized under the Loan. The Second National Community Development Project is currently under preparation. Based on the experience of NCDP, NCDP II will focus on rural areas and seek to enhance income generating activities among rural producer and community groups. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation (to current country and global priorities, and Bank assistance strategy) The NCDP development objectives of improving access to and quality of services in poor communities and building community capacity remain highly relevant today. The World Bank's current CAS for Jamaica (2006-2009) is aligned with the government's Medium Term Socio-Economic Policy Framework and is based on three mutually-reinforcing pillars: accelerating inclusive economic growth, improving human development and opportunity, and crime prevention and reduction. The NCDP was seen as a core part of the Bank's current CAS, as well as the ICBSP and NCDP II follow-up operations. The CAS also emphasizes the two cross-cutting themes of governance and environmental sustainability. The NCDP, with its CDD approach to increased bottom-up accountability and community-level environmental management, further these objectives. 3.2 Achievement of Project Development Objectives (including brief discussion of causal linkages between outputs and outcomes, with details on outputs in Annex 2) Output quantity, quality and targeting NCDP financed 287 community subprojects in poor communities at an average subproject cost of US$86,000. These subprojects generated a significant increase in infrastructure and services, including 194 rehabilitated schools, 7 health centers, 6 agro- processing facilities, 4 water and sanitation facilities, 183 life coping skills programs, 11 integrated community service centers and over 42 kilometers of rehabilitated and upgraded community roads (see Annex 2 for outputs). Over 90 percent of the portfolio was concentrated in education, health and roads. Over US$4 million was implemented using CBC, surpassing the PAD target of US$3 million. Hurricane Ivan reconstruction 9 accounted for 93 projects (16 percent of the portfolio), resulting in rapid reconstruction of about one-third of the affected schools. Overall, an estimated 149,000 residents of poor communities in Jamaica benefited from NCDP. The outputs were of good quality and highly relevant to community needs. According to the infrastructure evaluation, "all the project facilities funded by the NCDP compare favorably with the control facilities visited in terms of both construction quality and level of services being provided." The evaluation further noted that quality of the infrastructure provided appeared to be due to the level of supervision of construction and the enthusiasm and commitment of the sub-project sponsors. Participatory community prioritization methods were carried out in all communities and only the two top ranked needs were eligible, ensuring community relevance. Across the education, health and road subprojects, an average of over 90 percent of community members were satisfied. Although there were no specific key performance indicators related to poverty targeting, subprojects appeared well-targeted to poor communities. 61% of subprojects were in the poorest Quartile IV communities, well above the guidelines of 40% in JSIF's Operational Manual. From the Summative Evaluation, over 95% of respondents felt the community selected was `deserving'. In terms of who benefits within communities, 72% of people in NCDP communities felt NCDP subprojects benefited poor people versus comparison communities where only 44% said development projects benefited the poor. Access to infrastructure and services These outputs in terms of social and economic infrastructure created resulted in improved outcomes for community residents. The Summative Evaluation (see Annex 5 for methodological details) covered the three main sectors of intervention: education, health and roads. Overall, 62% of NCDP community members felt that the community was a better place to live than before, compared with 45% in comparator communities. Specific findings by sector include: Education. Enrolment in basic education is compulsory in Jamaica, so NCDP education investments were focused on improving school quality, expanding access to pre-school and, after Hurricane Ivan, preventing a disruption in schooling. The impact evaluation found significant improvements in the NCDP-supported schools: Improved work and study conditions for NCDP education sub-projects (96% of NCDP beneficiaries versus 67% for comparison groups). This finding was reinforced by the general agreement among focus group participants across all the school sub- projects that the work and study conditions had improved. Improved school operations (64% of respondents ­ not queried of comparators). Better learning outcomes, with improvements in student performance noted in 60% of treatment communities versus 48% in comparison communities. Health facilities. NCDP impacts on health services were noted by beneficiaries: 10 Increased access to health services (75% in NCDP communities noted an increase in their access to health services versus 65% in comparison communities) Increased utilization of health care facilities (cited by 79% of NCDP beneficiaries versus 70% in comparators). A slight improvement in perceived quality of health services (58% cited improved quality in NCDP communities versus 54% in comparators). Improved operation of health centers (cited by 70% of NCDP community members ­ not queried of comparators). Perceived improvements in the quality of health service were relatively equal between the NCDP communities at 57% and the comparison communities at 54%. Women were the main beneficiaries of this health impact, with focus groups noting a large increase in utilization by pregnant women. Roads. Road investments focused on rehabilitation and upgrading rather than creating new roads since Jamaica has one of the highest road densities in the Region. Overall, 98% of NCDP community members were satisfied with road improvements versus 19% for comparison communities. In terms of impact, NCDP community members noted: Improved quality of life (noted by 93% of NCDP beneficiaries versus 30% for comparators) Decreased travel times (92% of households reported reduced travel time in NCDP communities versus 47% in comparison communities) Increased safety of mobility (cited by 68% of beneficiaries in NCDP communities versus 40% in comparators) Reduced transport costs (25% of respondents in NCDP communities felt that transport costs had reduced significantly versus 4% in comparison communities) Economic activity increased as a result of the road improvements, with surveys noting an increase in perceived access to market and shops (86% for NCDP communities and 78% for comparison communities), although there was no significant difference noted in perceived availability of market trucks and public service vehicles. Temporary employment outcomes 15,420 of person-months of unskilled temporary employment were generated under NCDP with wages accounting for 60% of subproject costs, consistent with a labor- intensive approach. The impact assessment found that 79% of households in communities with road projects cited the creation of many temporary jobs in the community as a result of the project, versus 61% for health projects and 39% for education projects. Focus group interviews found that, given economic hardships in these communities, the creation of employment opportunities during construction was highly prized by the community. Social development objectives The NCDP's social objectives were twofold: (a) to embed NCDP investments within participatory community planning via SDC and (b) transfer greater control and responsibilities to the community level via community-based contracting. Several notable achievements were made, though the community planning experience was mixed. 11 The NCDP developed and successfully implemented Jamaica's first experience with CBC. JSIF spent considerable effort in training sponsoring organizations and monitoring CBC activities to ensure proper fiduciary management. Although there was initial JSIF staff resistance as it required more community interactions, the experience was found to be an effective option, with possible community capacity spill-over benefits. CBC projects were cost efficient, with 30% of communities able to generate a savings that could be applied to other community efforts. These savings were largely due to greater community oversight over contractors and community purchasing. Moreover, CBC approaches were able to generate twice the average level of community contributions, at 25% versus 12% across the NCDP portfolio. The infrastructure evaluation found generally acceptable quality of construction across the board, with no differences by contracting type. In addition, the creation of Benevolent Societies and their hands-on experience with managing funds and contracts created community capacity beyond the NCDP, with one-quarter able to successfully raise funds from other sources. Overall, there was an increase in the number of communities which developed the capacity to manage investments. The other goal, that of embedding NCDP subprojects within participatory community development plans, was not fully realized. Although 183 CDPs were developed, only 15 viable subproject proposals stemmed from them. SDC lacked capacity and the agreement between JSIF and SDC was ill-fitting in terms of the different objectives and rhythms of the two institutions. SDC was recently moved to the Ministry of Youth and Sport, rather than Local Government, and it is not clear whether its community facilitation mandate will persist. At the same time, Jamaica's decentralization agenda has not advanced, with parishes having relatively few responsibilities, little funding and no tradition of participatory community planning. It is also worth noting that with the rapid response to natural disasters, NCDP fostered the creation of a mechanism that can reach poor communities quickly and efficiently rebuild affected infrastructure. This is an important element of community-based risk management, particularly as shocks to poor communities tend to have more lasting long- term effects. 3.3 Efficiency (Net Present Value/Economic Rate of Return, cost effectiveness, e.g., unit rate norms, least cost, and comparisons; and Financial Rate of Return) As detailed in Annex 3, efficiency and effectiveness was achieved under NCDP. Looking at the program as a whole, overhead costs were contained to 18%, well below a PAD estimate of 38% and within international norms for CDD and social fund operations. This allowed for more funding to be dedicated to investments. NCDP leveraged an average of 12% in community counterpart contributions, surpassing the 10% target. Cost Efficiency of NCDP Subprojects Looking at the economic efficiency of the community investments reveals that, in terms of cost efficiency and control of unit costs, the Summative Evaluation found that "JSIF 12 rates for construction work are considered `below the market rate'. While comparisons with unit costs from other agencies was limited, the ICR mission did find that on construction of a 3 classroom module, JSIF's costs were between 12-17% lower than construction of a similar classrooms by Jamaica's Culture, Health, Arts, Sports and Education Fund. Cost variations during construction were also contained within industry norms, and below 20% for the NCDP portfolio as a whole. Community-based contracting appears to have encouraged additional cost savings, with over 30% of CBC subprojects generating savings that were directed to complementary community investments. Cost-Effectiveness of NCDP Subprojects Cost-benefit analysis is not applicable to the education and health investments since benefits are not fully quantifiable, but evidence points to significant improvements in access to and quality of services with positive impacts on households. Road rehabilitation and upgrading typically have high economic rates of return in Jamaica and similar roads under the previous JSIF project had acceptable economic rates of return. NCDP's road investments were in line with national priorities and higher-return investments. The Summative Evaluation confirmed the economic impact from NCDP road investments, including significant improvements in access to markets and shops and safety and significant decreases in travel times and costs compared to comparator roads. The methodology of the Summative Evaluation did not allow for the calculation of an ERR. 3.4 Justification of Overall Outcome Rating (combining relevance, achievement of PDOs, and efficiency) Rating: Satisfactory The satisfactory rating is based on the NCDP's relevance as part of the national poverty reduction strategy and Bank's CAS, the strong performance on improving the wellbeing of residents of poor communities and creating temporary employment as well as progress on some of the social development objectives, and the continued efficiency of JSIF's performance in facilitating investments in poor communities in Jamaica. This rating is consistent with the achievements against NCDP's key performance indicators. 3.5 Overarching Themes, Other Outcomes and Impacts (if any, where not previously covered or to amplify discussion above) (a) Poverty Impacts, Gender Aspects, and Social Development The immediate poverty impacts appear strongest for roads, where community-wide economic effects were noted. Investments in human capital through improved education and health will have medium-term poverty reduction effects measurable beyond the narrower timeframe of project implementation. Moreover, the rapid response to natural disaster effects provided poor communities with effective coping with these shocks. Women benefitted significantly from the NCDP subprojects. While social and economic infrastructure conferred community-wide benefits, the focus group discussions pointed to women as the prime beneficiaries of the health centers, particularly in terms of expanded access to pre- and post-natal services. Gender-disaggregated data on community group 13 members was not collected, but again focus group discussions pointed to a very active role for women in these organizations. Social development aspects related to participatory community planning and community control over project implementation aspects is discussed at length in Section 3.2 above. (b) Institutional Change/Strengthening (particularly with reference to impacts on longer-term capacity and institutional development) Institutional Impacts at the Community Level The institutional impacts of NCDP within targeted communities are significant. JSIF enabled 186 community sponsor groups to carry out community investments. The ability of these sponsor groups to identify and implement their goals was empowering and strengthened community capacity. Over 50 CBOs and 640 CBO members received training in community resource management skills. Participants of CBC evaluation workshops identified the benefits from this approach in terms of leadership experience, management skills, bookkeeping skills, conflict resolution and communication skills, and construction skills and that these skills acquired could be used `in personal life' and `future community projects'. They also noted greater social cohesion (e.g. enhanced community development as it brought together various groups within the community') and that `communities can do things without political involvement'. Because each community using CBC required a legal entity to manage the funds, 33 new Benevolent Societies were formed. These organizations are under the direction of the Government's Department of Cooperatives and Friendly Societies, which provides ongoing support and carries out annual audits, thus enhancing the sustainability and capacity of these groups. Institutional Development of JSIF JSIF has an established track record in implementing GoJ poverty alleviation projects. Its institutional capacity has enabled it to attract about US$90 million in on-going funding, including OPEC, CDB and EU-financed social/community-level infrastructure investment programs using the approaches developed in the NCDP. Of note in terms of institutional development impacts: The NCDP created rapid response capacity for natural disasters, a consequence of JSIF's engagement in the Hurricane Ivan reconstruction. The creation of this capacity was important in responding to subsequent natural disasters. For example, JSIF's response to Hurricane Ivan under NCDP resulted in the organization also becoming responsible for the Hurricane Dean ERL and JSIF now plays a core function in Jamaica's ability to reconstruct from its recurrent natural disasters. In one of the very few examples worldwide, JSIF's efficiency and transparency has attracted private sector contributions, including Digicel Foundation, Red Stripe, Rotary, and Chase who have been using JSIF as an implementing agency for privately-funded community development projects. Impacts on Other Institutions 14 NCDP's effect on SDC was more limited than anticipated at the outset. NCDP supported SDC's shift, initiated in 1998, from a social welfare-oriented agency to its current community empowerment and mobilization focus. NCDP enabled SDC to carry out Community Development Plans in 183 communities, or about half of SDC's overall target of reaching 45% of Jamaica's 786 communities. SDC's field staff were trained in operations and maintenance through the NCDP. Lift UP Jamaica, a GOJ program involved in school construction, also partnered with SDC to facilitate community mobilization. Lift UP Jamaica recognized the value of the training JSIF provided to SDC and indicated that their decision to work with them was based on this acquired competence. However, SDC lacks the technical capacity and credibility to be the primary community mobilization and participatory planning mechanism for the country. Its interventions are still viewed in an ad-hoc nature, with SDC brought in where either an assessment or facilitation is needed and Community Development Plans have not yet gained traction as a core organizing vehicle for community efforts in Jamaica. In terms of the education sector, JSIF pioneered more appropriate designs for equipping basic schools. JSIF's Hygiene Training Module is being incorporated into the national school curriculum and has already been modified for use in basic schools. JSIF's designs have now become standards under the Early Childhood Commission. And, JSIF's Community Maintenance Manual is being used in Lift UP Jamaica projects. (c) Other Unintended Outcomes and Impacts (positive or negative) 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops (optional for Core ICR, required for ILI, details in annexes) Findings of the Summative Evaluation are presented in the sections on Achievement of Development Objectives and Risks to Development Outcomes. Details on methodology and approach are presented in Annex 5. 4. Assessment of Risk to Development Outcome Rating: Moderate NCDP deepened JSIF's capacity to assist poor communities as part of the Government's poverty reduction strategy. Engagement in the process and satisfaction with the outcomes has generated community and sectoral commitment to maintaining these benefits. Performance in this area has been better than in comparator communities, though systemic challenges remain. The overall social objectives of furthering community-driven development and building responsive community capacities is continuing to be implemented through JSIF's expanding portfolio of operations. 15 5. Assessment of Bank and Borrower Performance (relating to design, implementation and outcome issues) 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry (i.e., performance through lending phase) Rating: Moderately Satisfactory The Bank's initial support for the creation of JSIF was an important step in the GOJ's poverty eradication initiative and it opened the door for assistance from other donors and lenders to be channeled through JSIF. NCDP was a core part of the Jamaica's national strategy and the Bank's prevailing CAS and served a clear need in poor communities. The Bank provided significant value-added to the technical, financial and economic aspect of project design. The Bank's extensive experience in social funds and community-driven development lent critical technical guidance during project preparation. The Bank facilitated the design of the CBC approach through workshops in Washington and contact with consultants from other countries with experience in CBC. The Bank also engaged effectively with JSIF during the preparation process in ensuring its fiduciary and environmental safeguard systems were ready prior to project launch. Moderate shortcomings in quality at entry include: General monitoring and evaluation approaches and instruments were identified, but specific outcome indicators, baselines and targets were not developed ex-ante for most of the key performance indicators. Key risks were identified, including the ability of SDC to work with communities throughout the project cycle. While training and periodic reviews were called for, it does not appear that an institutional capacity assessment of SDC was carried out. Moreover, a signed implementation agreement would have been useful to require up front. As it turned out, there was a year delay in coming to agreement and significant differences in understanding and priorities between the two parties. (b) Quality of Supervision (including of fiduciary and safeguards policies) Rating: Satisfactory The Bank team carried out consistent supervision efforts, including semi-annual supervision missions and a Mid Term Review. There were three Task Team Leaders over the life of the project, though the same TTL was in place during the bulk of project implementation. Team composition was strong on FM/procurement and provided technical support to the implementation of the CBC component, including conducting a roll-out of the Bank's CDD Fiduciary Management Training Package in Jamaica in 2002 and arranging training events in Washington and study tours abroad. During the ICR mission, the client noted that the more frequent rotation in the procurement and financial management staff members of the Bank team caused some differences in guidance given. 16 The Bank supervision team identified most implementation problems rapidly and worked with JSIF to address them in a timely fashion. Supervision was particularly effective in: Conducting extensive subproject site visits during missions which allowed the team to collect first-hand impressions of implementation issues in the communities. Intensive supervision efforts in financial management and procurement, including frequent ex-post procurement and SOE reviews and capacity building activities was important as the NCDP was the first project in Jamaica to use CBC methods. Significant attention was also placed on environmental management issues, with the Bank supporting JSIF's ISO accreditation process. The Bank displayed flexibility in working with JSIF to develop its rapid response to Hurricane Ivan and in adjusting the financing parameters for subprojects to simplify execution when the Bank's new Country Financing Guidelines were put in place. Overall commitment to keeping implementation on track, for example monthly contracting and disbursements schedules were submitted to the Bank However, there were some minor shortcomings in supervision. There was incomplete monitoring of key performance indicators in the ISRs. In addition, although a potential lack of cooperation between JSIF and sector entities was rated as a high risk, the issue was underappreciated in supervision activities, with few meetings with the sector ministry or parish staff during supervision missions. While the Bank provided a high level of technical support to M&E including inclusion of an MIS expert in the project team and incorporation of impact evaluation experts to advise JSIF, the difficulties in effectively implementing monitoring and evaluation tools should have been reflected in changes to the ISR M&E ratings during those periods. (c) Justification of Rating for Overall Bank Performance Rating: Satisfactory Overall, the Bank team's performance was satisfactory, with effective partnership to develop a well-designed project and a strong showing on most implementation issues, in particular fiduciary management and safeguards and extensive field visits to gather insights on subproject implementation and community dynamics issues. 5.2 Borrower Performance (a) Government Performance Rating: Satisfactory The role and mandate of JSIF was clearly established in the national poverty reduction strategy. The Government was highly supportive of NCDP's innovative approaches furthering community-driven development. GoJ consistently provided counterpart funding to NCDP and when the Government was facing major fiscal constraints, requested that OPEC funding fulfill the GoJ's contribution. Government representatives on JSIF's Board of Directors ensured that JSIF's operations were transparent and efficient, which resulted in JSIF continuing its operations through changes in Administration. Higher levels of Government might have better mediated the difficulties 17 between JSIF and SDC, but in the end SDC's state of readiness upon signing the MOU with JSIF was not compatible with the design of the project (b) Implementing Agency or Agencies Performance Rating: Satisfactory JSIF effectively implemented NCDP. JSIF staff took on the challenge of transferring greater control and responsibility to community groups, which represented a departure from usual practice and increased their workload and implementation complexity. Administrative and investment costs were efficient compared with both national and international benchmarks. JSIF's transparent procedures, including being the only agency in Jamaica to publish contract amounts, won it a national award. JSIF responded quickly to the reconstruction needs following Hurricane Ivan, adapting procedures to ensure timeliness in its response capabilities. Despite this overall success, JSIF faced some challenges in implementing the NCDP. Turnover of technical staff, often times to international agencies due to their high quality, was responsible for some slowdowns in implementation. JSIF failed to develop the MIS as an effective management tool, despite significant investments in system design. In terms of the breakdown in the agreement between JSIF and SDC, there is blame on both sides. The basic problem was a mismatch in the timeframe and quality of community facilitation services needed for NCDP and SDC's institutional priorities and capabilities. And, while JSIF was exemplary in terms of the cost efficiency of the organization, it was not able to meet the ambitious processing efficiency targets set at appraisal. Many of the factors, like land titling, wait times for no objections from line ministries and parishes, and the buildup of demand from enthusiastic communities, do not reflect on JSIF's efficiency. And, longer processes to ensure proper completion of infrastructure or training of communities in CBC enhanced the quality of the end products. (c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory The Government of Jamaica and JSIF implemented the NCDP with due diligence and flexibility in responding to needs and implementation issues as they arose. The core objectives of the NCDP were met and in many areas targets were surpassed. 6. Lessons Learned (both project-specific and of wide general application) JSIF has assumed an important institutional niche in Jamaica and has become the platform for experimenting with community-driven development approaches across multiple funding agencies. Initially set up as a temporary agency, under the NCDP JSIF has continued to innovate with approaches to poverty reduction at the community level and to provide an example of transparency and good governance. As a result, the NCDP experience has shown that a multi-sectoral, poverty-targeted agency can be a well- integrated instrument of Government policy. 18 Ideally, community projects should be embedded in participatory local planning, but this is a challenge to develop as a national instrument. Jamaica did not have a functioning system of participatory community development planning. NCDP correctly identified SDC as the agency having the closest to this mandate, but had little capacity to readily fulfill this role. CDD projects face a quandary in this circumstance and good institutional analysis upfront to determine capacity and responsibilities is essential. Even without this broader community planning, very relevant and sustainable community investments can be made provided there are effective sectoral linkages and good community outreach. Community-based contracting is an important option to develop within CDD projects. The NCDP experience found cost savings with no loss of quality when carrying out CBC. There may be spillover benefits to local capacity building, as seen in the creation of Benevolent Societies that have been able to generate additional funds for their communities. However, staff may initially resist the added complexity of transferring responsibilities to communities and will need intensive training and orientation. Rather than an either/or approach to community-based contracting arrangements, providing a menu of options for different types of CBC alongside traditional fiduciary management creates flexibility and allows for tailoring to circumstance and community preference. Service access goals can be met over the time horizon of a single 5 year project, but broader changes at the community level may take a longer time to take root and more experimentation with effective approaches. Social development objectives, like developing community institutions, building social capital, developing trust in and between communities, and changing longstanding inter-community relationships may take longer and not be a straight path. These broader social objectives should be modest and well-focused in the short-term. And, it is important to find ways of documenting these on-going changes where simple quantitative indicators are insufficient. NCDP's response to Hurricane Ivan demonstrated that natural disaster reconstruction capacity is important in reducing vulnerability of poor communities. Based on its experience in Hurricane Ivan, JSIF was able to respond quickly to Hurricane Dean. Existing agencies like JSIF can be put to the task quickly rather than resorting to ad hoc arrangements. This role of vulnerability reduction should be recognized in project development objectives and reflected in key performance indicators. In monitoring and evaluation, simple, implementable system designs and upfront definition of targets, methodologies and expected MIS system outputs are essential. Acknowledging that the NCDP spent considerable efforts on the development of the MIS and evaluation studies, in both cases the quality of results were less than hoped for. The focus on revisions to the MIS to build a more sophisticated system detracted from ability to update and fill in data and create a user-friendly system. In terms of evaluation, given the complexity of the challenge of building robust evaluations, projects need to start out with a clear idea of the specific evaluation strategy and terms of reference, rather than develop approaches during implementation. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 19 The ICR team has no comments on the issues raised by the Borrower as their contribution is consistent with the findings of this ICR. (b) Cofinanciers Not Applicable. (c) Other partners and stakeholders (e.g. NGOs/private sector/civil society) Not Applicable. 20 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Actual/Latest Components Estimate (USD Estimate (USD Percentage of millions) millions) Appraisal SUBPROJECTS 18.30 24.58 134% TA AND INSTITUTIONAL STRENGTHENING 3.04 0.45 15% ADMINISTRATION 8.16 4.79 59% Total Baseline Cost 29.50 29.82 Physical Contingencies 0.00 0.00 0.00 Price Contingencies 0.00 0.00 0.00 Total Project Costs 29.50 0.00 Front-end fee PPF 0.15 0.00 .00 Front-end fee IBRD 0.00 0.00 .00 Total Financing Required 29.65 0.00 Note: Institutional support (TA and administration) were dimensioned at appraisal prior to knowing future funding commitments to JSIF. These have turned out to be higher than expected, resulting in spreading JSIF's fixed cost over a larger pool of funders, lowering the relative share of administration and TA to be financed by NCDP, freeing up Project resources for more subproject. (b) Financing Appraisal Actual/Late Source of Funds Type of Estimate st Estimate Percentage Cofinancing (USD (USD of Appraisal millions) millions) Borrower 11.15 11.87 106% Local Communities 0.91 2.95 325% International Bank for Reconstruction and Development 15.00 15.00 100% Foreign Multilateral Institutions (unidentified)* 2.59 0.00* 100% 29.65 29.82 *US$5 million from OPEC's funding is included under Government counterpart (c) Consolidated JSIF Financing 21 Amount (US$M) Closing Date Type Closed Funding Agreements Government of Jamaica 15.93 Counterpart Beneficiary Communities 2.55 Counterpart Government of Japan 0.5 12/1996 Grant Government of the Netherlands 2.8 3/2001 Grant OPEC 2.0 6/2001 Loan IBRD (JSIF 1) 20.0 7/2001 Loan DFID 0.77 12/2001 Grant IADB 9.5 7/2002 Loan Japanese Policy and Human 0.65 3/2006 Grant Resources Development (PRHD) European Union (Poverty 3.5 6/2006 Grant Reduction Programme) Caribbean Development Bank 14.25 12/2006 Loan Japanese Policy and Human 1.8 10/2006 Grant Resources Development (PRHD) TOTAL 77.15 Phase II Funding Government of Jamaica 13.4 On-going Counterpart Beneficiary Contribution (NCDP 2.95 6/2008 Counterpart estimate) IBRD Emergency Recovery 10 6/2011 Loan Project EU Poverty Reduction Programme 9.7 12/2011 Grant 2 PHRD NCDP II 0.5 4/2009 Grant IBRD ­ Inner City Basic Services 29.3 12/2011 Loan Project OPEC 5 6/2009 Loan Basic Needs Trust Fund (CDB) 2.86 6/2010 Grant IBRD - NCDP 15 6/2008 Loan Red Stripe .028 TOTAL 88.74 Not including NCDP 2 under preparation: $26 million total, of which $15 million IBRD, $9 million Government of Jamaica 22 Annex 2. Outputs by Component The NCDP has three components: (a) subprojects, consisting of demand-driven community investments in social and economic infrastructure, social services and training; (b) technical assistance and institutional strengthening, including support to the institutional, operational and administrative capacity of JSIF as well as technical assistance to the Social Development Commission, NGOs and community groups in the identification, preparation, implementation and management of subprojects; and (c) administration to enable JSIF to carry out project activities, including monitoring and supervising the NCDP. Specific outputs by component encompass: 1. Subprojects NCDP Portfolio Breakdown Table 1: by Project Type Table 2: by Poverty Quartile Type Total Quartile Total Agro Pro. 6 (wealthiest) 1 10 Conflict Resolution 1 2 33 Counseling 2 3 71 Health 7 (poorest) 4 179 Homes 3 Integrated Community Spaces 11 Life Coping Skills 38 Road 21 Total 293 Sanitation 2 School 194 Skills Training 3 Skills Training - Equipping 2 Skills Training - Infrastructure 1 Water 2 Total 293 Table 3: By Sector Sector Total Economic Infrastructure 29 Social Infrastructure 227 Social Services 37 Total 293 23 Table 4: Distribution of NCDP Subprojects by Implementation Type Funding Source Implementation Type No. of Subproject WB CBC 51 Ivan 89 Traditional 91 SUB-TOTAL 231 GOJ Ivan 4 Traditional 29 SUB-TOTAL 33 GOJ / OPEC CBC 1 Traditional 22 SUB-TOTAL 23 TOTAL CBC 52 Ivan 93 Traditional 142 TOTAL 287 Table 5: Distribution of NCDP Portfolio by Amount, including Community Contribution % Implementation Community Community Type No. NCDP Cost J$M Total Cost Contribution* Contribution CBC 52 211,438,987.00 281,456,346.00 70,017,359.00 24.88% Ivan 93 240,419,795.00 240,419,795.00 0.00 0.00% Traditional 142 867,884,986.00 982,984,690.00 115,099,704.00 11.71% Total 287 1,319,743,768.00 1,504,860,831.00 185,117,063.00 12.30% *Per community financing agreements 2. Technical assistance and institutional strengthening Key activities and outputs under the technical assistance and institutional strengthening include: Training and study tours to Washington, Africa and Eastern Europe for JSIF staff to gather international best practice in community-driven development Extensive CBC training at the community level, which included in addition to technical assistance to individual community sponsors, multi-community 24 workshops and evaluation workshops to gather community input on the performance of the mechanism. Technical assistance and external consultants to evaluate environmental impacts of NCDP subprojects and management practices in NCDP as well as improve the Operational Manual in these aspects Baseline study and Summative Evaluation contracted to international firms Technical assistance to upgrade the Management Information System Technical assistance funding to the SDC for the development of 135 Community Development Plans and 15 operations and maintenance plans completed (in addition, 135 Community Development Officers of SDC were trained in O &M through complementary EU financing). 3. Administration NCDP supported: Two JSIF senior managers were financed under the Loan Annual external audits Equipment, including computers, printers and one vehicle as well as accounting software to support JSIF as NCDP implementing agency 25 Annex 3. Economic and Financial Analysis (including assumptions in the analysis) At appraisal, an overall NPV or ERR was not calculated as it was not applicable to this type of demand-driven project where the sectoral distribution of investments are not known upfront. Moreover, in most instances, the type of the sub-projects financed, their size, location and other characteristics, such as school rehabilitation or health center construction, do not allow for traditional economic rate of return analyses. However, a very strong proxy for such analyses emerges when beneficiaries are given the opportunity to assess their needs and establish priorities. Moreover, Internal Economic Rate of Return (IERR) analyses were done on a sample of subprojects under the first JSIF operation showed substantial economic benefits. In 2001, an impact-evaluation study was conducted by an external consultant to measure the impact of JSIF sub-projects. In particular, economic analyses were performed on water and road projects with very favorable results. The Internal Economic Rate of Return (IERR) for the 12 rural road projects examined ranged from 7% to 23%, with an average of 13%. The IERR for the 9 water projects ranged from 83% to 303%, and averaged 198%. Such results can be expected when beneficiaries are given the opportunity to assess their needs and establish priorities. Additionally, by requiring that beneficiaries contribute a minimum of their own resources (in cash or in kind), the benefits from their choices become very evident. Efforts were been made to ensure that in the selection and design processes for subproject selection, least-cost consideration would also be more explicitly taken into account Ex post analysis shows that the expected cost efficiency and effectiveness was achieved under NCDP. Overall efficiency of NCDP As a program, the NCDP was consistent with international and national measures of efficient program implementation: Overhead costs were contained to 18%. The PAD estimated that overhead (technical assistance and administrative costs) would account for 38% of NCDP costs. This is quite high, but the estimate was made without knowing how much additional financing JSIF would be able to attract to amortize its fixed staff and operational costs. The final overhead (all non-subproject costs) was about 17.5%, well within international norms for CDD and social fund operations. Additional community counterpart contributions were crowded into the NCDP financing plan. Previously, JSIF operated with a 5% community contribution requirement. NCDP targeted 10% and achieved 12% overall. Using community- based contracting boosted community contributions to 25% on this part of the NCDP portfolio. Cost Efficiency of NCDP Subprojects The NCDP portfolio, in part due to Hurricane Ivan reconstruction needs, was largely focused on education subprojects (76% of NCDP funding), roads (17%) and health facilities (5%). Looking at the economic efficiency of these community investments reveals: 26 In terms of cost efficiency and control of unit costs, JSIF has continued to lead in cost constraint among the Jamaican public sector. A national unit cost data base was updated regularly and helped constrain costs. The Summative Evaluation found that "JSIF rates for construction work are considered `below the market rate'. This is partly due to a conservative cost data base and lack of recent updates. However, the lower rates do not appear to translate into a lack of interest to bid for JSIF projects. This might be related to the geographic areas in which JSIF projects tend to be: remote rural or volatile inner city which are apt to attract small scale and rather local contractors for whom the construction boom has had less benefits." While comparators with unit costs from other agencies was limited, the ICR mission did find that on construction of a 3 classroom module, JSIF's costs were between 12-17% lower than construction of a similar classrooms by Jamaica's Culture, Sports and Education Fund. Cost variations during construction were also contained within industry norms. Typically, public sector investments place a 20% contingency on contracts. NCDP cost variations as a percentage of subproject costs averaged 17% on traditional, 15% on CBC and 21% on Hurricane Ivan subprojects. Community-based contracting appears to have encouraged additional cost savings. Over 30% of subprojects contracted using CBC methodologies generated significant savings that were directed to complementary community investments. Cost-effectiveness of NCDP Subprojects Cost-benefit analysis is not applicable to the education and health investments since benefits are difficult to quantify, but qualitative evidence points to effective investments. The Summative evaluation found that, in the education sector, NCDP investments results in qualitative improvements to education including increased learning outcomes over comparators. In health, although household level health outcomes were not measured, the impact evaluation found an increase in access to and utilization of health services. Road sector investments were concentrated in rehabilitation and upgrading, which typically have high economic rates of return in Jamaica, and community members confirmed the economic impacts over comparators. Poor road maintenance across Jamaica had contributed to very high rates of accidents and fatalities and had lowered the economic efficiency of the road system overall. NCDP's road investments were in line with national priorities and higher-return investments. The Summative Evaluation confirmed an increase in economic impact from NCDP road investments, with community members perceiving significant improvements in access to markets and shops and safety and significant decreases in travel times and costs compared to comparator roads. The methodology of the Summative Evaluation did not allow for a strict quantification of the magnitude of these observed benefits. 27 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Responsibility/ Specialty Lending Supervision/ICR Eduardo Bayon Consultant LCSUW Jorge Cavero Consultant LCSPT Katherine M. Shafer Operational Coleman Consultant LCSHE support Leanne Farrell Junior Professional Associate LCSSA Daniel A. Hoornweg Sr Environmental Engr. LCSUW Mary Morrison Country Officer LCC3C Ahmadou Moustapha Lead Financial Management Ndiaye Spec LCSFM Emmanuel N. Njomo Consultant LCSFM Guido Paolucci Sr Procurement Spec. LCSPT Norma M. Rodriguez Procurement Analyst LCSPT Heinrich K. Unger Consultant EAPCO Evelyn Villatoro Sr Procurement Spec. LCSPT (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands No. of staff weeks (including travel and consultant costs) Lending FY02 32 193.74 FY03 9 36.59 FY04 0.48 FY05 0.00 FY06 0.00 FY07 0.00 FY08 0.00 Total: 41 230.81 Supervision/ICR 28 FY02 0.00 FY03 7 42.77 FY04 19 131.12 FY05 25 127.61 FY06 23 115.67 FY07 18 102.71 FY08 6 18.13 Total: 98 538.01 29 Annex 5. Beneficiary Survey Results (if any) An Overview of the Summative Evaluation Methodology Secondary Data Review (Fund Operational Manager + SFMS) Performance Process Tracing Assessment Beneficiary Assessment Summative (Household Survey + Community- Evaluation Impact Based Research) Assessment Secondary Data Review (Fund Manager + Surveys) Infrastructure Evaluation Site visits Review of technical documents The methodology for the Summative Evaluation comprised three elements, each with its own set of research instruments: 1. An Operational Performance Assessment which evaluated the efficiency and effectiveness of JSIF operational management of the NCDP; The team conducted a Secondary Data Review by analysing the JSIF's "Fund Manager" and Social Fund Management System (SFMS) databases in order to elicit quantitative data. The team additionally employed a Process Tracing instrument to elicit qualitative information on a selected sub-sample of 6 sub-projects. 2. An Impact Assessment which evaluated the impact of NCDP sub-projects, including the perspectives of sub-project beneficiaries, other sub-project stakeholders and community residents. The Impact Assessment combined methods, generating both quantitative and qualitative data, to evaluate the impact of NCDP subprojects based on 2 research instruments: a household survey and a community score card administered through focus group and key stakeholder interviews. For the household survey, data was collected from 700 households in intervention communities and in comparison ("control") communities. A group-based Community Score Card (CSC) instrument, along with semi-structured interviews with secondary stakeholders (such as health providers and teachers) and with key informants (able to provide an objective assessment of processes and impacts). 60 CSC focus group interviews were conducted and 102 key informant interviews were held. 30 3. An Infrastructure Evaluation examined the quality and sustainability of physical infrastructure. The Infrastructure Evaluation was conducted in a subset of the sampled communities receiving NCDP health, education and road infrastructure interventions and in comparison ("control") communities, although sample sizes make these results indicative. The control communities were selected using a nearest neighbour approach. In some cases the control facility had either had rehabilitation work or had been constructed within a similar period to the intervention facility. Sample weighting Sub projects were selected using a simple random stratified sampling system that stratified the "universe" of NCDP sub projects by type and by contracting modality, weighting their significance by (financial) project disbursement before using probability- based method to select sub projects for evaluation. The evaluation encompassed education, health and road sector subprojects as they accounted for 97 percent of NCDP's value of subprojects. The weighting of the NCDP portfolio according to the spread of (financial) disbursement means that disbursement becomes the key indicator for NCDP "effort". Based on a list of 189 NCDP sub projects provided by JSIF on Jan 23rd 2008, which were completed at least 12 months prior to the time of evaluation (i.e. by February 2007), the sample frame included 20 NCDP sub-project recipients and 8 controls concentrated on education, roads and health projects. Sampling frame (number of respondents) for Household Survey CBC traditional Ivan total Health 50 50 0 100 Road 50 50 0 100 School 100 100 100 300 Comparison ("control") 75 100 25 200 communities Total 275 300 125 700 treatment 500 comparison 200 Sampling frame for NCDP infrastructure evaluation CBC CBC Traditional Traditional Ivan Total "control" "control" Health 1 1 1 1 n/a 4 Roads 1 1 1 1 n/a 4 Schools 1 1 1 1 1 5 Total 3 3 3 3 1 13 Findings from the Summative Evaluation are presented throughout the ICR text. The ICR team reviewed in-depth the Summative Evaluation and found significant differences between the textual write-up and the empirical evidence presented in many cases. Where there are differences, the ICR takes the empirical data as evidence of main findings. 31 Annex 6. Stakeholder Workshop Report and Results (if any) 32 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Jamaica Social Investment Fund (Loan no. 7148-JM) National Community Development Project (NCDP ­ 1) Implementation Completion Report Borrower's Assessment Introduction The following comments serve as an assessment of the Government of Jamaica's achievements regarding the project design, implementation and operational experience in the execution of IBRD loan agreement no. 7148 ­ JM. These comments will reflect on the lessons learned throughout the implementation of the project. 1. Assessments Project Objective, Design, Implementation and Operation Experience 1.1 The primary objective of the NCDP was to assist the Government of Jamaica, through a fifteen million dollar (US $15,000,000) loan, in offering a continued effort in promoting greater social and community development through assistance to the poorest of the poor through the provision of basic services and temporary employment opportunities. 1.2 The Project Appraisal Document (PAD) for the NCDP made several critical assumptions for the conditions of implementation, many which have not held such as; economic stability; no major socio / political disruptions during project implementation; SDC and other entities will adopt improved community development criteria. Although there were critical movements in the political environment throughout the period of the Loan, the vicissitudes of the political climate had little to no impact on the day-to-day processing and implementation of subprojects within the portfolio. On the contrary, the level of improvement in community development criteria was not as anticipated, due, in part to lack of sustained support by the SDC. The relationship with SDC thus proved unsatisfactory, particularly as it pertained to the expected number of subprojects that were to benefit from JSIF financing after being identified by SDC in Community Development Plans. For instance, only 8% of the total NCDP portfolio was eventually financed by JSIF. In contrast, a large percentage (47%) of requests received with CDPs, had land issues outstanding. These included incomplete or no land documents to support the project applications submitted by 33 SDC. Other requests received were deemed invalid for not meeting other JSIF funding criteria. 1.3 One major assumption, which affected the delivery of subproject interventions in the manner prescribed by the appraisal documents, was that of Hurricane Ivan which occurred in 2005. The advent of this natural disaster resulted in the government reallocation approximately 30% of the loan portfolio to address critical areas of infrastructure repair especially in the area of education. Sub- Projects that were completed under the Hurricane response did not reflect the full intent of the overall project design as strategies related to the capacity building of community based organizations were not implemented due to the immediacy of the programme. 1.4 This impacted on the staff of the project as much of the projects other resources were deployed to ensuring the efficient delivery as part of the disaster response. This impact was not as detrimental to the project, however, as overall the project made inroads in capacity development in community based organization and there is evidence of a willingness to maintain the investments made on the sub-projects implemented as part of the hurricane intervention (see Evaluation Report) 1.5 Overcoming any obstacles within the five-year loan period the Government of Jamaica has effectively fulfilled most of the obligations to the loan under the NCDP with 98% of the portfolio achieving completion and creating approximately 3855 temporary employment opportunities (15,420 man months). 1.6 JSIF increased the number of projects financed by identification by Community Development Plans by 8%. 1.7 The NCDP varied from previous funding from the Bank as it included explicit provisions for the Community Based Contracting Methodology of which accounted for 18 % of the project portfolio, hence an increase of the number of communities which developed the capacity to select their own priority projects and manage project contracting and implementation effectively. 1.8 At least 62% of the subprojects financed through the loan, were targeted quartile 4 communities. An additional 25% were of quartile 3 ranking, with the remaining communities having been captured as `pockets' of poverty. Accordingly, the JSIF was able to ensure the inclusion and protection of the poor by improving their access to basic amenities within the individual beneficiary communities. Similarly, it can be said that a wide range of community priority needs were met based on the financing contribution made in as much as 14 different sub-project types. 1.9 JSIF was approached by additional public and private sector organizations (DIGICEL Foundation, Red Stripe, Rotary, CHASE, NCB) to manage the implementation of similar infrastructure projects that fell within the portfolio of the NCDP. These projects accounted for 6% of the NCDP portfolio, evidence of 34 increased complementary investment generated through JSIF assistance to community groups. 1.10 JSIF received `no objections' for all subprojects indicating continued support from the required line ministry and / or local authorities. Likewise an Inter- Institutional Agreement was signed between JSIF and the SDC on February 4, 2004, stipulating that the SDC be responsible for conducting maintenance training for NCDP sub-project communities. However, the last updated report from the SDC on the progress of communities trained and Maintenance committees formed by SDC regions was submitted in March 2007. Based on that report: - 17% (31) of groups with Maintenance plans and who completed Maintenance training; - 37% (69%) of groups are still to be trained and formulate Maintenance plans: 17% (32) of these already have Maintenance groups formed, though training had not initiated; the remaining 20% (37) of groups are scheduled to have their Maintenance Committees formed under a Consultancy funded by the World Bank's Emergency Relief Project scheduled for the period October 2008 to end of January 2009 (4 months). 2. Assessments Project Objective, Design, Implementation and Operation Experience - CBC 2.1 The JSIF was able to implement and complete approximately 96% of the CBC portfolio of subprojects within Loan validity period. This approach covered a range of sub-project types island-wide, such as basic and all age schools, roads, water supply systems, integrated community spaces and health centers. 2.2 Approximately 85% of subprojects were completed within budget, with the remaining 15% experiencing cost overruns. Cost variations were experienced due to a number of factors such as natural disasters; and the cement crisis prolonged implementation periods making the projects vulnerable to increased cost of materials and inflation;. 2.3 A particular benefit realized through the CBC methodology is the savings to costs on approximately 30% [16] of the subprojects financed. The majority of savings [19%] was gained by subprojects utilizing the CBC Model 1 where PMCs fully executed the construction works, in addition to having hired and managed the labourers for the project. Savings were realized under Model 2 and 3 also ranging from 8% and 5% respectively. 2.4 Due to the late development of a CBC portfolio of projects in addition to the time spent edifying itself on this methodology; the JSIF had certain challenges finding a fit for CBC projects under its regular modus operandi. As a result, the time 35 constraint met in having this component implemented, forced the JSIF to circumvent the project cycle to ensure that the CBC component could coincide with the time and disbursement deliverables committed to in the Loan Agreement. For instance the formulation of projects was moved to post board approval with an approval of the sub-projects being granted on a general cost estimate according to the sub-project type and scope of works to be done. While this approach proved an advantage in advancing the pace of implementing the subprojects, certain problems were realized through cost increases by understating the initial estimates in costs at the Board Approval level. 2.5 The quality of construction work delivered and management of funds under CBC proved comparative to traditionally implemented sub-projects. However, the value added to this approach is evident in the achievement of communities to implement this level of quality and management on their own. 2.6 CBC proved to be an effective community development tool considering the development of skills that were otherwise lacking in the communities prior to the intervention. Through considerable training, communities were empowered through their leadership groups, which were strengthened by becoming legal entities, thereby reducing the risk of sustaining their management of the completed facilities at post implementation. The training received in basic fiduciary management and procurement procedures helped to build the sustainable skills of the community. Other benefits cited by beneficiaries included the capacity building of PMC members (eg. in leadership, management skills, bookkeeping skills and communication skills). The added project experience has served to make them more self-reliant and capable of initiating other development projects on their own. The technical skills base of the community also grew based on the hands-on training received on site. Some labourers also seized the opportunity to become certified or upgrade their skills levels thereby adding value to the skills base at the community level. For example, approximately six (6) persons in twenty-one (21) projects have become masons as a result of the project. 2.7 CBC contracts were valuable given that they allowed community management to sub contract local skilled labour. This alleviated the problem of communities hiring contractors who failed to capitalize on community-based skilled labour, to avoid the risks for non-delivery or poor quality work. 2.8 The low input of the SDC in maintenance training of the NCDP beneficiary communities set the JSIF back in meeting its commitment within the stated Project period under NCDP. Resources had to be spent on building the capacity of SDC to administer the training. However, their lack of staffing proved to be a shortfall in meeting the target of Maintenance Committees to be formed and trained. The JSIF was therefore not able to meet the output targets of project appraisal. 36 2.9 As much as 27% of CBC subprojects were able to attract complementary investments as a result of the assistance received from JSIF. 10% were able to generate income, as well as attract additional funding assistance; 12% were able to strengthen their technical and management capacity by partnering with service clubs to implement their JSIF funded project; and 6% were able to benefit from equipping and fencing of their subprojects by donations from the private sector. 3. Lessons Learnt ­ CBC 3.1 The Project commenced with a sensitization session held with the NGO community on the CBC methodology and the JSIF criteria for funding under this model. The attraction to the NGO community was a means of qualifying the need to have legal entities implement the sub-projects with a sustainable factor. The proposals submitted by the majority of NGOs however ranged from being too ambitious, not meeting funding criteria or were not technically feasible. The shift was subsequently made to source sub-project applications from the pool of demand-driven applications already registered at JSIF. Although the screening of these applications to assess their appropriateness for CBC took time, further work had to be done at the community level to sensitize and assess the capacity of CBOs to manage a CBC-type project. A CBC Capacity Assessment tool for CBOs was thus designed and an allowance of time had to be advanced to the CBOs giving them a chance to collectively evaluate their own ability and willingness to use this community development approach. Consequently the `teething' phase of CBC saw some projects initially being appraised for CBC, but later being converted to traditional type sub-projects because of the emerging inability of some CBOs and communities to manage the demands of this approach. 3.2 The governance of the CBOs was realized through collaboration with the Department of Cooperatives and Friendly Societies. CBOs were trained, assessed and became legally registered as Benevolent or Cooperative Societies given the nature of their sub-project type. This partnership not only allowed CBOs to benefit from a CBC type sub-project, but it also served as means of mitigating the risk to sustainability of these subprojects and by extension, to the funding investment made through this Project, as the CBOs have a long-term accountability to the governing authority of the DCFS. 3.3 The cost ceiling for CBC projects at USD 100,000 was reasonable for subproject allocation as this reduced the responsibility of the PMCs to manage complex project types. For instance, some cost variations, which emerged during implementation stemmed from the need to widen the scope of works, as the initial estimate of works to be done could have been understated to ensure that simple solutions were implemented. Hence, the increased scope of works meant that projects were technically compounded, usually beyond the ability of Project 37 Management Committees to supervise the works and manage the expenses incurred. 3.4 Implementing a CBC project required a multifaceted training approach with considerable input of time and resources from the JSIF staff. For this reason, a team of staff members exclusively assigned to CBC projects became necessary to ensure that beneficiary communities received continuous training and technical assistance throughout the development of their respective subprojects. The failure and difficulties encountered with some projects particularly in the earlier phase of the Project can also be attributed to the lack of hands-on monitoring applied by JSIF. For example, most PMCs neither had the technical expertise to supervise construction activities nor the know-how to properly manage a Contractor or Foreman. Although a Terms of Reference was developed to assist the community in understanding the difference in roles of the PMC and foreman, this did not make up for the lack of technical supervision of the project. The hiring of Site Supervisors apart from technical officers at JSIF allowed technical staff to focus on the administrative issues for the project, while site supervisors were able to provide hands-on technical field assistance to PMCs. The placement of a CBC Officer ensured that the training needs of PMCs were continually re-assessed throughout the implementation of the project and that the training gaps were filled where necessary. This role also served as a control factor to ensure that CBC procedures were being adhered to and were properly administered by the PMCs. 3.5 The manipulation of the project cycle to accommodate CBC projects proved to be beneficial in reducing project implementation timelines. However, the organizations ran the risk of practicing due diligence as certain adjustments to the institutional arrangements and procedures were not made to adequately support CBC. Although these support systems were developed during the course of the Project, more preparation and planning at the design stage of the Project should have been conducted in an effort to examine the implications of this methodology against the standard institutional composition of JSIF. 3.6 The lack of adequate technical expertise and resources within a community proved to thwart the ability of PMCs to proficiently and independently execute their project. Moreover, given the complex technical nature of the subprojects, the introduction of external supervisors hired by JSIF to support the PMCs became crucial as a measure of control for JSIF to facilitate the achievement of the technical outcomes of the sub-projects. Communities were also encouraged to partner with service groups operating in their region such the Rotary Club, where the technical expertise and resources usually available to these groups, could be used to assist the PMCs in effectively executing their project with limited reliance on the JSIF. This approach also helped to alleviate project management costs for JSIF hiring external supervisors to monitor the work of PMCs and Contractors on the ground. 38 3.7 The defects liability period was reduced for CBC projects from the traditional 6 months to 3 months during the course of the Project. This proved essential as PMCs, having the more hands-on reach in executing the works and controlling the funds, had the opportunity to address defects in a shorter space of time, based on the immediate resources available to them in the community to correct problems as they arose. Most PMCs were able to correct the problems soon after the defects list was prepared by the JSIF Site Supervisor, thereby curtailing the extensive time before final completion could be granted. 3.8 The failed partnership with SDC in the delivery of maintenance training has forced the JSIF to take steps to strengthen the maintenance component of the Loan, in addition to the quality of training of this kind. The JSIF will have more control over the quality of training delivered to its beneficiary communities by being solely responsible for ensuring the training of its sub-project communities in addition to improving the content of the training material to make the intervention more sustainable and have community facilities more resistant natural disasters. 4. Evaluation of Borrower's own performance and lessons learned (general) 4.1 Though JSIF was unable to secure it's footing through many attempts during the Implementation of the project on the Maintenance Training for the Portfolio, a Maintenance Programme is now being undertaken through funding from the ERL ­ Hurricane Dean for the NCDP project Portfolio. 4.2 JSIF being cognizant of the fact that Maintenance is of particular importance institutionalized a Technical Review Committee to ensure that the quality of work being produced from consultants was up to standards. This Committee was instrumental in ensuring that designs were revised to use maintenance friendly designs and materials as much as possible. 4.3 JSIF should have taken a harder line in terminating non­performing contractors over the implementation period as in affording contractors additional time caused unnecessary delays in the implementation of the project, which was then compounded with a general lag in timeline deliverables due to the Cement Shortage experienced in the construction industry in 2006 when implementation was at its peak. 4.4 Though the Technical Review Committee aided in identifying possible variations to the project due to incomplete formulations and was able to mitigate against these practices by having consultants present the project, JSIF would have benefited from adhering to industry practices and building in a contingency within the project cost so as to reduce additional time delays during implementation as the approval process for variations under 15% were tedious. 39 4.5 Of utmost importance to the operational practices of JSIF was ensuring a strong commitment to Environmental Assessment and Monitoring. An environmental evaluation of projects in implementation was carried out at the end of year two of the loan and it found that " in general the projects all appear to be well executed with apparent active stakeholder participation and consequently a reasonably strong commitment to ensuring sustainability". 4.6 Geared towards the satisfaction of the Community, Project Information Meetings (PIMs) were revamped during the early stages of the NCDP and used not only to inform the wider community of the upcoming project, but also to engage the Contractor within the community as well as the External Supervisor. It was a useful forum to ensure that the community solicited amongst themselves an Employment Coordinator and understood that they owned this project. 4.7 Subproject implementation timelines improved due to the addition of site supervisors hired to monitor supervision projects not withstanding impeding factors such as hurricanes, tropical storms, as well as an industry wide affected cement shortage and general contractor liquidity. 5. Evaluation of the performance of the Bank and lessons learned 5.1 The WB and JSIF worked together well as a team in the fiduciary management of the project and managed to build a very good working relationship while exchanging feedback on ways to improve effectiveness. Although there were changes in the bank staff during the period of the loan this did not impact negatively on the quality of support offered to us. For example, the processing of Statements of expenditure which were sent to the bank to effect disbursements, were for the most part done in a timely manner resulting in the project maintaining adequate cash flow for its operations. 6. Indicating its proposed arrangements for future project operation, and defining with the Bank the performance indicators to be used to monitor future operation and development impact 6.1 In assessing the overall achievements of the NCDP, the Government of Jamaica notes the contribution of the project in developing the local infrastructure and enhancing the capacity of communities to manage and contribute to their overall development programme. It is noted also that the project ably assisted in the achievement of the Government's reconstruction efforts after the devastation of Hurricane Ivan and the negative impact of that disaster on the Education infrastructure, which is essential for the sustained development of the Country. 6.2 The Government is also of the opinion that the investment of the project created value in poor, underserved communities across the island, addressing the 40 immediate needs of the communities with new and effective approaches to community based involvement and participation. 6.3 Given this assessment, and the success of participatory approaches to small development projects; the use of similar strategies such as those employed in the NCDP should be examined for other types of investment that have greater impacts at the local levels. Such investments should include areas related to community based tourism and agriculture where there is a need for high levels of consensus building and social capital development. 6.4 Critical capacity assessment of all institutions involved in the implementation of the project should be undertaken at the appraisal stage of the project. This assessment should not only look at the fiduciary obligations of the institutions, but should also assess the synergies and the expectations for delivery against their current capacity. These assessments should also provide a `road map' for capacity development and posit strategies for creating effective partnerships. 6.5 Based on the internal assessment of the CBC interventions, there is a link between communities with high levels of social capital stock and the relative success of the project interventions. Communities with low levels of social capital and high levels of fragmentation have less benefits being accrued as a cause of any intervention. It is therefore recommended that if CBC is to be used in communities that are assessed as highly fragmented; a social intervention action plan be implemented to address these issues before the investment is made. This action will improve the success rate of the project in that there will be higher levels of participation in the investment and greater levels of commitment to the goals and objectives of the project. 6.6 Further examination should be made of the use technology for infrastructure development where more model solutions are provided for the needs of communities. This should result in reduced response times for investments, which were a major concern in communities as outlined in the beneficiary assessment. 7. Indicators 7.1 There should be a clear indication of the baselines that are being used for the assessment of the project. The NCDP had no baseline data to which there could be suitable comparisons made regarding the success of the project. It is recommended that future demand driven projects establish baseline data for critical success indicators, in order to determine the effectiveness of project outcomes. 41 7.2 For demand driven subproject intervention, indicators that relate to the implementation of comprehensive social programmes should not be used to assess sub-project success. An example of this is the use of crime statistics and economic development indicators. These should be abandoned for indicators that relate to social cohesion and improvements in access to basic services. Higher- level impact indicators should therefore relate to the perceptions of community and that of social capital development at the community and national levels. 7.3 It is recommended that more indicators that assess the development of Civil Society and the ability of communities to advocate for improvements in their area be incorporated in projects that have a high CDD involvement. For future projects interventions, the ability of community based organizations to maintain the investments will be impacted by their ability to raise capital, however, improvements in advocacy skills, will enhance their link with key state agencies and should increase their ability to channel resources to their communities to address maintenance needs. This investment in advocacy, it is believed, should enhance the communities' ability to sustain the investments over longer periods. 7.4 Indicators that test the validity, efficiency and effectiveness of key partnerships in project implementation should be incorporated as a part of the lower level output indicators of the project. The need for constant review and revisions to existing relationships is important to project success, as there are constant changes in the project environment. As such the project management should have a mechanism to measure the standards of performance of key stakeholders so as to make the necessary management adjustments using the best evidenced based practices that may be provided. 42 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Comments on ICR submitted by Dr. Edwin Gutierrez, Senior Economist, Operations Management Dept., The OPEC Fund for International Development (1) Implementation capacity. It is fair to say that JSIF has demonstrated it has capacity to implement community development projects. However, in view of the range of shortcomings faced by JSIF during the implementation of the NCDP, it is clear that this capacity needs to be supported by technical assistance from major donor institutions. (2) Lessons learned. The choice of JSIF as the primary implementing entity for the project allows lessons learned from community-based projects to be incorporated in subsequent operations. (3) Participatory approach to projects. The experience with NCDP has reinforced the success of the participatory approach to small development projects. Indeed, the sub- projects under NDCP were requested by the communities, were primarily executed by CBOs, NGOs and small-medium private contractors using locally available inputs and simple construction technology. The completed projects, in turn, provided the communities with better services including health, sanitation, education and basic economic (rehabilitation of feeder roads) and production (i.e., rehabilitation of agro- processing facilities) infrastructure. (4)Participatory planning mechanisms. NCDP has demonstrated the importance of participatory planning mechanisms. Indeed, a particularly important shortcoming in NCDP was the relative absence of such mechanisms at the beginning of the project in order to enhance community ownership and harness, empower and guide political/social energy, all in order to better accomplish the objectives of NCDP. (5) Maintenance and operations. While it is important to achieve the physical outputs under NCDP, it seems to be necessary to emphasize the post-construction stage. That is, the issue of maintenance planning and execution should has been given a greater profile within the subprojects implemented under NCDP, in order to support operational sustainability in the longer term, after the project has finished. (6) Financial execution. Overall, the different auditing reports of JSIF and of NCDP financial statements, showed that JSIF has a suitable organizational structure and follows appropriate administrative and financial practices. However, there is room to improve some of JSIF's operational procedures and practices. For instance, i) management should take the necessary steps to ensure that all project variation orders are reviewed and approved on a timely basis and that there is evidence of approval, ii) management should ensure that the Annex Report use to monitor projects undertaken by JSIF is updated on a timely manner by responsible personnel, iii) a fixed assets register should be maintained showing important information of each asset, and assets should be physically verified periodically and their condition verified. Adjustments should be made in the register to eliminate assets no longer in existence or those that can no longer be used. 43 Annex 9. List of Supporting Documents National Community Development Program (NCDP) Project Appraisal Document JSIF Land Acquisition and Resettlement Policy Framework JSIF Operations Manual Revised March 2006 JSIF Operations Manual (Annexes) Revised March 2006 JSIF Funding Guide JSIF Environmental Management Framework Inner City Basic Project (ICBSP) Project Appraisal Document "Phase One Report: Environmental Screening and Supervision of 37 JSIF Projects Under Implementation" Prepared for the Jamaica Social Investment Fund, TEMN Technological and Environmental Management Network, Ltd. January 2005. "Summative Evaluation of the JSIF National Community Development Project ­ Final Report", Oxford Policy Management, UK, June 2008. Internal World Bank Documents: All NCDP PSR/ISRs and supervision mission Aide Memoires NCDP Financial Management Supervision Report June 27-28, 2005 NCDP Aide Memoire, Mid-Term Review, January 2006 44 45