ICRR 13304 Report Number : ICRR13304 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 01/28/2011 PROJ ID : P069086 Appraisal Actual Project Name : Community Based US$M ): Project Costs (US$M): 96.36 81.45 Poverty Reduction Project Country : Nigeria Loan/ US$M): Loan /Credit (US$M): 60.0 85.0 Sector Board : SP Cofinancing (US$M): US$M ): 26.74 n/a Sector (s): Other social services (24%) Roads and highways (24%) General water sanitation and flood protection sector (23%) Primary education (23%) General public administration sector (6%) Theme (s): Participation and civic engagement (25% - P) Poverty strategy analysis and monitoring (25% - P) Other social protection and risk management (24% - P) Gender (13% - S) Improving labor markets (13% - S) L/C Number : C3447 Board Approval Date : 12/20/2000 Partners involved : AfDB Closing Date : 02/28/2006 03/31/2009 Evaluator : Panel Reviewer : Group Manager : Group : Judith Hahn Gaubatz Kris Hallberg IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: According to the Development Credit Agreement (DCA) and the Project Appraisal Document (PAD), the objectives of the project were (i) to improve access of the poor to social and economic infrastructure and (ii) to increase the availability and management of development resources at the community level . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): US$ 6.56 million; Actual : US$21 (1) Capacity Building (Appraisal : US$6 21 .62 million ). Activities included strengthening US$ 21. federal, state and local government agency (LGA) capacity for poverty reduction policy design, advocacy, and management of community-led infrastructure projects. (2) Community -Based Initiative s in Basic Social and Economic Infrastructure (Appraisal : US$78 78 .53 million; Actual : US$ 78. US$59 US$ 59. 59 .83 million ) ). Activities included the establishment of independent, state -level social funds in eight states (out of twelve included in the overall project in two phases; the remaining four states were to be funded by a co -financier) to provide funding for community-level infrastructure projects, as well as support for community -led identification of needs, preparation of microproject proposals, and monitoring and evaluation . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The DCA was amended in May 2005 to allow two additional states to receive project funding . It was amended a second time in August 2007 to change the coordinating unit from the National Planning Council (NPC) to the Federal Ministry of Finance (FMOF). A supplemental credit of US$25 million, requested by the borrower, was approved in February 2007, The ICR reports that the additional credit was requested to cover a financing gap for microprojects that had already been initiated but could not be completed without more funds . The TTL reports that the project became so popular that many communities were applying for grants, and so the government wanted to use the opportunity to concretize results of the community-driven approach and bridge the gap towards a nation -wide roll-out of the approach. However, the actual project cost came to about US$ 15 million less than estimated (including the supplemental credit), a difference most noted in the second component which came to about US$ 20 million less than estimated due to lower than expected cost per unit for the microprojects . This appears inconsistent with the justification for the supplemental credit. In contrast, the costs for the first component on capacity building increased by about US$ 15 million; according to the project team, this was due to significant underestimation of the capacity building costs at appraisal. The African Development Bank (AfDB) was to provide co-financing for the project; the ICR does not provide an actual figure for the contribution but the TTL reports that the final amount came to about $ 3 million less than expected. The TTL reports that state- and local government-level counterpart funding was eventually waived upon agreement with the Bank after changes in the Country Financing Parameters, which allowed 100 percent financing in all categories. The closing date was first extended to August 2006 to allow completion of activities in the two added states . It was extended again to December 2008 due to the additional funds from the supplemental credit . It was extended for a third time to March 2009 to allow for completion of the project impact assessment activities . 3. Relevance of Objectives & Design: The relevance of the objectives is rated Substantial . The objectives are consistent with country and Bank strategies. No Country Assistance Strategy (CAS) had been prepared at the time of Appraisal . The project fit with the recommendations of the Interim Nigeria Country Strategy Note for the period FY 00 -05 to focus on poverty reduction and support community-based activities. The current Country Partnership Strategy (2010-2013) emphasizes "increasing access and utilization of human development services by the extreme poor and vulnerable ." However, the second objective was somewhat vague, particularly with regards to increased "management" of resources. The relevance of the design is rated Substantial . Numerous aspects of the project were well -considered, including: the identification of the poorest states based on geographical and poverty -score rankings; a self-targeted social fund approach in which proposals for microprojects were solicited from amongst the poor communities; risk factors, which were appropriately assessed as "High" and mitigation measures prepared, such as including states in the preparation and supervision process to build state -level commitment to community-level activities; and capacity building on multiple fronts, including at the federal level which was adopting a significantly different approach to poverty reduction than it had in the past . In addition, lessons learned from previous Bank projects in Nigeria and on social funds were taken into account in the project design . However, significant shortcomings were the lack of a clear results chain and, subsequently, a weak monitoring and evaluation framework . The indicators primarily measured outputs and process rather than outcomes - while these indicators were certainly relevant, they did not, for example, capture multiple dimensions of access to infrastructure that would be critical for achieving outcomes leading to poverty reduction, nor were indicators for the second objective clearly defined . 4. Achievement of Objectives (Efficacy): Objective 1: To improve access of the poor to social and economic infrastructure . Achievement is rated Substantial . The project achieved its targets related to the number of microprojects completed, and there is some evidence of the various dimensions of "improved access" (i.e. quality, affordability) reported in the ICR. However, the evidence is drawn from the relatively small sample sizes and there is no data provided on control groups. The ICR states that geographic targeting was used to select poor communties Outputs: The project funded nearly 3000 microprojects in eleven sectors, surpassing the target of 2200. 1119 in education, including 3689 classrooms that were built and/or rehabilitated. 925 in water supply and sanitation (i.e. provision of clean water) 303 in health, including 255 health centers that were built or rehabilitated . 277 in transport, including 243 km of new roads and 2,699 km of rehabilitated roads. 168 in rural electrification (i.e. connections to main grids) 51 in market infrastructure (i.e. construction of market stalls ). The ICR notes that the number of microprojects expected to be funded increased significantly (from 1200 to 2200) as the cost per project was much lower than originally estimated . Outcomes: The following outcomes are based on a Rapid Impact Assessment conducted in 2009, with the following samples of communities surveyed in each sector : Education: 75 communities (1119 microprojects completed). Water supply and sanitation : 31 communities (925 microprojects completed). Health: 10 health care facilities (303 microprojects completed). Transport: 22 communities (277 microprojects completed). Increased access: 3085 communities (10.3 million residents) now have access to infrastructure, compared to the target of 2000 communities. Given the poor state of and/or limited access to infrastructure prior to the project, it is presumed that the baseline figure for access is close to zero . (Note: Although an increase in "the number of poor with access" is a key project indicator, evidence is primarily presented in terms of the number of communities with access, which is then extrapolated to the total number of persons benefiting directly and indirectly from the infrastructure microprojects.Geographic targeting was used to select poor communities ) Primary school enrollment increased 60% (from 18,124 to 29.575 students), compared to a target of 40%. However, there is no information provided on the proportion of eligible children enrolled before vs . after the project to assess level of coverage . Patient attendance at health care centers increased five -fold (from 112 to 746 patients). Households with access to rural water supply increased from 8.1% to 87.8%. Households with access to solid waste disposal increased from 28% to 74%. Equitable access, by gender : The proportion of total beneficiaries who are women is 70%, achieving the target figure provided in the ICR of 70%. According to the TTL, this figure was an estimate of the number of women utilizing the infrastructure, based on the rapid assessment . Primary school enrollment for girls increased at the same rate - 61% - as overall enrollment (from 4,780 to 7,685 girls). Female patient attendance at health care centers increased four -fold (from 66 to 255 patients), compared to five-fold overall. Greater ease of access Average commuting time decreased from 75 minutes to 24 minutes. Note: This figure is based on data from four communities within the 22 communities sampled that implemented transport projects . Average time spent collecting water per day decreased from 95.7 minutes to 16.4 minutes. Improved quality of services Total number of deaths from disease decreased from 256 to 65. Total number of qualified personnel at health care centers increased from 54 to 126. Increased affordability Average cost of treatment at health care centers decreased from 4,924 naira to 1,916 naira. Average sum of money spent on water decreased from 99 naira to 17 naira. Average cost of transport decreased from 126 naira to 76 naira. There was no information provided on other factors that may have contributed to the improvements . Objective 2: To increase availability and management of development resources at the community level . Achievement of this objective is rated Modest . Although the target for the "amount of development funds controlled at the community level" was met, the definition of "increasing management of development resources " is unclear and thus difficult to evaluate properly . Outputs: $59.83 million was disbursed for infrastructure microprojects . More than 3000 communities participated in the process of identifying development needs, preparing proposals for funding, and managing projects . 208 workshops were organized in the areas of management, planning, monitoring and evaluation, with 2,697 community-based organizations participated in the training . Outcomes: $59.83 million was made directly available to poor communities for development projects, achieving the (revised) target of $60 million. The ICR reports that local communities demonstrated increased capacity by "being able to conceive of communal goals, prioritize goals in order of importance through participatory methods, form groups, elect leaders into positions, understand the difference between leadership for action and ceremonial leadership, and integrate women into leadership." The project team also reported that the number of formal and informal groups getting involved in community microprojects increased and that the number of microproject group leaders being elected into formal political elective positions has also increased . 5. Efficiency (not applicable to DPLs): Efficiency is rated Substantial . The ICR reports that findings from the Background Study for the Mid-Term Review of the Community-Based Poverty Reduction Project (2004) demonstrated that the microprojects were about 26 percent more cost effective than similar alternative government agency projects . Annex 3 provides data from the comparative cost analysis, in which microprojects - such as rural feeder roads, electricity generators, boreholes, hand-dug wells, school classrooms, open market stalls, and health care centers - range from 2.73% to 38.9% more cost-effective. The ICR also reports increased efficiency in project management at community and LGA levels; however, there is no evidence provided to substantiate this statement . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project outcome is rated Satisfactory . Development objectives and design were substantially relevant, the objective of improving access to social and economic infrastructure was substantially achieved, the target for amount of funds controlled at the community level was met, but evidence on increased management at community level was only limited. Efficiency was substantial. a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: Although almost all of the microprojects were still being maintained for at least one year after completion, there is some risk to longer-term maintenance. As part of the project design, LGAs were to play a critical role in this aspect of sustainability, including signing agreements with the communities to partner in current cost and maintenance . However, the ICR notes that some LGAs did not fulfill these obligations . A specialized training and awareness program for the maintenance of community assets was established and a large number of Community Project Maintenance Committees instituted maintenance and sustainability strategies . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at entry is rated Moderately Satisfactory . As discussed in the section on relevance, the project design included numerous elements that adequately addressed implementation issues such as capacity and risk management. However, the results chain was not clear, as reflected in the inadequate M&E design, including the narrow choice of indicators . In addition, according to the project team, the costs for the component on capacity building were significantly underestimated at appraisal. Supervision is rated Satisfactory . The Bank team worked regularly and consistently with the counterpart staff in resolving implementation issues, helped in part by transferring task team leadership to the field . The ICR notes the beneficial results from overall continuity of the project team, including consistency in relationships and knowledge of the project throughout the project period, although the ICR also cites the Borrower's concerns that the increased implementation support in the field came at expense of more frequent supervision missions from WB headquarters. The project team reported that the Bank team worked to redefine the results chain and improve the M&E system (including being able to better measure outputs ). at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government performance is rated Moderately Satisfactory . At the federal level, the government demonstrated commitment to a community-driven poverty reduction approach and recruited highly qualified staff, whose salaries were competitive with the private sector . However, LGA performance was less than satisfactory . The ICR reports that there was overall a weak link between the LGAs and the microproject proposal process . The TTL reports that although most LGAs signed agreements to provide recurrent and maintenance costs, many of them did not actually provide the financing and thus communities were left to find other ways to sustain the infrastructure projects. In addition, counterpart funding was not provided in a timely manner during the first two years and was eventually waived . Implementing Agency performance is rated Satisfactory . The project coordinating unit was moved from the NPC to the FMOF in the fourth year of the project in an effort to facilitate implementation . The ICR reports that there was a conflict between NPC's mandate to provide policy guidance on government activities and its implementing role for the project, thus contributing to bureaucratic delays . Fiduciary matters were adequately carried out, despite some initial problems due to lack of experience . At the state level, the ICR reports that the autonomy of individual state agencies facilitated implementation of microprojects as there was limited bureaucratic interference. The qualified and committed staff also ensured effective outreach to communities and supervision of microprojects. a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design : As noted previously, indicators were narrowly chosen and did not adequately reflect the broader aspects of improved access to services and community -managed development. The ICR notes a number of weaknesses in the M&E design such as lack of clarity in the responsibilities of agencies for collecting data, lack of baseline data, and initial lack of harmonized reporting templates, making it difficult for the Federal team to summarize indicators across all states . M&E Implementation : The M&E system was revised during the last 2-3 years of the project to collect more standardized data on a regular basis, and one impact evaluation, albeit a rapid assessment, was carried out . However, the indicators tracked were still largely focused on process and outputs, rather than outcomes . M&E Utilization : The ICR reports that the impact evaluation report was published and widely disseminated, although there is no information on how the information was utilized . The ICR states that data collected from semi -annual and annual reports of indicators were evaluated and used for decision -making in the areas of safeguards assessments, financial management reviews, operational choices, planning and resource reallocation decisions, and that "M&E activities helped communities to focus on maintenance and usage "; however, there is no further information to substantiate these claims. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards compliance was satisfactory . Environmental and social management plans were implemented, including an environmental audit which rated compliance as satisfactory . The ICR reports that there were no significant deviations from Bank fiduciary policies and procedures . The Community-Driven Development (CDD) approach, which had been met with skepticism at the outset of the project, gained wider acceptance due to the demonstration effect of the project . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Moderately M&E framework was inadequate, Satisfactory including the lack of a clear results chain. Borrower Performance : Satisfactory Moderately LGA performance was unsatisfactory, Satisfactory namely in the area of providing support to sustain infrastructure projects . Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Legal independence and operational autonomy of state agencies can greatly facilitate implementation of local-level activities, as bureaucratic obstacles are minimized . Recruitment of qualified staff can be a key factor in ensuring timely and effective implementation . 14. Assessment Recommended? Yes No Why? This is one of the first Bank projects in Nigeria to incorporate a community -driven approach to poverty reduction. In addition, given the multi-tiered system of government in Nigeria, it would be of interest to learn more about the roles played by each level - starting from the federal level down to the individual community level . 15. Comments on Quality of ICR: The ICR was overall consistent with ICR guidelines . Although the ICR reports numerous data related to the achievement of objectives, the evidence is not always presented in a clear and logical results chain . Project component costs reported on pp . 2-3 of the ICR do not match the figures reported in Annex 1. a.Quality of ICR Rating : Satisfactory