ICRR 12994 Report Number : ICRR12994 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/29/2008 PROJ ID : P074361 Appraisal Actual Project Name : Social Investment US$M ): Project Costs (US$M): 30.20 34.50 Fund 2 Project (sif 2) Country : Georgia Loan/ Loan /Credit (US$M ): US$M): 15.00 16.20 Sector Board : SP US$M ): Cofinancing (US$M): 3.00 0.00 Sector (s): Other social services (25%) General energy sector (20%) General education sector (20%) General water sanitation and flood protection sector (20%) Health (15%) Theme (s): Social safety nets (20% - P) Participation and civic engagement (20% - P) Rural services and infrastructure (20% - P) Decentralization (20% - P) Improving labor markets (20% - P) L/C Number : C3758 Board Approval Date : 05/15/2003 Partners involved : USAID Closing Date : 09/30/2007 12/31/2007 Evaluator : Panel Reviewer : Group Manager : Group : Kris Hallberg John R. Heath Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: According to the PAD, the Georgia Second Social Investment Fund Project (SIF II) had the following development objectives: Participating local self-governments (LSGs) use enhanced management skills and a participatory process in establishing local development plans and setting investment priorities . Participating poor communities apply for, manage, and use micro -projects in a sustainable manner . Participating poor communities benefit from improved social and economic infrastructure and related services . The Credit Agreement has a different wording of the PDO : to assist communities to (a) organize themselves to solve common problems; (b) improve management, administration, and introduce participatory processes in LSGs; and (c) improve the access of communities to social and economic services on a sustainable basis . The PAD statement of objectives is used in this ICR Review because it is more specific . The ICR also uses the PAD statement of objectives. While the project was national in scope, the primary target group comprised the poor communities in peri -urban areas, rural towns, and villages . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): Capacity Building (estimated cost at appraisal $ 3.8 million, or 12.6 percent of total cost; actual cost $ 1.7 million, or 4.9 percent of total cost): training and technical assistance linked to micro -project development and implementation, delivered to communities and LSGs. There were two sub-components: (i) community development planning and related micro-project development, and advocacy; and (ii) institutional strengthening of LSGs in line with their evolving responsibilities for local utility services, taxation, budget planning, and local management . Community -Based Micro -Projects (estimated cost at appraisal $ 22.2 million, or 73.5 percent of total cost; actual cost $29.7 million, or 86.0 percent of total cost): grants for community-level economic and social infrastructure micro-projects located in poor peri-urban areas and rural towns and villages . The component also financed consultant services for micro -project design and supervision . Institutional Support to GSIF (estimated cost at appraisal $ 3.5 million, or 11.4 percent of total costs; actual cost $ 3.1 million, or 9.1 percent of total cost): office equipment, vehicles, consultants' fees, and operating costs, as well as training and technical assistance, and monitoring and evaluation . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: SIF II was approved in mid-2003 and closed in December 2007, three months later than the original closing date . The closing date of the previous project, SIF I, had been extended from mid -2002 to September 2007 in order to finance school reconstruction following an earthquake; these activities were financed by a Supplemental Credit . Although there was considerable overlap between the two projects -- more than four years -- most of the activities originally designed for SIF I had been completed when SIF II began . In November 2004, nearly one year into project implementation, the Government decided to pay the micro -project contribution on behalf of poor communities as well as assume the responsibility for the operations and maintenance of micro-projects. USAID's expected co-financing of $3.0 million did not materialize due to a change in USAID's assistance strategy for Georgia. KfW provided parallel financing of $ 10.17 million, significantly more than the appraisal estimate of $ 4.7 million. Total disbursements in US$ were greater than the original credit amount because of changes in the exchange rate with the SDR. 3. Relevance of Objectives & Design: The objectives of SIF II were substantially relevant. They were consistent with the 2003 CAS, which highlighted the need to focus on community development through rehabilitation of small -scale economic and social infrastructure in poor rural and peri-urban communities. The objectives also were consistent with the lessons learned from SIF I, which showed that more work needed to be done to build the organizational capacities of communities and facilitate interactions between community members and local government authorities . Project objectives became less relevant during implementation due to shifts in the Government's decentralization agenda . Initially, the Government's decentralization strategy was to decentralize economic management to LSGs and promote community -driven development through participatory processes . However, this strategy changed during the life of SIF II, as discussed in Section 9 below. According to the ICR, project design complemented other Bank -financed projects in water supply, rural infrastructure, health, education, and social protection -- in particular, the Municipal Development and Decentralization Project which focused on improving the performance of local governments in larger cities . However, there were significant gaps in the project's M&E system, which did not include adequate indicators to measure improvements in inclusiveness, management skills, institutional performance, and service delivery benefits /outcomes. 4. Achievement of Objectives (Efficacy): Participating LSGs use enhanced management skills and a participatory process in establishing local development plans and setting investment priorities : substantially achieved . In terms of project outputs, 225 LSG officials participated in training programs and 225 development plans were prepared . LSG officials participated with community representatives and other stakeholders in setting priorities for micro -projects. The ICR provides insufficient evidence on improvements in management skills and participatory process . Participating poor communities apply for, manage, and use micro -projects in a sustainable manner : modestly achieved . A total of 108 community members and 117 Community Development Committee (CDC) members received training. 90 percent of those surveyed in the Beneficiary Assessment (BA) said that their community had been involved in the micro-projects in one way or another; 43 percent said that they personally participated in the project identification process . However, this evidence should be interpreted with caution : Annex 5 of the ICR suggests that the BA "appears to have been skewed to favor opinions of community leaders over those of normal residents and may not have been conducted in fully representative project sites ..." (ICR p. 34). There are also questions about the degree of community participation ("less than five percent of the community residents attended the initial meetings held to identify the micro -projects," ICR pg. 32). In addition, the ICR does not provide evidence on the utilization of micro-projects and their services . Sufficient evidence on whether community management and use were achieved "in a sustainable manner" is also lacking. These data deficiencies are a particular weakness in this project (compared with the first project) because community participation and management were among the main project objectives. Participating poor communities benefit from improved social and economic infrastructure and related services : substantially achieved . The project financed 225 infrastructure projects, compared to the target of 300. The shortfall was mainly due to an increase in construction costs during project implementation . Most of the micro-projects were in education (52 percent of the number of projects ) and water systems (24 percent). The BA indicated that respondents were satisfied with the projects : 82 percent said that the micro-projects had a positive impact on their community, 93 percent said that access to services had improved, and 84 percent believed that the quality of service had improved . However, as noted above, the BA sample may not have been representative of the population of project beneficiaries, and there is no quantitative evidence on the service delivery benefits /outcomes. 5. Efficiency (not applicable to DPLs): There is little information available to allow an assessment of project efficiency . No ERR was estimated at appraisal due to the demand-driven nature of the project. An ex-post cost-effectiveness analysis was discussed but not carried out, as the project team was of the view that there were no similar publicly -funded investments that could have been used as comparators . The ICR recognizes, however, that an ex -post cost-effectiveness analysis would have been useful. Sharp escalations in construction costs during project implementation almost certainly had a negative effect on project efficiency, but changes in procedures that shortened the duration of the micro -project cycle from 7 to 5 months probably had a positive effect . In the absence of sufficient information, project efficiency is rated modest. ERR)/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Project outcome is rated moderately satisfactory . There is insufficient evidence on various aspects of the project objectives (see Section 4 above) as also on project efficiency which is rated modest . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The institutional sustainability of the project is likely to be favorably affected by the Government's mainstreaming of the implementing agency, GSIF, into the Government structure . The sustainability of decentralized decision-making and management of community investments is, however, less assured . The ICR also notes "What is less clear if the sustainability of capacity building activities focusing on local self -government, because of the political uncertainties in this regard with how the new streamlined local self -government structure will work" (ICR pg. 20). a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at entry and quality of supervision were moderately unsatisfactory . Despite being a follow-on project, the weaknesses of the first project were not adequately addressed, especially regarding M&E . Both in design and supervision, the Bank failed to give sufficient attention to the measurement of community participation, institutional performance, and service delivery outcomes . The lack of attention to measuring the process and institutional objectives was a particular weakness because these objectives were main objectives in this project (unlike in the first project where they were secondary objectives ). The Beneficiary Assessment, which was used as the main M&E tool, appeared to be flawed in that it was not representative of the project population . Several other aspects of project performance were not adequately monitored (quality of works, etc.), see ICR pg. 21. Nor was an attempt made to adequately measure project efficiency despite this being a weakness in the first project . The ICR also notes some problems with access and connection rates of water supply systems and gas networks (34 percent of project financing went to water supply systems and gas networks ) which could have been addressed through project supervision --"Certain of the projects chosen, particularly those involving water and gas, may have been selected by these non -representative persons and benefited the few over the many, with conections to these services ." (ICR pg. 32). a. Ensuring Quality -at-at -Entry :Moderately Unsatisfactory b. Quality of Supervision :Moderately Unsatisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Government performance was satisfactory . Frequent changes in the Government affected SIF II's performance . There were several changes in the membership of the SIF II Supervisory Board, and the Executive Director stepped down after the presidential elections in January 2004. During this disruption, however, the Ministry of Finance tried to ensure smooth continuation of the project . Changes in legislation led to uncertainties regarding decentralization and the LSG structure as well as initial delays in the implementation of the capacity building component. The performance of the implementing agency, GSIF, was satisfactory . Building upon its experience with SIF I, GSIF effectively administered the project despite the uncertain political situation . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: The quality of M&E design was negligible. Several of the performance indicators defined in the PAD were in fact objectives rather than measurable indicators, and no targets were established . More attention should have been paid to defining performance indicators for the project's "soft" objectives of increased community participation and improved institutional performance. M&E implementation was also negligible. The ICR states that project supervision focused on solving procedural problems and trouble shooting, thus devoting less effort to in -depth analyses of project outcomes, poverty targeting, and cost effectiveness . As noted above, the sample of beneficiaries used in the Beneficiary Assessment may not have been representative . Especially in this follow-on project where the process and institutional objectives were made part of the main project objectives (unlike the first project where they were secondary objectives ), more systematic measurement of process ond institutional outcomes would have been expected . Service delivery outcomes should also have been measured . The ICR identifies other weaknesesses in M&E, ICR pg . 21. The MIS system appeared to be well designed and implemented, with the results used to prepare progress reports for GSIF staff, the Supervisory Board, the Government, the Bank, and other donors . As with other MISs, the system focused on inputs and outputs, and thus needed to be complemented by information on project outcomes to allow better project management for the achievement of the intended outcomes . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The ICR (p. 11) notes that some of the micro-projects had minor negative environmental impacts, and that these impacts were mitigated, but does not say how . Project financial audits provided unqualified opinions, and no major issues were raised in procurement reviews carried out by the Bank . 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately There is insufficient evidence on Satisfactory various aspects of the project objectives (see Section 4 above) as also on project efficiency which is rated modest. Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Moderately Despite being a follow-on project, the Unsatisfactory weaknesses of the first project were not adequately addressed, especially regarding M&E. Both in design and supervision, the Bank failed to give sufficient attention to the measurement of community participation, institutional performance, and service delivery outcomes. The lack of attention to measuring the process and institutional objectives was a particular weakness because these objectives were main objectives in this project (unlike in the first project where they were secondary objectives). The Beneficiary Assessment, which was used as the main M&E tool, appeared to be flawed in that it was not representative of the project population. Several other aspects of project performance were not adequately monitored (quality of works, etc.), see ICR pg. 21. Nor was an attempt made to adequately measure project efficiency despite this being a weakness in the first project . Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The required community counterpart funds in CDD projects should be carefully considered during project preparation. In SIF II, poor communities were unable to pay their required contribution, leading to an assumption of this responsibility by the government . Social and institutional performance indicators are difficult to define, but this needs to be done in CDD projects that attempt to improve community participation, ownership, and management . Even when economic rates of return are not estimated at appraisal due to the demand -driven nature of sub-projects, an effort should be made to evaluate project efficiency at completion by doing cost -benefit or cost-effectiveness analysis on a sample of sub -projects. Long-term sustainability of micro-project infrastructure will only be assured if --as in this case--local authorities take over responsibility from community groups, bringing to bear the necessary fiduciary and technical competence. 14. Assessment Recommended? Yes No Why? A combined PPAR for SIF I and SIF II is recommended in order to verify the assessment, particularly of community participation, institutional performance, and service delivery outcomes . 15. Comments on Quality of ICR: The ICR is candid and well written, and contains an excellent discussion of data deficiencies, including weaknesses in the Beneficiary Assessment methodology . The ICR could have assessed efficiency through cost benefit or cost effectiveness analysis of a sample of completed micro -projects. a.Quality of ICR Rating : Satisfactory