ICRR 13826 Report Number : ICRR13826 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/07/2014 Country : Bangladesh Project ID : P053578 Appraisal Actual Project Name : Social Investment US$M ): Project Costs (US$M): US$M 22.54 US$M 60.3 Program Project L/C Number : C3740; C4450; Loan /Credit (US$M): Loan/ US$M ): US$M 18.24 CH349 Sector Board : Agriculture and Rural US$M ): Cofinancing (US$M): Development Cofinanciers : Board Approval Date : 03/18/2003 Closing Date : 06/30/2007 06/30/2011 Sector (s): Other social services (50%); General transportation sector (40%); General education sector (5%); Media (5%) Theme (s): Other rural development (29% - P); Rural services and infrastructure (29% - P); Rural policies and institutions (14% - S); Gender (14% - S); Poverty strategy; analysis and monitoring (14% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : John R. Eriksson Ridley Nelson Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The Project Appraisal Document (PAD) states the Project Development Objective as : "to develop effective and efficient financing and institutional arrangements for improving access to local infrastructure and basic services through the implementation of community -driven small-scale infrastructure works and social assistance programs " (p.2 and Annex 1, p.28). The Project Credit Agreement provides an identical definition (p.13, Schedule 2). These formulations of the objective mix means and ends . "... implementation of community-driven small- scale infrastructure works and social assistance programs " are means chosen by the project to achieve the objective . This review will adopt the first clause of these formulations as the Project Development Objective . Namely, to: "...develop effective and efficient financing and institutional arrangements for improving access to local infrastructure and basic services." b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: There were four components throughout the project from design to closure . The component costs at appraisal are shown in the PAD (p.6). Three Additional Financing Credits are included by component in the figures shown below . A fifth component was added to the second Additional Financing Credit, as explained in (e) and in 2-d. Notwithstanding these additions and some substantive changes, the project objective remained the same . (a) Strengthening the Social Development Foundation (appraisal : US$1 US$ 1.91 million; actual : US$ 7.63 million ) This entity was established according to the PAD (p.4) as a not-for-profit company and is referred to by the ICR as "autonomous under the Ministry of Finance " (p.5). This project component was comprised of four sub - components: (i) information and communication; (ii) capacity building; (iii) monitoring and learning; and (iv) project management support (including environmental assessment and tribal development ). Technical assistance, monitoring and evaluation studies, establishment and incremental staff expenses, and operating costs were to be financed . US$ 0.38 million; actual : US$5 (b) Institutional Development at the Community -level (appraisal : US$0 US$ 5.00 million ) This component was to support communities in raising awareness, motivating and engaging the rural poor to participate in community-driven initiatives, changing attitudes and behaviors among local stakeholders, development of organizations at the village-level, and preparation of Community Action Plans . The component had two sub-components: (i) information and communication campaign; (ii) formation and strengthening of Village Development Committees, Community Groups and Action Plans . The component was to also finance local information dissemination and environmental assessment and screening . (c) Implementation of Community Action Plans (appraisal : US$19 19 .35 million; actual : US$32 US$ 19. 32 .26 million ) The Plans US$ 32. were to prioritize community needs for small -scale infrastructure and social assistance based on informed choice and eligibility criteria. The project was to finance: (i) up to 85 percent of expenditures of community infrastructure sub-projects; (ii) costs of NGO services to implement social assistance programs, including one -time seed capital and cost of legal assistance for the very poorest and vulnerable groups, and implementation of the tribal development plan; and (iii) costs of services for appraisal and supervision of sub -projects, including environmental supervision. US$ 0.90 million; actual : US$0 (d) Pilot Private Financing in Community Utilities (appraisal : US$0 US$ 0.40 million ) The project was to finance technical assistance and a maximum of 70 percent of approved costs of water and electricity sub-projects. These resources were channeled to leverage private financing of piped water supply to communities through matching grants. The technical assistance was to help the Social Development Foundation identify, develop, appraise, and supervise pilot sub -projects in piped water supply and off -grid electricity. US$ 15 million ) This (e) Livelihood Restoration through Micro -Credit to Flood -Affected Communities (actual : US$15 assistance was implemented from resources provided by Additional Financing Credit II in December 2007 through the Palli Karma Sahayak Foundation, an independent NGO, to provide micro -credit to flood-affected communities. It is shown as a stand-alone component by the ICR, separate from the four original components (p.27). Since this component was entirely financed by an Additional Financing Credit, there are no costs to show at appraisal . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost The total project cost estimated at appraisal was US$ 22.54 million. This was followed by three Additional Financing Credits in 2007-2008, which brought the total actual cost to US$ 60.29 million. Financing The original cost was to be financed by an IDA Credit of US$ 18.24 million and government and local contributions of US$4.30 million. In April 2007 an Additional Financing Credit I (called a "Consolidation Credit‿) of US$8.00 million was provided to augment Component 3 ("Community Action Plans") to support income generating activities . The Project Objective and the four Components otherwise remained the same . In December 2007 an Additional Financing Credit II ("Flood Credit") of US$25.00 million was provided for livelihood restoration after two waves of floods in Bangladesh in 2007. The program was divided into two parts : (1) US$10 million was implemented through the Social Development Foundation to provide emergency assistance and restore livelihoods in flood -affected communities; and (2) US$15 million was implemented through the Palli Karma Sahayak Foundation to provide micro -credit to affected communities (as noted above in 2c(e). In July 2008 an Additional Financing Credit III ("Cyclone Credit") of US$50 million was provided to support restoration and recovery from cyclone damage to livelihoods and long -term disaster preparedness. This support was provided through the original four project components . On April 19, 2011, US$40 million was cancelled from Additional Financing Credit III . The reasons include.(i) over-estimation of Social Development Foundation capacity to expand its operations; (ii) interventions were in a logistically difficult geographic area; (iii) occurrence of a second cyclone, which resulted in uncoordinated relief operations that disrupted longer -term development operations; and (iv) utilization of an NGO as implementing agency that had little operational experience, delaying implementation by 10-12 months. The above figures are amounts made available by agreement but they are not disbursement amounts . The ICR provides the following information on actual disbursements by Credit (pp.4. 45): Original Credit and Additional Financing Credit I (combined):US$27.98 million Additional Financing Credit II: US$22.94 million Additional Financing Credit III: US$ 9.38 million Total : US$60.30 million This total corresponds (with a small rounding error) to the sum of actual cost by component, or US$ 60.29 million. Borrower Contribution The borrower, including Government and local communities, was to contribute the equivalent of US$ 4.30 million (PAD, pp. 1,89) at the time of appraisal or 19.1% of the estimated total project cost of $US 22.54 million. If only the proposed Government contribution of US$ 1.61 million is counted, its share of estimated total project cost was 7.1%. Actual disbursements at project closure were US$ 60.30 million of which US$3.18 million constituted reported Government contributions or 5.3% of total costs. It is assumed that the latter figure includes local community contributions, since neither the ICR nor the project team made the distinction . It is also assumed that this figure represents actual contributions since it is shown in an ICR table (p.4) of “Actual Disbursement,‿ albeit under a “Project Budget (PAD)‿ column. Net of the “Government‿ contribution of US$3.18 million, the implied total cost contributed by IDA would be US$57.12 million or 94.7% of total project costs. The implied increase in the IDA share of total cost from appraisal to actual reflects the substantial IDA infusions for the Additional Financing Credits, especially for flood and cyclone rehabilitation . Dates The closing date of 06/30/2007 was extended to 06/30/2011, thereby roughly doubling the project implementation period. Three reasons are given: (i) some project activities --confidence building, institution development, building long term capacities, and pace of community decision -making-- required more time than anticipated at appraisal; (ii) disruptions caused by floods and cyclones; and (iii) lack of implementing agency management capacity during some key periods. (ICR, p.9) 3. Relevance of Objectives & Design: a. Relevance of Objectives: The relevance of the Project Development Objective is Substantial . The first part of the Objective -to develop effective and efficient financing and institutional arrangements - was still relevant at project closure for those areas of the country where such arrangements had not yet been implemented . The second part of the Objective - to improve access to infrastructure and basic services - also remained relevant in much of the country . In addition, the ICR cites a focus on "strengthening community-driven institutions of the poor " for Additional Financing I (AF I), financial support for flood rehabilitation for AF II, and cyclone assistance for AF III (p.1). But these were not formal objectives . The Project Development Objective, as well as these foci, are relevant to sustained development of the country . All are consistent with the aim of the Government ‘s 6th Five Year Plan to bring Bangladesh to middle income country status through massive poverty reduction efforts . The objectives of this project were also consistent with the overall objective of the Bank's Country Assistance Strategy for Bangladesh (FY 2011-2014), namely, to contribute to "accelerated, sustainable and inclusive growth, underpinned by stronger governance at central and local levels " (p.i). (The ICR refers to the reviewed project as "Social Investment Program Project I" because a second project, "SIPP II," was anticipated that would scale -up the reviewed project.) At one juncture, the ICR labels "improved livelihoods" and "improved resilience to natural disasters " as "new objectives" for AF I and AF II-III, respectively (p.12). But nowhere else in the ICR or in the Additional Financing Agreements is there any indication of a modification of the original Project Development Objective nor any approval by the Board of modified objectives . In fact, the AF II Agreement makes it clear that while the AF's purpose was to "help in meeting the Recipient’s increased funding requirement caused by the impact of floods, " this was to be within the original objectives of the Project (see Financing Agreement, Grant-H349-BD; February 7, 2008; Preface, paragraph 3.01, Schedule 1). b. Relevance of Design: The relevance of the design of the project is Modest . While the original four components plus the fifth additional component were relevant to developing financing and institutional arrangements for improving access, they were not very relevant to the effective and efficient provision of such arrangements, as explicitly embodied in the project objective. Only after the Midterm Review in May 2006 were a number of steps identified to bring about more effective and efficient arrangements (outlined in a Community Operational Manual), including measures to increase community participation, mobilize saving and credit, formalize financial systems, build capacity, etc . The original project design called for the Social Development Foundation to be an umbrella organization overseeing support to communities through NGOs. The Midterm Review modified this approach, with the Foundation rather than NGOs conducting field-level implementation by: Giving priority to strengthening community institutions and developing their capacities in contrast to NGOs, which often delivered goods, training and services, but paid less attention to developing capacities of community members, instead giving more attention to recovering their cost to sustain their own institutions . Empowering communities to design, implement, monitor and maintain infrastructure and other activities, instead of their previous role as passive beneficiaries of interventions designed by NGOs . 4. Achievement of Objectives (Efficacy): The objective of the project was to develop effective and efficient financing and institutional arrangements for improving access to local infrastructure and basic services . The efficacy of achieving this objective is rated as Modest . The main reason is the lack of evidence to demonstrate the extent to which the project developed effective and efficient financing and institutional arrangements . Available evidence indicates poor performance against targets before the Midterm Review and during subsequent project restructuring . The ICR reports several actions under the heading of fiduciary compliance that indicate progress toward effective and efficient financing and institutional arrangements (see below and section 11-b). However, a number of the output level and more of the outcome level achievements lacked both a target and a baseline . The 39 achievements listed in Section 4 show that at the Output level, 7 targets were met or exceeded and only 3 were below target. But at the Outcome/Intermediate Outcome levels, only one achievement met the target and 8 did not. Moreover, there were 10 reported achievements at the Output level and 10 at the Outcome/Intermediate Outcome level for which no targets had been set or baselines established, These results do not provide sufficient basis for upgrading Efficacy to Substantial. Moreover, there was limited information to provide a basis for assessing attribution to the Project (the only reference to attribution is in the discussion of the end -of- project survey on the benefits of road construction, ICR, p .14 – see below under Outcomes). Outputs (listed roughly in order of percent of target achieved ) 1. 95% of women benefited from the project (ICR, p.13) (target of 50% against a baseline of zero, but the Results Framework Analysis shows a target of 90%, which the team says is erroneous . While “benefited‿ is not defined, it presumably means number of women who were beneficiaries of the sub -projects). 2. 1224 Gram Samities (higher level village institutions ) covering at least 87% of target households accessed village development funds in adherence with guidelines (target of 1125 with achieved number representing 109% of target, against a baseline of zero; the figure of 1224 includes data from the 3 Additional Financing Credits in contrast to the target of 760 villages in the Results Framework Analysis, which according to the Project Team reflects only the original project before the Additional Financing Credits ). 3. 1238 villages (about 70% of all 1768 project villages) had Gram Parishad (basic village level organization ) quarterly meetings (ICR, p. 13) (target of 1125 villages or 110% of target). (The Results Framework Analysis reports an achievement of 353 villages, which according to the team represents only those villages with quarterly meetings under the third Additional Financing “Cyclone‿ Credit). 4. 1734 community-driven small-scale infrastructure works were built, including 1376 works under the original project and 358 under the flood and cyclone Additional Financing Credits (PAD target of 1800 works against a baseline of zero but no Additional Financing targets ). Four categories were completed : about 2,490 km of earthen roads, 2,940 drinking water tube wells, 3,160 culverts, and 85 schools repaired. No baselines or targets were provided for these subcategories . 5. 100% of villages had established an operation and maintenance plan and 77% had mobilized funds for this purpose (target of 80% against a baseline of zero). The ICR (pp.42-44) reports several quantitative achievements and targets at the output level for the third Additional Financing Credit as shown below : 1. 353 villages had 70% of target households mobilized into livelihood groups and started savings (target of 350 villages or 101%) 2. 250 village organizations set-up early warning systems and trained in providing help to community members during emergencies (target of 400 villages or 63% of target) 3. 353 Gram Samities had at least 50% of decision making positions occupied by women ( target of 476 or 74% of target) 4. 353 villages had regular quarterly Gram Parishad meetings attended by at least 70% of target members (50% women) (target of 400 villages or 88% of target) 5. 55,000 households accessed employment generation schemes (target of 55,000 households or 100% but 55,000 equals 60% of targeted households) 1. The ICR provides no targets or baselines for the following reported output level achievements : 1. Project funds were transferred from the Social Development Foundation directly to accounts of communities, which were given responsibility for their management . 2. Financial accountability issues at the community level were addressed mostly by an independent center and identified corruption cases were minimal (2% of villages and 0.02% of people. 3. Independent community-level social committees reviewed procurement, funds management and related decision making although procurement training lagged . 4. 129,960 hard core poor and poor households, most of which were normally not deemed creditworthy to access traditional microfinance, accessed livelihood funds loans (p.13) 5. Access to education was also improved as a result of the rehabilitation of 85 schools and of road and culvert construction, which enabled increased school attendance of 58 percent from school rehabilitation and an increase in school attendance of 41 percent from road and culvert construction . 6. Under the Social Assistance Program, a dedicated pilot health program supported by the project, 65,000 rural poor in 22 villages were reached, which according to the ICR (pp.14-15) led to increased awareness of health issues, increased capacity building, and access to local level health care services . However, no evidence is provided for these assertions . 7. A focus on livelihood improvement introduced after the Midterm Review resulted in 258,113 people engaged in income-generating activities who saved a total of US$ 1.23 million (92 million Bangladesh taka). (ICR, p.15) 8. The second and third Additional Financing Credits are reported to have strengthened community capacity to cope with future disasters. Communities were introduced to disaster vulnerability analysis, training in disaster management and plans, and basic equipment was provided, e .g. hand microphones, first aid kits, radios, etc . But community members (with NGO help) rehabilitated their dwellings and recovered from agricultural losses to floods and cyclones. Consultations during ICR preparation suggested that assistance from the Project might not have been sufficient to improve preparedness and resilience of communities to floods and cyclones . 9. A community financing model, envisioned by the PAD, was inaugurated on a trial basis for the first time in Bangladesh and then confirmed by the Mid -Term Project Review (MTR) as a key element of the project . 10. Other relevant institutional changes included (drawn in part from comments by the Country Team ): participatory targeting methodologies to enable the poor and hard core poor to form their own inclusive and accountable village level organizations in accordance with detailed guidelines (Community Operational Manual) a results based financing mechanism was set up to help institutions of the poor to directly access a village-based envelope for participatory prioritization of needs and budgeting milestone based release of funds through a participatory appraisal and transparent decision making process for utilization of funds, social auditing etc . Outcomes/Intermediate Outcomes Listed roughly in order of percent of target achieved . The percentages of target achievement are significantly less than at the output level . 1. 2.5 million hard core poor and poor “benefited‿ from the Project (target of 3 million with achievement representing 83% of target, against a baseline of zero ) [Figures from “PDO Indicator 1,‿ Results Framework Analysis, where “benefited‿ is not defined.] 2. 736 Gram Samities (77%) mobilized operations and maintenance resources to cover their costs (target of 760 (80%) against a baseline of zero). 3. 421,709 employment days were generated for flood affected communities in target villages through provision of community infrastructure works (target of 600,000 or 70% against baseline of zero). 4. Communities saved a total of 92 million taka (target of 141 million taka or 65% against a baseline of zero). This shortfall was reportedly due to slower than planned implementation and less than expected follow -up by the Social Development Foundation. (ICR, p.13) The following are reported results at the outcome level for the third Additional Financing Credit that focused on cyclone rehabilitation and preparedness (ICR, pp. 42-44): 1. 388 infrastructure sub-projects funded and 92 completed (target of 400 villages, or 97%, with at least one sub-project completed per village) 2. 1,840 most vulnerable persons received assistance (one time grant) and accessed the livelihood assistance fund – 604 employed or self-employed (target of 2.250 persons or 82%) 3. 1,514 affected unemployed youth accessed loans /funds from skill development training and started income generating activities (target of 4,500 youth or 44 % of target) 4. 7.243 households benefited from project intervention (infrastructure, skill development, savings and credit ) and have started income generating activities (target of 30,000 households or 24%) 5. 78,154 employment days generated for the Cyclone affected communities through activities (target of 400,000 days or 20% of target) The ICR provides no targets or baselines for the following reported outcome level achievements : 1. An end-of-project survey estimated that beneficiaries of road construction experienced an increase in trade and commerce of 48% of respondents and of agricultural marketing, 39%. These percentages are 35% and 33%, respectively, for beneficiaries of culvert construction . Since non-project Government infrastructure works address inter-village infrastructure not covered by the Project, the ICR argues that these benefits can be largely attributed to the Project (p.14). However, there are other factors, such as weather and volume of crop production that could influence these results as well . 2. As a result of culvert construction, 60 percent of households reported time savings . 76 percent of households saved time from road construction . A 2010 impact evaluation of community infrastructure works showed that while the majority of people spent up to 60 minutes on transportation per day before the Project, this amount was reduced to less than 30 minutes a day as a result of the Project . A weighted average was calculated, and the time saved averaged 17 minutes per person per day according to the survey . 3. 90 percent of the 1407 Social Audit Committees formed in the villages submitted quarterly reports to their respective Gram Parishad, with the aim of ensuring transparency and accountability at community level (p.13). 4. On time repayment rates of 93% and 92% for loans are reported from the saving funds and from the livelihood funds, respectively, as compared with the internationally accepted standard of 95%. (ICR, p.13). 5. At project completion, 7,893 youth (ICR, p. 16) were employed as a result of targeted professional training financed by the Project and 47,300 were self-employed with support from Livelihood Loans, making for a total of 55,193 employed as a result of the Project (ICR, pp. iii and 16, and information from project team). 6. 5.75 million employment-days were estimated to have been generated by construction of community infrastructure works built mostly using paid community labor (ICR, p.16). 7. Of 17,710 “most vulnerable‿ who received a one-time grant, 14,361 graduated to accessing loans from Village Credit Organizations and initiated income generating activities . 8. Provision of safe drinking water led to a reduction of 34 %, 30 % and 19% of incidence of diarrhea, dysentery and hepatitis, respectively . Calendar time intervals are not given for these reports . 9. The project financed 2940 hand tube wells. Given that an average of 12 households were reported to use one tube well, the total number of beneficiaries was estimated at 176,000 persons. 10. Livelihood loans provided by Additional Financing Credits I and II were invested in goats or cattle, small trade business, rickshaws and agriculture and fisheries . On average, these activities generated an incremental monthly income of about 1300 taka (US$17.3), which represents an increase of over 100 percent from before the project (according to the financial analysis in Annex 4). 5. Efficiency: The efficiency of achieving the Project Development Objective is rated substantial . Financial and economic rates of return were estimated by both the PAD and ICR . But the types of activities covered by the two estimates differed considerably. The PAD estimated an economic rate of return of 30.6% for the value of time saved in fetching water as a result of tube well construction and the ICR estimated an economic return (ERR) of 24.7% for income generating activities supported by microfinance . The coverage of total Project cost by the PAD and ICR estimates differs, with the PAD estimate covering 12% and the ICR covering 30%. The expost ERR is probably an underestimate since the methodology used total project costs but employed the benefits from the income generating activities only . Therefore, as compared with the standard minimal project ERR of 12%, the expost ERR is strong. The initial closing date of June 30, 2007, was extended to June 30, 2011, which doubled the implementation period . This was attributed to several causes, including : (i) institution, capacity and confidence building taking longer than anticipated at appraisal; (ii) community-led development resulting in a slower pace set by some communities; and (iii) lack of Social Development Foundation management during some key periods (e.g., no managing director for 5.5 months, no general manager for programs for 1.5 years) (p.9). On the other hand, disruptions caused by the floods and cyclones and developing new approaches and procedures were an inevitable cause of delay . Another indicator of efficiency presented by the ICR is the percentage of project funds that actually reached final beneficiaries in the form of community grants . The share of funds directly or indirectly benefiting the communities was 85 percent of project costs. The latter figure takes into account institutional development as well as community capacity building activities. The ICR also analyzes efficiency by comparing cost per village and cost per beneficiary of the two main implementing agencies: the Social Development Foundation and the Palli Karma Sahayak Foundation. It argues that even though these costs were less for the latter foundation than the former, there were non-valued benefits provided by the former foundation that tend to make up for the cost difference . However, these figures and arguments do not provide an adequate basis for assessing efficiency for the project as a whole . (p.18) ERR)/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The rating of Development Outcome is Moderately Unsatisfactory . While the relevance of the project objective is substantial as is efficiency, the relevance of design is modest in view of the fact that it did not address the effectiveness and efficiency of financing and institutional arrangements that were key elements of the project objective. Efficacy was rated modest owing to incomplete and inconsistent information, particularly regarding effective and efficient institutional arrangements at the outcome level . In sum, given modest efficacy, the shortcomings are assessed as significant, leading to a moderately unsatisfactory rating of development outcome . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Of five risks, three are significant and the two are in the moderate range (nos. 3 and 5).. Therefore, overall risk to development outcome is rated significant . 1. Resilience to natural disasters (floods, cyclones). Natural disasters are likely to reoccur in the future with significant casualties, damage and adverse impacts on incomes and employment . As a result of the project, community preparedness was partially improved and the growth of incomes and employment that occurred increased resilience, but there was a perceived need for basin and region -wide planning and stronger linkages to national disaster management initiatives . 2. Political Interference in Project Management . The Social Development Foundation suffered from an absence of governing bodies and a managing director, as well as a politically appointed staff, and inaccurate or inappropriate decisions regarding project scope and costs . The Foundation did build procedures and working methods aimed to mitigate the risk of interference (e.g., a 2008 human resource policy). 3. Institutional Sustainability of the Social Development Foundation : The new external recruitment process created anxiety among existing staff that needed to reapply . There was a lack of clarity whether the Foundation would be able to retain experienced staff . Bank and Foundation management were aware of the issue and it was hoped that implementation of a new human resources manual could ameliorate the situation . The formation and funding by the Government of an Endowment Fund could have a positive impact on Institutional Stability . 4. Sustainability of Village Institutions : It was estimated that only about 20% of community institutions aided by the project were sustainable without continuing Foundation support . The second Social Investment Project Program was intended, among other things, to help these institutions become self -sustainable. Nonetheless, 20% constitutes a steep climb to reach substantial sustainability . 5. Sustainability of Infrastructures . There is a risk that infrastructure built under the project will not be adequately operated and maintained, leading to its deterioration and disuse . 83% of the villages under the consolidation phase (Additional Financing Credit I) developed operations and maintenance plans for community infrastructure works and 79% implemented these plans as scheduled . Since the communities planned, implemented, own and monitor infrastructure, it can be argued that sustainability is more likely . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: Quality at entry is rated as moderately unsatisfactory . There were significant shortcomings in the design of the original project that might have been anticipated, e .g. the reliance on NGOs as implementing entities . This was corrected at the Midterm Review when the decision was made to use the Social Development Foundation, an autonomous organization under the Ministry of Finance, as an implementing agency down to the village level . The consolidation phase of the project, including the first Additional Financing Credit, proposed more efficient processes and arrangements, building on existing experiences and introduced necessary flexibility . However, they did not adequately anticipate the risks that could adversely affect implementation of the flood and cyclone Additional Financing Credits. The choice of the Palli Karma Sahayak Foundation and an NGO as entities for implementing cyclone relief under the third Additional Financing Credit was poor, given their lack of experience in disaster relief and was a main factor leading to cancelation of US$ 40 million. In addition the M&E design was moderately unsatisfactory. at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: Quality of supervision is rated as satisfactory . Two of four Task Team Leaders were located in Dhaka, the latter deemed an advantage. According to the Project Team, supervision missions were undertaken every six months. . They addressed substantive issues, provided technical assistance and guidance to the project implementation team. Supervision missions were concluded by aide -memoires, with recommendations and agreed actions. TTLs and the Social Development Foundation were responsive to issues raised by missions . The Bank Team simultaneously undertook post MTR redesign and addressed the need to meet what the ICR deems substantial disbursement targets resulting from rapid financial expansion . Technical assistance and south -to-south exchanges were organized, drawn from similar projects in the region . These exchanges contributed to the Foundation ’s capacity and its ability to contribute to meeting project performance targets. For the second and third Additional Financing Credits, the supervision team provided continuous technical and capacity -building assistance to the Foundation . This included guidance for a "community to community" approach to scaling up the quality of village level processes (information drawn from Country Team). The team also supported the Foundation in creating a Disaster Risk Management Unit and assisting expansion of its activities in disaster affected areas . There were relatively minor shortcomings in the quality of supervision and the rating is judged to be satisfactory . Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: Government performance is rated moderately satisfactory . The Economic Relations Division of the Ministry of Finance supervised all externally financed projects and was extensively briefed by all Bank supervision missions. The Division subsequently ensured that the implementing agency, the Social Development Foundation, followed Bank recommendations. The ICR reports that the Government provided good support to community driven development activities implemented by the Foundation and was responsive to the need for changes during implementation, including as a result of the midterm review . But the Government failed to appoint the Foundation Governing Body for a period of 5.5 months. The Government and Governing Body held infrequent meetings with the Foundation, which prevented timely decisions . They also failed to appoint the Managing Director for about five months in 2010. These are moderate-to-significant shortcomings. Government performance is rated therefore moderately satisfactory. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: Implementing agency performance is rated moderately satisfactory . The main implementing agency was the Social Development Foundation. At completion, it had 800 staff, mostly dedicated, experienced and competent . The Foundation adjusted to changing circumstances over the course of implementation . However, as a relatively new institution (created in 2001), the Foundation encountered several obstacles, including the lack of a human resource policy throughout the project and the absence of a Managing Director for 5 months in 2010 and of a General Manager for Programs for 1.5 years in 2008 and 2009; and an ineffective M&E system (see section 9 below). However, during the last project year, the Foundation made progress in several areas, including : (i) completion or rollover of unfinished tasks of the flood and cyclone Additional Financing Credits to Social Investment Program Project II; (ii) successful preparation for and negotiation of the latter project; (iii) adoption and implementation of a new human resource policy; (iv) a significant increase in delivery capacity, resulting in disbursement of approximately US$ 13 million (not attained by the Foundation previously ), constituting an increase of over 30% compared to previous years, while keeping the same quality standards at field level . The Palli Karma Sahayak Foundation was a secondary implementing agency, which implemented the livelihoods activity under Additional Financing Credit II and, together with NGOs, implemented disaster relief activities under Additional Financing Credit III. While the ICR does not rate the performance of this foundation, the discussion of efficacy of achievement of the livelihoods activities suggests that the Foundation performance was satisfactory . However, the Palli Karma Sahayak Foundation may share some responsibility with the Social Development Foundation for the less-than-satisfactory performance under Additional Financing III . Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The design of the M&E system suffered from a number of weaknesses . The definitions of performance and results indicators were unclear, which created “major difficulties‿ in preparing the (ICR p.9). The indicators were not consistent between: (i) the PAD core text (the key performance indicators section ); (ii) the PAD project design summary in the log frame in Annex 1; (iii) the indicators developed for the 3 Additional Financing Credits; (iv) the indicators reported by the Social Development Foundation in its end -of-project Results Framework; and (v) the indicators measured by independent impact assessment surveys . The inconsistencies and absence of “Comments‿ sections in the ICR Results Framework Analysis that might explain the inconsistencies severely limit the usefulness of the Analysis. The ICR further notes that: “The results framework established in the original PAD was not strictly followed throughout the project, nor did the additional financing project papers provide a solid basis upon which to track progress‿ (p.28). As a result, Annex 2, “PDO Achievement Table,‿ in a number of instances could not use the PAD results framework indicators because they were not reported but instead used proxy indicators from the Social Development Foundation results framework (see ICR, Annex 2 tables). b. M&E Implementation: The design weaknesses led to difficulties in implementation of M&E activities . Insufficient attention was paid to systematic reporting on concrete, practical indicators of project achievements, such as the use of infrastructure works, development of livelihood activities and poverty reduction . M&E implementation was also made difficult by the post-midterm review switch to a more complex project . While the original M&E system was based on the PAD indicators, the new system, based on the Project Implementation Plan, was radically different . A results framework was created to reflect the new project design, but with inconsistencies among the indicators in the Additional Financing Credits and the PAD. c. M&E Utilization: Technical assistance provided by the Bank helped to develop the following activities that involved M&E utilization : A village matrix that aggregated information on community activities and achievements that was updated monthly in a participatory manner and posted on a board outside the Gram Samity office . This proved to be a tool for transparency and accountability . A third party process monitoring system was implemented by an independent agency, the Centre for Natural Resource Studies (CNRS). This system helped to restructure the project approach at the midterm review, including providing monthly recommended action tables and reporting on irregularities and corruption cases . Monthly monitoring review meetings took place at district and cluster levels to track progress and resolve implementation issues. Quarterly “Monitoring & Learning‿ forums including all field staff were held with community involvement that led to action plans to address emerging issues . Although a web-based Management Information System was developed to track project implementation, its effectiveness was limited due to continuous changes in project design and the fact that the Social Development Foundation was not equipped to handle management of the software . Notwithstanding these utilization activities, overall M&E quality is rated modest owing to severe weaknesses in design and in implementation. M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The PAD (p.22) indicates that two Safeguards were triggered : (1) Environmental Assessment (OP/BP/GP 4.01) and (2) Indigenous Peoples (OD 4.20). The Environmental Safeguard was designated as Category B, Moderately Significant Environmental Impacts. A screening process is described in the PAD intended to select sub - projects with minimal or no environmental impact. No mention is made of an Environmental Assessment . Regarding Indigenous Peoples, a Tribal Development Plan was developed in consultation with tribal populations (plan details are in Annex 2, Attachment 12 of the PAD). However, the ICR contains no mention of either of these Safeguards except for an indication in the Data Sheet that the Project was designated Environmental Category B . Three environmental specialists are mentioned among the Task Team Members (Annex 5). There is no mention of “tribal‿ or “indigenous‿ peoples. b. Fiduciary Compliance: Financial management conducted by the implementing agency (Social Development Foundation) was consistently reported as satisfactory (ICR, p.10). It developed an internal audit arrangement to reduce fiduciary delinquency . It also appointed an External Audit Agency to investigate compliance issues . A governance and downward accountability framework was introduced and a community -driven financial management approach was followed . The ICR does not indicate how many audits were performed or what the findings were . In an environment where misappropriation of funds was a major issue for development activities, the project established mechanisms to minimize leakages and maximize the share of project funding directly used by communities for development activities . Project funds were transferred from the Foundation directly to the accounts of communities, which were given responsibility for their management. Disbursements were made in installments based on Foundation appraisals of village proposals and completion of physical milestones . The process monitoring system mentioned in the previous section was employed for ensuring financial accountability of interventions. In an estimated 90% of cases, the independent Centre for Natural Resource Studies addressed problem cases directly with those concerned, without needing to involve Foundation management . At project completion, corruption cases occurred in about 20 to 25 villages (2% of villages) and concerned 40 to 50 people (0.02% of people involved). In rare cases, members of Foundation staff were involved and were consequently fired . A Community Operational Manual developed with and adopted by communities included a procurement manual to serve as a guide to procurement committees for the purchase of goods and contracting of works . At completion, an estimated 2,000 community members had been trained in procurement . However, the limited staffing of the Foundation procurement team meant that the pace of delivery of these trainings was slower than optimal (ICR, p.10). Community-level social audit committees, independent from other community and project institutions, had the mandate of reviewing procurement, funds management and decision making processes . Bank procurement rules were adhered to by the Foundation although, in some cases it found them cumbersome and not flexible enough, which led to delays caused by the need for Bank approval on a number of items . c. Unintended Impacts (positive or negative): The ICR reports two types of positive unintended impacts : 1. A restoration of “trust, pride and some positive behaviors ‿ in the communities, including a trusting relationship with the Social Development Foundation, which was contrasted with initial fear and skepticism born of previous experience with external institutions . 2. The second impact may have been unintended but it would not be unexpected . The ICR reports, partly based on survey evidence, that food and nutrition security improved resulting from: (i) more reliable transportation resulting from rural road and culvert construction; (ii) that 55 % of loan recipients invested in livestock, leading to increased consumption of milk and meat; (iii) increased welfare from the Social Assistance Program; and (iv) wages from infrastructure construction and other livelihood activities . An impact assessment survey revealed that households having three meals per day increased to 60 % of the households compared to 8% before the project. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately The relevance of design is modest in Satisfactory Unsatisfactory view of the fact that it did not address the effectiveness and efficiency of financing and institutional arrangements that were part of the project objective. While efficiency and the relevance of objectives were substantial, there was insufficient evidence, particularly at the outcome level, of achievements demonstrating that effective and efficient financing and institutional arrangements had been achieved. Therefore, efficacy was judged modest and outcome rated moderately unsatisfactory. Risk to Development Moderate Significant Of five risks identified by the ICR, three Outcome : are significant and two are in the moderate range. The overall risk is therefore significant. Bank Performance : Satisfactory Moderately Quality at Entry is moderately Unsatisfactory unsatisfactory in view of significant shortcomings in the design of the original project and the choice of implementing agency for the third Additional Financing Credit. In accordance with the IEG/OPCS Harmonized Criteria, overall supervision is rated Moderately Unsatisfactory. Borrower Performance : Satisfactory Moderately Owing to failure to fill gaps in Satisfactory governance and management of the implementing agency, government performance is rated moderately satisfactory. The two implementing agencies had moderate performance shortcomings, resulting in an overall moderately satisfactory rating . Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The following lessons are drawn from the ICR with editing and some modification .. (a) Community -Driven Development Projects need time to mature before being rapidly expanded in scope and financing . An adequate assessment of the implementing agency ’s capacity for a Community-Driven Development Project is required. An agency with a weak capacity will not be able to effectively manage a rapid expansion in the scope of its activities and funding . Moreover, the addition of responsibility for addressing needs of logistically difficult areas, such as those affected by natural disasters, will add further strain on institutional capacity . (b) The fiduciary accountability of a Community -Driven Development operation is more likely to be assured when a community financing model is employed that provides (i) improved access by the hard core poor and poor, including women, to financial services; (ii) social auditing and community assessment mechanisms; (iii) a governance and accountability action plan; and (iv) an independent process monitoring system in order to ensure timely addressing impediments as well as learning . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR misplaces virtually all of its discussion of fiduciary compliance issues under “Safeguards,‿ but as noted in Section 11, notwithstanding the fact that Environment and Indigenous Peoples Safeguards were triggered, they are not discussed in the ICR at all. This major omission is unexplained . The ICR is inconsistent in its depiction of the project development objective, stated at three junctures as to : "develop effective and efficient financing and institutional arrangements for improving access to local infrastructure and basic services" but adds the following means (as opposed to ends): "through the implementation of community - driven small-sale infrastructures works and social assistance programs " (pp. iii, 1, 12). But at one juncture (p.12), the ICR adds "improved livelihoods" and "improved resilience to natural disasters " as "two new objectives" introduced by the three Additional Financing Credits . As noted above, the AF Agreements make it clear that the program objective remained the same. No modified program development objective was approved by the Board . The overall quality of the evidence and analysis in the ICR and its results -orientation is weakened by the lack of consistency among results reporting in the Results Framework Analysis, on the one hand, and in the main text, on the other hand. In some instances, there are inconsistencies between the figures reported for the same result in the Framework and those reported in the text, and in other instances a result will either be reported in the Framework but not in the main text or the reverse . According to the project team, the targets and achievements reported in the Results Framework Analysis refer only to the original project and do not include the three Additional Financing Credits. This is not explained anywhere in the ICR (Annexes 2 and 3 provide output and some outcome breakdowns by original project (i.e., the PAD) and each Additional Financing as well as by Component, but for the most part do not show totals for the entire operation ). However, even the assertion that the Results Framework Analysis contains data only from the original project is not consistent since the project team reports that the figure given for Indicator 5 in the Results Framework Analysis for number of villages holding Gram Parishad meetings is from Additional Financing Credit III. This is not explained anywhere in the ICR . Evidence and analysis are also weakened by missing baselines and targets for a number of results (as noted in Sections 4 and 10 above). The ICR does indicate awareness of some of these problems (p.12) when it notes: As mentioned earlier, the main challenge of the ICR team has been to assess the achievement of the most relevant performance indicators, as these have not been consistent between the PAD, project papers prepared for the three AFs, the final SDF Result Framework Table, and indicators collected by independent impact assessments. The ICR is marked by a lack of conciseness, some irrelevant material and repetition, and a tendency to use hyperbole not supported by evidence . The sections on “Quality at Entry‿ and “Government Performance “ are very thin and tend to lack specific evidence . The Results Framework Analysis contains nothing in the “Comments‿ boxes. This is unusual since in virtually all other ICRs reviewed the ICR team has chosen to enter some explanatory or qualifying comments in the comments box for each indicator . This is partially rectified in Annexes 2 and 3 of the ICR where explanatory comments are provided for a number of the results reported for detailed indicators . Similar explanations should have been included in the Results Framework Analysis . This would have clarified some of the issues with the Results Framework Analysis noted above . As it stands, the usefulness of the Analysis is severely limited. The first Project Cost and Financing Table in Annex 1 of the ICR (p.26) contains an arithmetic error of US$ 0.20 million. This is relatively small but it is not trivial and for a main financial table should have been cross - checked. The “Total Government‿ cell should be US$3.18 million, rather than US2.98 million. Similarly, the “Total Budget Allocation‿ cell should be US$64.42 million, rather than US$64.22 million. The table misleadingly indicates that the figures in the first three “Budget‿ columns are from the PAD. The figures also include the Additional Financing Credits, which were not anticipated at the time of the PAD . a.Quality of ICR Rating : Unsatisfactory