IEG Report Number: ICRR14860 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 10/30/2015 Country: Djibouti Project ID: P088876 Appraisal Actual Project Name: Dj-Urban Poverty Project Costs (US$M): 3.00 5.74 Reduction Project L/C Number: Loan/Credit (US$M): 5.90 5.74 Sector Board: Urban Development Cofinancing (US$M): Cofinanciers: Board Approval Date : 05/31/2012 Closing Date: 05/31/2012 12/31/2014 Sector(s): General water; sanitation and flood protection sector (40%); General transportation sector (30%); General education sector (10%); Central government administration (10%); Other social services (10%) Theme(s): Urban services and housing for the poor (50%); Participation and civic engagement (25%); City-wide Infrastructure and Service Delivery (25%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Peter Nigel Freeman George T. K. Pitman Christopher David IEGPS1 Nelson 2. Project Objectives and Components: a. Objectives: The Djibouti Urban Poverty Project was part of the 2008-2012 multi-donor Djibouti Poverty Reduction Program supported by IDA, African Development Bank, Islamic Development Bank and Agence Francaise de Developpement. The development objective of the overall program was to increase access to and use of local infrastructure and basic urban services, community development and micro-finance by the targeted poor population of Djibouti-Ville and secondary urban centers. The specific objectives of the IDA project according to the Financial Agreement (page 5) and Project Appraisal Document (PAD, page 4) were: "to increase access to basic economic and social infrastructure and to community development opportunities within Quarter 7 in the City of Djibouti ". b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project had three components: (i) Infrastructure improvements: (appraisal cost estimate US$ 1.73 million; plus US$ 1.60 million additional financing; cost of completion US$ 3.47 million). Rehabilitation of the 1.5 km of road network, including drainage, upgrading of latrines, and 1 km of street lighting; and (ii) construction of a 400 square meter Community Development Center (CDC) for community activities, rehabilitation of a 4,500 square meter public garden, as well as preparing a sports field and supplying sports equipment - based on the experience accumulated under a Social Development and Public Works Project. ii) Community development component: (appraisal cost estimate US$ 0.35 million; cost at completion US$ 0.58 million) to focus on four types of demand-driven activities (or sub-projects): (i) literacy and post-literacy campaigns for men and women separately; (ii) vocational training and professional placement for the youth; (iii) strengthening the capacity of community women and youth groups; (iv) and environmental management and community hygiene, primarily consisting of a garbage collection program. The component also included support for the commune of Boulaos. iii) Technical assistance: (appraisal cost estimate US$ 0.48 million; cost at completion US$ 0.68 million ). A study was to be carried out to assess the baseline conditions in the neighborhood, as well as a social impact study before project completion to assess the results. Support would be given for the establishment and operation of the Djibouti Social Development Agency (ADDS), the project entity, which would replace the Djibouti Agency for Implementation of Public Works. In addition the project supported the establishment of a fully functional Management Information System in ADDS. Additional financing and restructuring/ revised components: The results framework was subject to minor revisions, however, the PDO remained unchanged throughout. In 2010, under basic infrastructure, two outcome indicators were added: 2,300 residents would be provided with access to improved sanitation and 5,000 residents would be provided with access to solid waste collection. The team then had second thoughts and these indicators were later dropped in the 2013 restructuring. There was no change in the other outcome indicators. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost: Planned costs were US$ 3.00 million including US$ 0.44 million unallocated; actual costs were US$ 5.90 million. In 2010 an additional grant of US$ 2.9 million was extended after discussions with the local communities to enable ADDS to expand ongoing activities and increase the impact of the project. Construction of a health center and of a second CDC was added under the first component, as well as further drainage and some landscaping. Additional funds were also provided under the second component for community mobilization, professional training, sanitation and solid waste.Under a restructuring in 2013, the components were revised to reflect prevailing circumstances. Under component i) the total length of the streets to be rehabilitated, which was initially 1.5 km was increased to 3 km with the additional grant, but later reduced to 2.2 km, when the unit cost turned out to be much higher than expected. Under component ii) the garbage collection sub-component was limited to technical assistance, as two other donors were prepared to finance a community based scheme in Djibouti City. A survey was also carried out to assess the willingness of households to pay for latrines. When the cost was deemed too high by the population, direct private investment was no longer considered as an option in this area. The preparation of an urban master plan for Djibouti City was added. ADDS fees were estimated at approval to be US$ 0.35 million and US$ 0.37 million was expended on this item by completion. Financing: The project was financed by two IDA grants; an original project grant of US$ 3.0 million and additional finance of US$ 2.9 million under the Crisis Response Window in February, 2010. This window was intended to assist IDA countries to deal with the global financial crisis. The lending instrument was a SIL. At closure US$ 5.7 million had been disbursed and the remaining US$ 0.2 million cancelled. Borrower contribution : None was planned or contributed. Dates: The closing date was extended three times mainly due to procurement delays. In the January 2013 level-2 restructuring the closing date of the two grants was extended to May 31, 2014. It had already been extended by one year to May 31, 2013 when the additional grant was approved. Finally, there was a further extension to December 31, 2014 for a total of two and a half years. 3. Relevance of Objectives & Design: a. Relevance of Objectives: High Despite an income per capita of US$ 1,040, Djibouti experiences very high poverty and unemployment rates. More than three-quarters of the population located on the left bank of the Ambouli river lives on less than US$ 3 per day. Djibouti-Ville manifests itself via a high incidence of informal and sub-standard housing, low rates of access to basic urban services including water supply, sanitation, electricity, solid waste management and access roads. Consequently, it has very low health indicators. The government, therefore, aimed to improve this situation; based on participatory consultations, with a focus that included a Community Development Center to form a social hub for vocational and recreational use. Investments in the vicinity are designed to maximize the use of short-term employment opportunities. This specific project concerns an urban subdivision of only 35 hectares on which live some 27,000 people. The project objectives were highly relevant to Djibouti's urban development challenges; project objectives were in line with the 2007 National Initiative for Social Development, Djibouti's Vision 2035 and consistent with the FY 05-08 country assistance strategy. The Djibouti Country Assistance Strategy (2005-08) aimed to accelerate the development of human resources through the implementation of social programs aimed at reducing poverty and gender disparities, and to reduce unemployment through the development of labor-intensive activities, micro-finance provision and job training. It is also consistent with the CAS intentions of reinforcing the coordination with other development partners. Project objectives remain relevant to the FY 14-17 Country Partnership Strategy (CPS). The first pillar of the CPS seeks to support the government's efforts to alleviate poverty and boost shared prosperity by reducing the vulnerability of Djibouti's population directly through improvements in social protection and disaster risk mitigation, as well as indirectly through enhancements to education, health, and basic infrastructure services that aims to reduce vulnerability. Under its outcome 1.4 to increase access to basic infrastructure in disadvantaged rural and urban areas, the CPS aims to maintain and expand assistance to the urban and housing sector focusing on the poorest and most vulnerable neighborhoods of Djibouti-Ville, specifically Q7. b. Relevance of Design: Modest The objective of the project was modest and the project design was simple, in keeping with the limited amount of financing available. The decision to exclude micro-finance was pragmatic and reduced the level of complexity. In broad terms, the causal chain between the Bank funding and the intended outcomes was clear and straightforward. Because of the very limited time available for preparation, consultation with beneficiaries to fine-tune the project interventions only took place during the early stages of implementation. Even so, the design team failed to come up with a plan to ensure poverty targeting to reduce inequality in the Q7 area. Given the subsequent reactive nature of the design - partly as a result of the need to avoid overlap and duplication with other donor partners' inputs - the results framework also later had to be modified several times. 4. Achievement of Objectives (Efficacy): "To increase access to basic economic and social infrastructure and to community development opportunities within Qarter 7 in the City of Djibouti ." (a) to increase access to basic economic and social infrastructure: Modest Outputs:  Completion rate of road rehabilitation at mid-term was 0%, but all roads were completed at grant closing.  Number of person days of short-term employment created under the project was initially targeted as 8,000, was revised to 20,000 at restructuring, and actual accomplishment was 50,525,  Number of direct beneficiaries (target 16,800 of which 50% female); actual 12,016 - the percentage of females is unavailable but believed to be less than 50%. The target was too high because no improvements were planned for the Djaga Bouldouq section of Q7..  The number of properties within 50 m of surfaced roads was targeted to increase by 35% and this target was exceeded (48%). There were 1,170 properties located within 50 m at the beginning of the project and an additional 575 by 2014.  Although all properties bordering on renovated streets were to have access to appropriate sanitation, appropriate sanitation was not defined. The intended link of sanitation with the road rehabilitation component did not occur and the indicator was dropped at project restructuring.  Percentage of households aware of the benefits of investing in new latrines. The target was 10%. Although this was not monitored a socio-economic survey in 2011 showed that 22.8% were aware of the benefits - how far this is attributable to the project is not known. Outcomes:  The project partially achieved its objective to increase access to basic economic and social infrastructure and to community development opportunities within Q7. For example, the project increased basic economic and social infrastructure through the construction of four roads and associated drains, resulting in an increase of 48 percent of households within 50 m of a surfaced road, (37 percent over the target). Many people benefited from regular solid waste collection by truck from the Djibouti municipality because of improved accessibility on the roads built in Q7 through the project.  A major shortcoming was that project activities focused on the richer section of Q7, where it was easier to build roads and find space for social infrastructure. Under normal circumstances, the project preparation team would have flagged the disparities between the two sections of Q7, and have proposed that a comprehensive restructuring plan for the Djaga Bouldouq slum be prepared. Had this been done, the follow up project could have included specific actions to improve the living conditions of the slum residents. As it was, there were expected to be 16,800 beneficiaries of the project, but only 12,016 actually enjoyed benefits and most of the people living in the poorer area were excluded. No provision was made to tackle the lack of land titles. (b) to increase access to community development opportunities: Modest Outputs:  The Community Development Center (CDC) construction was completed by June 2010.  Djibouti City urban master plan was completed: launched 2012, delivered 2013, review workshop 2014.  10% of youth in Q7 to make use of CDC facilities and equipment. Data for youth specifically were not collected and the indicator was dropped at project restructuring.  540 residents of Q7 make use of CDC facilities and equipment. This indicator was added at restructuring and 1,512 users were recorded. However, the concept of regular users was not considered so there may be serious double counting in this figure.  47% of Q7 residents make use of the health center. This was added at restructuring, but usage was zero as the center was not yet in operation at project closing.  Percentage of Q7 residents of all age groups (of which 50% women) benefit from community development projects. Original target 10%, revised to 5% at restructuring, 3.2% actual. 809 people benefitted of which 139 received vocational training (90 men and 312 women), 192 members of associations (20 women's associations out of 38), 12 persons training for facilitation (10 men and 2 women), 39 elected officials for special training and 25 for first aid (gender not specified). The women's percentage was likely met, but exact figures are unavailable, Outcomes:  Within Q7, several community facilities were built including a Community Development Center (CDC), a health center, two sports fields and a children’s garden. The initial target of 1,000 Q7 residents making use of the CDC was over-ambitious and was lowered to 540 during project restructuring. About 936 young people used the center, which also served as a venue for two conferences. It was expected that 47% of Q7 residents would make use of the health center, which was built, but was not in operation at grant closing. A number of literacy and post-literacy, as well as vocational training courses were also provided, but usage was well below target. The two sports fields were intensively used, and lighting was installed to facilitate night usage. 5. Efficiency: Negligible Economic and financial efficiency: The PAD, which was completed under time pressure, did not provide an estimate of the economic rate of return (ERR) of the project. Instead, it recommended two methodologies to be used for cost benefit analysis (hedonistic price regression and resource savings approaches). The PAD indicated (page 57) that the economic analysis would be completed prior to effectiveness. This did not happen for reasons not fully explained, but the fact that household budget data were unavailable meant that the PAD methodology could not be followed. Alternative approaches would likely introduce a bias towards more easily measurable items such as road infrastructure. The ICR refers (page 21) to an earlier socio-economic study carried out in 2009 in which Q7 was compared with another neighborhood. However, no estimate of benefits was provided to justify the claimed ERR of 12 percent or explain why this was based on a 7.5 percent discount rate (normally only relevant to calculate net present value). In IEG's view this result is in any case highly questionable, as the study sample was small and the details of estimating the benefits were not provided. Similarly, analysis carried out on the follow-up project is only broadly indicative of the efficiency of the project under review and cannot be used as a substitute. The cost-benefit analysis of the project was simply not done. Given the nature of the project, no financial analysis was necessary. Operational and administrative efficiency: Due to the iterative nature of the design and implementation, the project began before the beneficiaries had been properly consulted and before the establishment of ADDS. This delayed effectiveness by nearly a year. The closing date was also extended three times for a total of two and a half years. There were inefficiencies in ADDs, especially in procurement, Some bids were higher than expected, while competition for short-term jobs in road building led to conflicts that turned violent at times. There was no mitigation strategy for this. There was also a lack of commitment to conducting beneficiary surveys that could have provided a proper measure of project efficiency. An agreement reached at mid-term review on how to deal with the higher cost of infrastructure led not only to restructuring, but a revised institutional framework for CDCs following a new government decree. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: With modest efficacy for both objectives, negligible efficiency, high relevance of objectives, but modest relevance of design, the overall outcome rating is moderately unsatisfactory. a. Outcome Rating: Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Maintenance of infrastructure and facilities remains a major issue even after years of Bank support to the urban sector in Djibouti. The ICR (page 12) advised that there is still no systematic policy for maintenance of existing infrastructure and facilities. A Road Agency has been created recently to take responsibility for road maintenance, but at this time, although a step in the right direction, funding allocations for that purpose remain far less than required. The roads built with project support are not cleaned routinely and are likely to deteriorate rapidly. The management of the CDC initially was a cause for concern, but the government's impact on the institutional framework for such centers has been a positive step. More usage needs to be made of the CDC, requiring a more hands-on approach, while the Health Center had not yet opened at project closing. Regarding the CDC, there was a disconnect between the training provided and the types of job available. The usage of the facility has also been lower than expectations. For example, girls cannot use the facility in the evenings as they are not permitted to go out at night. An outstanding issue is still the question of land titling - only seven percent of owners on Q7 had a land title. While this had been recognized in the PAD, there was no Bank strategy to overcome the problem, although the government independently enacted a degree making it easier to get a title. The completion of the Djibouti master plan is a positive step forward, but implementation needs to be actively pursued in the follow-on project. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The project objective was modest and the design was simple. in line with the limited amount of IDA resources available. Moreover, the project itself was part of an overall program to which the Government had committed. On the other hand, the analysis of the sector and of the neighborhood that would benefit from the project was seriously inadequate because of the limited time for preparation, including discussions with the affected community. The risk section was weak. There was no economic analysis of the project. At the negotiation stage, much remained to be done before the start of project implementation, including steps to establish ADDS and complete the necessary technical studies. The Bank did not allocate sufficient time and resources for a high risk project in a difficult environment. Quality-at-Entry Rating: Moderately Unsatisfactory b. Quality of supervision: Three task team leaders were responsible for project supervision. They endeavored to help ADDS address project issues and implement the project; they were reasonably flexible and mobilized additional funds. The level of focus on safeguards policies by supervision missions was appropriate for an environmental category C project (later upgraded to category B because of a change in definition). However, the ever-changing results framework was not conducive to a sound monitoring and evaluation (M&E) system. There was also no real commitment to M&E and this aspect was consequently conducted poorly with incomplete results. Project restructuring was a missed opportunity to streamline the results framework and set realistic targets. The implementation progress and project development outcome ratings were generally too generous in view of persistent delays in project implementation. Financial management encountered persistent certification and audit issues and the team was unsuccessful in avoiding serious procurement delays, which led to three project extensions. Since the appraisal did not take into account the fact that Q7 comprised two sections, a richer part and a much poorer one, Djaga Bouldouq (average per capita income of about 37 percent of residents in the other section), implementation activities focused only on the richer section of Q7, which was better off than a large number of neighborhoods in Djibouti City. There was no attempt to rectify or retrofit the project through poverty targeting even though the project restructuring presented an opportunity for this. Quality of Supervision Rating : Moderately Unsatisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government’s decisions concerning the initial project concept left little time for adequate preparation and appraisal of the project. Delays in requesting the Bank to restructure the project and in opening up the health center for service also had a negative impact on project benefits. The decision to establish ADDS as the implementing agency through the merging of Public Works (ADETIP) and the Social Development Fund, was too ambitious and led to the delays in this entity, which persistently slowed progress. There was, moreover, little interest from Government in a strategy to ensure the new infrastructure would be adequately maintained or to ensure that the poorest people in Q7 would also benefit from the project. On the other hand, a 2011 decree to address the issue of inadequate management of CDCs had a positive impact on the Q7 CDC. The council of elders and the Mayor of the Boulaos commune were also helpful in solving conflicts arising during the selection of candidates for road construction jobs. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: ADDS was established in haste and the inevitable teething problems and unfamiliarity with Bank procedures led to lengthy delays and poor support for M&E, According to the ICR (page 13), its project implementation function may have been overwhelmed by the number of projects it was responsible for. For example by 2014 it was responsible for 10 projects from 7 external partners as well as a further 13 projects financed by the National Solidarity Fund. Project implementation suffered accordingly and took much longer than initially expected. Inadequate procurement methods contributed substantially to these delays and was the major factor in the request for three extensions of time for the project. ADDS was not prepared to conduct a beneficiary survey or stakeholder workshop at the conclusion of the project and was not sufficiently committed to monitoring and assessing results. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Monitoring and evaluation of the outcomes and results of the project was designed to be carried out by ADDS. According to the Decree governing its procedures and organization, one of ADDS’S functional departments which was to be fully dedicated to project identification, monitoring and evaluation. The project was also to provide resources for the training of the staff assigned to this department. An independent impact evaluation study was to be conducted prior to project closing. However, ADDS was new to this task and the results framework went through several changes resulting, according to the ICR, (page 8) in some confusion. Some indicators were also poorly designed because the needed data were not collected or available. However, four of the five initial outcome indicators such as number of direct project beneficiaries and person-days of short-term employment generated were kept throughout. b. M&E Implementation: The indicator related to sanitation was changed from "all properties bordering on renovated streets have access to appropriate sanitation", to "the percentage of households aware of the benefits of investing in new latrines". An outcome indicator dealing with regular solid waste collection was added in the process of preparing the additional grant, but dropped again during project restructuring. (These changes were due to the higher than expected cost of improved latrines, and the decision to address solid waste issues through a separate project). A new outcome indicator was added, namely, the number of direct project beneficiaries, of which 50 percent were to be female. This indicator, however, required a nominative registration system to avoid double counting, which was not pursued. Output indicators were straightforward. Initially, the targets included at mid-term: completion of 25 percent of road and drainage works, construction of the CDC, four sub-projects completed and all other identified under the community development component as well as the generation of 3,000 person days of short-term employment. The targets were modified at restructuring so that by project completion(the end 2012) it was expected that the construction of (i) the roads and drainage works would be 50 percent completed, (ii) the CDC was 100 percent completed; and (iii) the health center launched. Also, it was expected that the indicator for the sub-projects mentioned earlier would be met, and that the preparation of the Master Plan for the city of Djibouti would be launched. At mid-term, the Bank was to conduct, jointly with ADDS, an evaluation of the project and assess the likelihood of achieving the project development objective. There is no documentation available to substantiate that such an evaluation occurred. c. M&E Utilization: Some data were collected, but with much delay and commitment to the process was generally lacking. In the last results framework (updated June 17, 2014), information was not available for key indicators. Most aide-memoires prepared by supervision missions did not even include an updated results framework. In general, data were not readily available and there was no evaluation for decision-making purposes. ADDS was expected to carry out a social impact study before project completion. A socio-economic study was carried out to provide the baseline for the second project, but it did not include an impact analysis of the project under review as had been planned. M&E Quality Rating: Modest 11. Other Issues a. Safeguards: At appraisal the project was classified as category 'C' under OP/BP 4.01 Environmental Assessment since it was not expected to have any negative impacts and would not involve involuntary resettlement. However, at the approval of additional financing the project was reclassified as category 'B' to conform with a more stringent definition of classifications that had been introduced. In the event, no safeguard policies were triggered (ICR page 6). However, the Bank recommended that a fair agreement be negotiated with a vendor whose kiosk had been displaced for the construction of the sports field. An agreement was reached, but its implementation was delayed for reasons not explained in the ICR. b. Fiduciary Compliance: Financial management was rated moderately satisfactory by supervision missions, because of submission of financial statements without certification by the auditor, and delays in submitting audit reports. Procurement, however, was seriously delayed because of multiple errors and irregularities. Ex-post reviews of procurement were carried out in September 2011 and September 2012. This is despite the fact that procurement had been flagged as a high risk during appraisal because of lack of experience of the project staff and the poor quality of record keeping. Measures to mitigate the risks included training, the introduction of software tools for procurement management and the preparation of an operational manual. In the event these were insufficient. c. Unintended Impacts (positive or negative): None d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development Significant Significant Outcome: Bank Performance: Moderately Moderately Unsatisfactory Unsatisfactory Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Four have been adapted from the ICR's lessons: Strategic decisions to ensure the IDA allocation for a country is fully utilized can have negative results if the consequence is a project with very limited time for preparation and appraisal. Street surfacing using cobblestones is substantially more labor intensive than the usual road asphalting technique. Before repeating this technique a thorough cost-benefit analysis comparing the two approaches should be carried out taking into account the benefits of greater employment and the costs of future maintenance. When maintenance of infrastructure is consistently neglected the Bank needs to take a strong stance and assist the Government to prepare an action plan for providing systematic maintenance of investment already in place in the sector and to mobilize donors to support its implementation. The Bank should play a leadership role in encouraging cofinancing donors to try to work out common procedures for project implementation when a program is prepared. 14. Assessment Recommended? Yes No Why? Given the performance and predilection for repeater projects. 15. Comments on Quality of ICR: For the most part the ICR is well prepared and candid in its findings, but the section on efficacy is rather lop-sided, It needs to include the income disparity within Q7 as an issue and does not fully lay out the case as to why a moderately unsatisfactory rating is appropriate for outcome. Similarly, the sections on Borrower performance (Government and implementing agency) could have contained more information. For efficiency, the ICR should have simply said that no economic analysis was carried out and why. The other studies cited have no relevance. a.Quality of ICR Rating : Satisfactory