Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) Report Number: ICRR0021893 1. Project Data Project ID Project Name P111034 Jakarta Urgent Flood Mitigation Project Country Practice Area(Lead) Indonesia Urban, Resilience and Land L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-81210 31-Mar-2017 91,072,221.62 Bank Approval Date Closing Date (Actual) 17-Jan-2012 28-Feb-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 139,640,000.00 0.00 Revised Commitment 91,072,221.62 0.00 Actual 91,072,221.62 0.00 Prepared by Reviewed by ICR Review Coordinator Group Cynthia Nunez-Ollero J. W. van Holst Christopher David Nelson IEGSD (Unit 4) Pellekaan 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement (FA, p.6) and the Project Appraisal Document (PAD, paragraph 13) the Project Development Objective (PDO) for the Jakarta Urgent Flood Mitigation Project (JUFMP) was "to contribute to the improvement of the operation and maintenance (O&M) of priority sections of Jakarta’s flood management system." This review will assess this singular PDO. Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components 1. Dredging, rehabilitation and flow capacity improvement of selected key floodways, canals and retention basins (US$176.1 million at appraisal, reduced to US$172.53 at the 2017 restructuring; US$111.79 million actual). This component financed the dredging and rehabilitation of 11 floodways/canals and four retention basins, and the transport and disposal of dredged material to proper disposal sites. This component also financed the replacement or repair of mechanical equipment such as pumps, gates, etc., as needed. At the March 31, 2017 restructuring, exchange rate savings due to the depreciation of the Rupiah led to (a) allocation of US$28.7 million from this component to finance the construction of a powerful pumping station at Sentiong to withdraw water from drains in subsided areas where gravity was no longer sufficient; and (b) allocation of US$3 million to review the flood management master plan for the Cisadane- Jakarta-Bekasi (Greater Jakarta) area (see Section (2)(b) Financing below). 2. Technical assistance for project management, social safeguards, and capacity building (US$13.4 million at appraisal, US$6.11 million actual). This component financed contract management, engineering design reviews, construction supervision engineers for the dredging and rehabilitation works and technical assistance (TA) for implementation of the project, including the establishment of a Flood Management Information System (FMIS), Resettlement Policy Framework (RPF), Resettlement Plans (RPs), a Grievance Redress System (GRS) and a Panel of Experts (POE) to advise on technical aspects such as dredging and dredge disposal, environmental and social management monitoring, and resettlement practices. This POE was later dropped during the 2017 restructuring because of adequate support from the Bank's Task Team in Jakarta and the project's Construction Supervision Consultant (ICR, paragraph 18). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total project cost at appraisal was reached US$189.85 million. The actual cost was US$ 117.91 million. Financing: The International Bank for Reconstruction and Development (IBRD) financed this project with a loan of US$139.64 million of which US$91.07 million was disbursed. Loan savings resulting from foreign exchange gains due to the depreciation of the Indonesian Rupiah to the US Dollar were to finance additional activities in this project but were not implemented. Consequently, US$48.5 million of the loan was cancelled (see ICR Data Sheet and paragraphs 19 and 46). Borrower Contribution: The Government committed US$50.2 million and disbursed US$26.83 million at closing. Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) Dates. The project was approved on January 17, 2012 and made effective on August 8, 2012. The Mid Term Review was conducted on February 2, 2016. The project was originally scheduled to close on March 31, 2017 but was extended to close on February 28, 2019. Restructuring: On March 31, 2017 a Level 2 restructuring resulted in the following:  Replaced the original 3 PDO outcome indicators because there was no useful baseline data available. The revised outcome indicators (see below) were aligned with the 2013 flood mapping and data formats of the Disaster Management Agency of DKI Jakarta (BPBD DKI Jakarta). The 3 new outcome indicators better measured flood mitigation results to the project, especially after taking into account the large physical changes in Jakarta due to land subsidence. Project Development Original Outcome Indicators Revised Outcome Indicators Objectives (PAD) (ICR) Water level of inundated areas Average flood depth (baseline Reduced flood water level (baseline of 70 cm, target 11 cm) 100-150 cm, target <20 cm) Number of hours of water logged Average length of inundation Reduced duration of inundation in inundated areas (baseline, >24 (baseline, >7 days or 168 hours, hours, target <24 hours) target 0.25 days or 6 hours) Extent of inundated areas in 57 Extent of inundated areas in 34 Reduced area of inundation kelurahans(baseline 100 percent, kelurahans (baseline, 100 target 20 percent) percent, target 15 percent)  Administrative changes o designated the National Research Center for Water Resources (PUSAIR) of the Ministry of Public Works and Highways (MPWH) as an additional Project Implementation Unit (PIU) to conduct the master plan review. This review was further confirmed by the task team in its November 25, 2019 email. o added incremental operating costs for the MPWH to support PUSAIR as a PIU o removed the need for an advisory panel of experts because of the adequacy of technical support from the Construction Supervision Consultants and the Bank's Jakarta-based team o reallocated costs among disbursement categories  Identified the equivalent of loan savings of US$44 million at the time of the Level 2 restructuring resulting from foreign exchange gains from the depreciation of the Indonesia Rupiah against the US$ and consequently adding two new key activities to component 1 (ICR, paragraph 16), amended components and associated costs, amended the results framework, and added intermediate outcome indicators corresponding to the following two new activities: o "additional drainage pumping capacity added by the project to reflect (expected) results of the Sentiong PS construction" o reduction of the end target value for "additional storage added to retention basins to exclude emergency dredging works completed under Daerah Khusus Ibukota (DKI or Special Capital Region) Jakarta's own financing to increase retention storage volumes following the January 2013 floods" (ICR, paragraph 17).  Extended the project closing date by 23 months to complete remaining project activities and implement 2 new activities financed by loan savings (see Financing above). Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) The above changes improved the project’s prospects for achieving its objective without changing the PDO or reducing the project’s level of ambition. Therefore, there was no reason for a split evaluation of outcomes. 3. Relevance of Objectives Rationale The project development objective (PDO) was relevant to the country's national development priorities. Evidence of the government's priority focus on disaster risk management was the Daerah Khusus Ibukota (DKI or Special Capital Region) 2019 Provincial Government Budget, which allocated Rp500 billion (US$35.5 million) for land acquisition around rivers and water channels to ease water flows from upstream areas and improve water capacity. Another Rp350 billion (US$24.9 million) were allocated to acquire land surrounding reservoirs, lakes, and retention basins. The DKI also pledged to clean the rivers and seas through the National Action Plan on Marine Debris (2017-2025) (ICR, paragraph 21). However, the PDO (to contribute to the O&M of the flood management system in priority sections of Jakarta) was a vague and modest response to a complex problem posed by significant economic losses from severe flooding caused by drainage failure rather than precipitation (ICR, paragraph 3). The PDO’s modest approach was only one of many actions that the city was undertaking under its DKI Medium Term Development Plan (2007-2012). The complex landscape of flood management involved a number of provincial and national government institutions (ICR paragraph 49) as well as factors such as uncontrolled land subsidence (gradual settling or sinking of the soil surface due to subsurface movement of earth materials) and inability of aquifers to recharge due to the vast impermeable concrete cover of the city (ICR, paragraph 5). Surging land subsidence, for example, meant that flood management could no longer rely on gravity to empty overflowing rivers and waterways to the sea. This problem brought to the fore the need for pumping and not just enhanced O&M such as restoring dredging capacities. When loan savings resulted from the devaluation of the Rupiah, two sub-components were identified to strengthen the project's relevance to Jakarta's flood management strategy. The first was the proposed powerful Sentiong pumping station designed to withdraw water from the drains in areas of Jakarta where gravity was no longer sufficient due to severe land subsidence. The second was the review of the flood management master plan in Cisadane-Jakarta-Bekasi area by updating the flood models and synchronizing the flood management works with the latest infrastructure details, topography maps, land use plans, land subsidence data, and coastal flood defense plans (ICR, paragraph 22). The project team added new information on November 25, 2019: that "DKI Jakarta has implemented new polder schemes (pumped instead of gravity-based systems) in North Jakarta (although unclear how this scheme affected the project), and that the National Capital Integrated Coastal Development (NCICD) initiative was underway to strengthen existing seawalls and identify new pumping stations..." The PDO was relevant to the World Bank's Country Partnership Framework (CPF) FY 2016-2019 for Indonesia. Flood risk management was a priority under Engagement Area 1: Infrastructure Platforms at the National Level, which focused on water, sanitation, irrigation, and dams. Infrastructure improvements to reduce flood risk and strengthen disaster mitigation were evident in the project’s dredging and embankment rehabilitation works. And yet, improved O&M alone as stated in the PDO, was clearly a modest response to Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) a sustainable flood control strategy. Pumps (as DKI Jakarta implemented in North Jakarta) evidently needed to be part of a contribution to solving to the problems posed by flooding. Conclusion: On its own, traditional O&M was not ambitious enough to address the complex flood management system challenges of Greater Jakarta. The development problems brought about by the city's vulnerability to coastal flooding and land subsidence because of rapid urbanization and population pressure were clearly stated, but the PDO "to contribute to the improvement of the O&M of priority sections of Jakarta's flood management system" was originally vague on what improvement meant and how the project would contribute to improving the O&M in the context of the complex land subsidence problem without a pumping station as part of an overall O&M approach to Jakarta's drainage problems. The relevance of the project's objective was therefore rated modest. Rating Relevance TBL Rating Modest 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective "to contribute to the improvement of the operation and maintenance (O&M) of priority sections of Jakarta’s flood management system." Rationale Theory of Change: The project’s key activities - the dredging, rehabilitation, and flow capacity improvement of selected floodways, together with technical assistance to improve institutional capacity - were expected to contribute to the improvement of the operations and maintenance (O&M) of Jakarta’s drainage system and reduce flooding impacts for some 1.8 million people living in kelurahans (urban villages or blocks) within DKI Jakarta (ICR, paragraph 12). Improved O&M would be measured by three outcome indicators that were introduced in a level 2 restructuring on March 31, 2017 using 2013 flood mapping data from Badan Penanggulangan Bencana Daerah (BPBD or Provincial/Regional Disaster Management Agency). There was no significant change in the project's ambition as a result of the restructuring. However, the performance of these activities in the project area were not linked to how they improved the O&M in Jakarta’s flood management system. The need for a pumping station and review of the master plan were key components that were expected to contribute to improving the O&M of Jakarta’s flood management system. These two key components were added but, in the event, not implemented by the project. OUTPUTS:  3,428,277 cubic meters of dredged material removed from the floodways, canals, and retention basins, including solid waste (baseline 0, original target 3,400,000 cubic meters, target exceeded). Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) This indicator measured the volume of dredged material removed. It indicated the restoration of floodways, canals, or retention basis to original capacities.  80,500 cubic meters of additional storage added to retention basins (baseline 0, original target 596,740 cubic meters, revised target to 80,500 cubic meters, target achieved). This indicator measured the extent to which flood water retention volume increased in the 4 retention ponds. The original target appeared to have been reduced because the Government financed DKI Jakarta to dredge 265,117 cubic meters in these retention ponds following the January 2013 floods. This was not counted toward the Government's counterpart contribution. Annex 1 in the ICR explained that there was no clear documentation regarding the reduction of the target value and that this indicator was already achieved by November 21, 2016.  55.66 km of embankment was repaired or constructed (baseline 0, original target 42.20 km, target exceeded). According to Annex 1 of the ICR, the Construction Supervision reports did not distinguish between repair of existing or construction of new embankment. In addition, a revised target of 43.50 km was noted in the December 2018 Implementation Status Report but not in the 2017 Restructuring Paper that supported the level 2 restructuring.  After the 2017 restructuring the Sentiong-Ancol pumping station was to be constructed and designed to pump 40 cubic meter per second, but the station was not built because procurement issues could not be resolved and hence, the target was not achieved.  24.60 km of canals were maintained or cleaned (baseline 11 km, original target 25 km, target achieved). This indicator measured the O&M of canals and reflected the extent to which DKI Jakarta's operational capacity improved its flood management maintenance. OUTCOMES: As previously noted, the project's objective was "to contribute to the improvement of the operations and maintenance of priority sections of Jakarta's flood management system." The following outcomes contributed to achieving the project's objective. 1. Reduced Flood Water Level. The average flood depth in the project’s 34 flood prone kelurahans during the most significant flood event of the financial year (not originally defined) fell from the 2013 baseline of 100-150 cm to 28 cm in 2018, and the target of 20 cm was 92% met or almost achieved. The lower average flood depth suggested that after significant rainfall (assumed to cause flooding), areas are no more than ankle-deep (as opposed to 1 to 1.5m deep without the project) in flood water, caused less physical damage or reduced economic losses. According to the ICR, there was no documentation on how the value was derived. Raw data was reported in the MTR and no further recording of raw data between the February 2016 MTR and project closing of February 2019. The actual value was reported as range-based data from 26 measurement points across East and South Jakarta. The reporting of only one actual value questioned how this was measured, and whether one value represented the conditions in all kelurahans. The target of <20 cm was missed by 8 cm but a significant 92.4 percent reduction (original baseline of 70 cm, revised to 100-150 cm, original target was 20 cm, target almost achieved). 2. Reduced Duration of Flooding. The average length (duration) of inundation in the project’s 34 flood prone kelurahans during the most significant flood event of the financial year, fell from a baseline of more than 7 days to 6.7 hours, essentially meeting the target of six hours in 2018. This indicator suggests that after significant rainfall, the 34 kelurahans were flooded for a much shorter period, leading to lower economic and social losses. Average length of inundation in 34 flood prone kelurahans during the most significant flood event of the FY (unclear which fiscal year was referred to) Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) measured the reduction in duration of flood water inundation. Rainfall was presumed to be the predominant cause of flooding. As mentioned in the previous outcome indicator, because of the variation in rainfall, the mismatch between the flood mapping locations of the known project-affected kelurahans and the unknown catchment boundaries, it was not possible to attribute a reduction in the duration of flooding to project funded works. Achieved 0.28 day or 6.72 hours (baseline, more than 7 days or 168 hours, original target was less than 0.25 day or 6 hours, target achieved). 3. Reduced Area of Inundation. 80 percent of the neighborhoods in the 34 kelurahans experienced inundation during the most significant flood event of the financial year (baseline 100 percent or all 34 kelurahans, revised target of 15 percent, target presumed not achieved). The ICR acknowledged that there was no reliable data to indicate whether this target was achieved or not (ICR, paragraph 28). In summary, the efficacy of the project was substantial based on the revised PDO indicators in place of the ambiguous, unmeasurable and discarded PDO indicators (ICR, paragraph 30). The volume of dredged material removed from the floodways, canals, and retention basins and additional storage added to retention basins were achieved. The length of embankment repaired or constructed was exceeded. However, capacity to withdraw water from underground drains by the proposed Sentiong Pumping Station was not achieved. The task team clarified in an email to IEG on January 23, 2020 that the following technical issues delayed its procurement and would not be completed in time: (i) the specific location of the pumping station among three options a few hundred meters apart; (ii) the hydrological relationship with the existing Ancol pumping station located nearby; (iii) the intended pumping capacity; and (iv) ensuring a no-regrets approach for alignment with the emerging long-term plan for investments under the National Capital Integrated Coastal Development (NCICD) initiative. These four issues were closely interrelated. The resulting consensus among the responsible PIU (the Regional Office of the Ciliwung-Cisadane River Basin, BBWSCC), other parts of the Directorate General of Water Resources at the Ministry of Public Works and Housing (MPWH), DKI Jakarta, and the Bank’s task team required revisions to the original detailed engineering design, joint review, and confirmation among the concerned agencies. The review of the master plan on flood management in the Cisadane-Jakarta-Bekasi area, meant to support the long-term strategy for flood and water management in the Greater Jakarta area, was also not implemented because of unresolved technical issues and procurement delays. This review was instead undertaken using central government budget support. There was no stand alone technical assistance or training activities to build the O&M capacity of the agencies involved in Jakarta’s flood management system, such as the Provincial Government of DKI Jakarta, and the Directorate General for Water Resources, the Directorate General for Human Settlements, the Regional Office of Ciliwung-Cisadane River Basin, the National Research Center for Water Resources, and the Research and Development Agency, all under the Ministry of Public Works and Housing in the areas of improved coordination and adoption of the Resettlement Policy Framework (ICR, paragraph 35). Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) Rationale The ICR (paragraphs 30 to 35) provided considerable (though mainly anecdotal) evidence of the project's impact on flood water levels, duration of flooding, and areas of inundation. However, the ICR did not explicitly lay out the extent to which the better performance of the project area was attributable to the improvement in the O&M system in Jakarta's flood management system for the priority areas. Nevertheless, according to the project team in its November 25, 2019 email to IEG, O&M was included in the budgets and work plans of agencies responsible for improving O&M in the long run although they did not adopt any formal mechanisms, policy, or program specifically geared towards O&M that could be attributed specifically to the project (ICR, paragraph 68). Overall there is substantial evidence that considerable investments were made by this project towards improving drainage of flood water and to the improvement of O&M in the project area. This review concluded that, while the specific future impacts of the project investments on flood control were not assessed, this project made a substantial "contribution to the improvement of the operation and maintenance of priority sections of Jakarta's flood management system" - namely to the project's weak objective. Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency: The methodology used for the project's cost benefit analysis at appraisal was replicated at project closing and reached the conclusions below (ICR, Annex 8, Table 1 replicated below): At At Project Indicators Appraisal Closing Project Cost (in US$ million) 189 166 NPV (in US$ million) 3,109 2,786 Economic Internal Rate of Return (EIRR) 381 413 (in %) Benefit/Cost Ratio 36 37 Table 3 in Annex 8 of the ICR summarized the economic analysis for three different flood cycle scenarios. Note that the project achieved a more modest EIRR of 29 percent based on a more realistic flood cycle (given recent experience) of 10 years, still greater than the assumed 8 percent discount rate. Note also that the flood management system was originally designed for a historical 25 year cycle (ICR, paragraph 3). Flood Cycle (5 Flood Cycle (10 Flood Cycle (20 Key Indicators years) years) years) Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) NPV (US$ million) 1,072 416 91 EIRR (%) 389 29 11 Benefit/Cost Ratio 14% 8 5 At appraisal, benefits from an improved O&M in the flood management system were derived from (i) avoiding to reallocate the national budget for specific flood related purposes such as rehabilitating schools, roads, government offices, or parks, and post flood health treatment; (ii) tax revenue losses from reclamation land taxes (i.e., with improved O&M, flood incidence or inundation would be reduced, reclamation would proceed, land taxes collected); incremental land taxes from flood free areas, electricity taxes, and from preventing the disruption of economic activity (i.e., improved O&M in flood management system would lead to efficient flood management and avoid lengthy disruptions in economic activities from flooding); and (iii) indirect economic losses (e.g., school days losses, work day losses, social and health losses) such as those experienced following the 2007 Jakarta floods. The benefit cost analysis at appraisal assumed a 2007 flood event every two years and a 40 percent contribution to flood mitigation. Using an unexplained 8 percent discount rate, the PAD estimated net present value (NPV) of US$3.1 billion and an economic internal rate of return (EIRR) of 381 percent. The extraordinarily high rate of return was caused by defining the benefits of investments on improved O&M as the avoidance of high costs of repairing the destroyed infrastructure, steep reduction of numerous sources of private and public revenues, and indirect losses such as reduced work days of workers living in flooded areas (ICR, Annex 8 provides details). The shorter the associated cycle of potential flooding the shorter was the assumed period during the high avoided costs and losses were incurred and hence the higher were the benefits and rates of return to investment in O&M. Adopting the same methodology used at appraisal, the ICR added benefits from loan saving, but removed tax revenues from the sales of the conversion of surface to drinking water since they never materialized. The ICR estimated that the EIRR at closing was 413 percent with an NPV of US$2.79 billion. A sensitivity analysis was conducted to review the project's economic performance if less frequent flood cycles, such as 10 and 20 year return periods were used following the PAD's use of the 2007 floods. If similar floods occurred every 10 years, the estimated EIRR of 29 percent was still much higher than the 8 percent discount rate used in the estimates of the NPVs, suggesting that the project would generate significant economic benefits over its lifetime. Clearly the EIRRs calculated with different time horizons are not comparable, although they serve some purpose in highlighting the value of urgent action on flood mitigation in Jakarta if more than usual heavy rainfall is anticipated in future because of the impact of climate change - irrespective of the cause of the subsidence of ground levels and the clogged underground drainage. Administrative and Operational Efficiency: Procurement and implementation delays, and opportunity costs because of these delays reduced administrative efficiency. Eighteen to 36 month delays in contracting construction packages reduced the project's NPV. The Government of Indonesia incurred a commitment fee on the undisbursed loan amount and remained unadjusted even after the cancellation of US$48.5 million. There were opportunity costs associated with the unrealized construction of the Sentiong PS and the review of the flood management master plan even when project restructuring added two years for implementation. The project was completed at about 62% of the total appraisal cost, and benefited 1,144 (96 percent of the revised target of 1,197) beneficiary households (ICR, paragraph 87). The PAD had estimated that 1,109 households would benefit from the project. The Government's commitment to the project was negatively affected by divergent views on flood management and resettlement from four governors who came into office between preparation (2009) and closing (2019). For Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) example, a new DKI administration in 2012 reviewed ongoing activities and formulated its own flood management policies and resettlement plans, which formed part of the revised O&M strategy. Project implementation slowed due to the review of activities and the learning curve required by frequent staff and management turnover (ICR, paragraphs 58, 62). There were also funding constraints during the early years of implementation that resulted in a lack of staff to complete and submit monitoring reports and documents such as procurement plan, interim unaudited financial reports, in a timely manner. The share of the first component to total project costs at appraisal (US$176.45 million /US$189.85 million = 92.9 percent) and at closing (US$111.79 million/US$117.91 million = 94.8 percent) provide the coverage/scope noted in the table below. Note that the time horizon for the EIRR at appraisal was two years and at the ICR (selected by this review from those calculated in the ICR) ten years Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 92.90 Appraisal  381.00  Not Applicable 94.80 ICR Estimate  29.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the project's objective was assessed by this review as modest because the objective, while clearly stated, namely a "contribution to the improvement of the operation and maintenance of priority sections of Jakarta's flood management system", it focused only on a segment of Jakarta, and was not ambitious enough to address the complex problems such as risk from flooding in an area with pressing environmental and social conditions. It was nevertheless clear that improved O&M without the Sentiong pumping station, the relevance of the project's objective to Jakarta's complex flood management challenges would inevitably be modest. The efficacy of the project's achievement towards its modest objective was rated substantial despite ambiguity in methodology and baselines to support the measurement of the outcomes achieved - such as reduced flood levels, reduced duration of inundation, and reduced area of inundation. Although there were minor shortcomings in the project's administrative efficiency, operational efficiency of the project remained substantial. The project's overall outcome was therefore rated moderately satisfactory. Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome According to the ICR, the following posed risks to the development outcome:  Institutional and Governance Risks - To address these risks, the project adopted an effective framework that reduced bureaucratic obstacles such that each agency was mandated with clear roles and responsibilities to avoid the overlap of functions (ICR, paragraph 57). o Balai Besar Wilayah Sungai Cilwung Cisadane (BBWSCC or River Basin Management Authority of Ciliwung Cisadane Rivers) was in charge of major floodways. o The Directorate General Cipta Karya (DGCK or Human Settlements Directorate General under the Ministry of Public Works and Highways) was in charge of critical national institutions and canals of national importance. o The Dinas Pekerjaan Umum (DPU or Public Works Department) of Daerah Khusus Ibukota (DKI or Special Capital Region of Jakarta) was in charge of the remaining drains, floodways, retention basins, and all social safeguards. DPU DKI Jakarta had the most important and intensive O&M responsibility, including O&M for pump operations and dredging, for the floodways, canals and retention basins (ICR, footnote 32).  Formal coordination among the Construction Supervision Consultants, the various project implementation units, and contractors ceased upon loan closing (ICR, paragraph 61). However, close working relationship among the respective agency staff was anticipated to continue to facilitate communication across the various implementing agencies within Jakarta. In 2016, agencies like DKI Jakarta Environmental Agency (Dinas Lingkungan Hidup) and the Water Resource Agency (Dinas Sumber Daya Air or SDA) created special task forces to support the systematic maintenance needs of the Jakarta flood management system. These special task forces consisted of several thousands of personnel. Pasukan Orange ("orange troops") installed floating plastic pontoons to intercept garbage and maintain drains. Pasukan Biru ("blue troops") targeted areas that were threatened by flooding or help clean up after flooding. Jakarta adopted a medium to long term comprehensive and integrated strategic approach to slow land subsidence, improve infrastructure and proactively manage the risk from floods (ICR, paragraph 109) by targeting solid waste management, integrated water resource management, disposal of wastewater, abstraction of groundwater, provision of bulk water services, and coastal flood defense. The Government is continuing cleanup and dredging as well as reviewing its flood management master plan similar to the activity added during restructuring but not implemented. DKI Jakarta, the central government and private sector stakeholders are preparing a long term National Capital Integrated Coastal Development (NCICD) program that will include coastal protection infrastructure such as a sea wall and large scale land reclamation in Jakarta Bay (ICR, paragraph 102).  Financial Risks - DKI Jakarta adopted an integrated flood management by pursuing maintenance activities such as garbage collection and maintenance dredging. There is a risk that DKI Jakarta may not continue these maintenance operations regularly or effectively. That inaction would result in sedimentation, blockages, and reduce the city’s drainage capacity. To maintain and improve the cleaning of the canals and rivers, the central Government pledged US$1 billion a year toward cleaning rivers and seas. The Government also launched the National Action Plan on Marine Debris (2017-2025) to reduce the city’s plastic debris by 70 percent by 2025. The project team clarified in its Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) November 25, 2019 email that O&M activities were included in budgets and workplans for various government agencies involved in flood management.  Environmental Risks - Climate change, land subsidence, rapid urbanization, and migration pressures continue to pose challenges to flood management. Residents also continue to pollute Jakarta’s rivers and canals with solid waste and sewage. To mitigate this environmental risk, the Government would need to upgrade the open sewerage system, secure O&M budgets to maintain the waterways, and conduct an information campaign to convince the public about keeping the waterways clean. 8. Assessment of Bank Performance a. Quality-at-Entry The Bank identified two limited project activities that were designed to achieve the improvement of the O&M of the flood management system in certain sections of Jakarta. This project began as an emergency dredging initiative. It was prepared over four years and was renamed (Jakarta Urgent Flood Mitigation Project (Jakarta Emergency Dredging Initiative). The PDO was clearly stated but focused only in a segment of Jakarta, and was not ambitious enough to address complex problems such as risks from flooding in an area with pressing environmental and social conditions (e.g., sea level rise, solid waste management, groundwater abstraction, land subsidence, rapid urbanization, and migration). The limited PDO was due to the initial design of the project as an emergency response. The project components were lacking in sustaining outcomes because the capacity of the existing pumping stations was not addressed, considering the predicted subsidence of the area. The technical, financial, and economic aspects of the project were focused on the dredging and rehabilitating embankments to return to the original capacity of retention basins as proxy to improving O&M in flood risk management. Environmental and design aspects benefited from a pilot dredging project initiated in November 2008 by the Government of Netherlands (ICR, paragraph 90). Drone photography was used to assist in the preparation of the project Resettlement Policy Framework (RPF). The project included environmental and social safeguards in each of the eight construction packages and promoted resettlement that diverged from the Government of Indonesia’s usual resettlement practices (ICR, paragraph 54). After almost 1.5 years of discussion the Government adopted the RPF that was compensation based (ICR, paragraph 92). The PAD noted a high overall implementation risk rating (ICR, paragraph 59). The project engaged various levels of government agencies involved in flood management resulting in a central Project Management Unit and one central Project Implementation Unit and three Project Implementation Units. The implementation arrangement resulted in reducing bureaucratic hurdles (ICR, paragraph 57) and fostered ease in longer term coordination (ICR, paragraph 55). The project, however, had shortcomings in the design of the capacity building component of the project, not focused on how to improve O&M capacity of the implementing agencies and results framework indicator to point to achieving this aspect of its operations. Among the recommendations following project closing was the need to dedicate resources for capacity building. Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) The rating was Moderately Unsatisfactory because of significant shortcomings in Bank inputs and processes. This project was originally designed as an emergency response to the floods of 2007. The theory of change and component activities were not adjusted to address the complex problem of a sustained effective O&M. The baselines (using 2002 data, 10 years before Board approval), and PDO target outcome indicators were vague and not attributable to the project (ICR, paragraph 63). The M&E system was problematic (see Section 9 below). In addition, inadequate mitigating factors to address fiduciary risks such as procurement capacities of the implementing entities were borne out at implementation even though all eight contract packages were eventually completed with the extension of project closing. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision At implementation, the Bank was focused on development impact with team supervision missions twice a year. However, there was a high staff turnover from preparation to implementation both at the Bank (4 task managers) and the government (4 governors) sides. This required frequent review, readjustments, and allowances for learning curves on both sides. On a positive note, most of the project team were based in Jakarta. Supporting specialists also remained the same throughout the project. This facilitated closer interaction with the clients. Site supervision was adequately carried out by the Construction Supervision Consultant but its local supervision staff did not carry out effective site supervision because of lack of awareness and low regard for environmental issues (ICR, paragraph 94). The project experienced procurement delays (see Section 10 Other Issues (b) Fiduciary Compliance, Procurement, below) as well as the delayed resolution of audit findings (Section 10 (b) below on Financial Management). Evidence of minor shortcoming in supervising safeguard aspects of the project were based on pending resolution of remedial action when the ICR was completed (ICR, paragraph 59). Candor and quality of performance reporting were evident in the reporting of the likelihood of meeting outcomes in the Implementation Status and Results Reports (ISRs). Implementation Progress (IP) ratings were moderately unsatisfactory leading up to the MTR. The MTR itself was delayed by six months (ICR, paragraph 60). And the trend of Moderately Unsatisfactory in IP rating continued even after the restructuring and up to project closing (ICR, data sheet). The Bank team did not generate studies that may have helped broadcast the outcomes of the project activities, including the benefits from the use of a compensation-based resettlement policy. There were no technical studies that could help assess project impacts. Analysis during the Mid Term Review could also have been used to highlight the benefits from the activities added as a result of the loan savings - the review of the master plan and construction of the pumping station. The project could also have benefited from using the elements of 2012 FMIS1 to generate flood risk related models (ICR, paragraph 95) rather than relying solely on BPBD's 2013 flood mapping data. Quality of supervision was affected by implementation factors that were subject to the control of the government. Government commitment to the project was evidenced by the funding constraints during the early years of implementation that resulted in a lack of staff to complete and submit monitoring reports and documents such as procurement plan, interim unaudited financial reports, in a timely manner. There was Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) no additional information in the ICR regarding the unmet Government contributions. With regard to the Results Framework, the baselines and target indicators were revised because of continuing rapid physical changes in the project area. But these indicators retained their weaknesses in how the baselines were set, the values that were measured, and the attribution of results from the project interventions (ICR, paragraph 96). There was no clear indication of the transition arrangements for building on improved O&M supported by the dredging and rehabilitation of the embankments. The project team clarified in its November 25, 2019 email that "... the project helped all PIUs recognize the need for better O&M of the drainage system. In practice, O&M activities [were] included in the budgets and workplans for various government agencies. However, to our knowledge, based on interviews of the PIUs and former task team members during the ICR mission, there [was] no formal mechanism, policy, or program specifically geared towards O&M that directly resulted from experience working on the project." Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The Theory of Change and component activities were not adjusted to address the complex problem of a sustained effective O&M for urban flood management. The project team clarified that the project aimed to restore existing drainage capacities or "low hanging fruits" among multiple priorities without new infrastructure. The PDO was simple and narrowly focused, primarily because it was originally designed as an emergency response (ICR, paragraph 54). Two activities - construction and capacity building - would achieve the improvement in operations and maintenance of the flood management system in target areas in Jakarta. The outcome indicators were identified at appraisal as placeholders to be adjusted during the Mid Term Review (PAD, Annex 1, footnote 1). Indicators were vague such as "the most significant flood event of the year." This indicator did not include the area and time covered by the indicator and did not consider that significant flood events varied from year to year. This indicator was also defined as having been caused by significant rainfall but not by other possible reasons for flooding such as tide rise, sea level rise, or overtopping of river banks. Also, significant flood events were mapped in DKI in 2013 and were used as reference to design flood management infrastructure even if that year did not reflect historical highs or a significant rainfall event. During the ICR interviews, the 2002 baseline, a good 10 years before Board approval, was based on anecdotal and not on systematic data (none available then), which were considered unreliable. The 57 kelurahans used in the original Results Framework were administrative boundaries that were in or adjacent to the project sites and not the catchment boundaries of the project sites (ICR, paragraph 24). In the 2017 Restructuring the number of kelurahans was reduced to 34 (but only overlapped with the 9 of the original 57 and did not entirely cover the project area) and three replacement PDO outcome indicators measured the extent of reduced frequency and intensity of flooding within project areas as a proxy for improved O&M of the key sections of the Jakarta flood management system. However, there was no clear methodology to monitor project progress and no formal documentation on how the baseline and target values were derived (ICR, paragraph 67). No additional tools were identified to Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) evaluate or verify achievement of targets. The intermediate results indicators in the Results Framework measured dredging and rehabilitation activities, provided targets, but links to the outcomes (return to original design capacities, improve O&M) were weak. b. M&E Implementation The Directorate General for Water Resources (DGWR) under the Ministry of Public Works and Highways implemented project monitoring. The Construction Supervision Consultant helped collect data regularly, monitor construction progress, and report on achieving outcomes, as well as compliance with environmental and safeguard activities. The PIU DKI Jakarta, together with the municipal walikotas (mayors), monitored and evaluated the resettlement plans. The project areas (kelurahan administrative boundaries) were different from the monitored (hydrologic catchment) areas, and unclear whether values reported were accurate or representative of the catchment area. There was a lack of well distributed measurements of indicators such as flood depth and duration. Even with in-stream gauges and water level readings in some areas, data were not properly recorded. A single actual observation value was used for multiple kelurahans that had different characteristics. The project used official regularly generated flood maps from Badan Penanggulangan Bencana Daerah (BPBD or Provincial/Regional Disaster Management Agency) to measure the extent and duration of flooding to quantify the impact of the project. These maps, based on administrative units of neighborhoods, were updated during flood events and provided spatial information about flood occurrence and hence flood indicators. However, there were no data collected between 2016-2019 (ICR, paragraph 26). The ICR indicated that there there was a missed opportunity in revising the indicators that relied heavily on the BPBD 2013 flood maps rather than making use of the outputs from FMFS1 (ICR, para 97). Data were also provided by Badan Meteorologi, Klimatologi, dan Geofisika (BMKG or Meteorology, Climatology, and Geophysical Agency) but the revised outcome indicators remained vague (i) 'most significant flood event of the year' did not consider the spatial (area) or temporal (time) variation from rainfall as a source of flooding; and other concurrent events such as tides and sea level rise; (ii) the arbitrary use of the 2013 flood data as reference did not necessarily reflect historical highs or rainfall events for which flood management infrastructure should be designed; and (iii) original and revised baseline data were based on anecdotal experience during the 2002 and 2007 flood events not systematic flood data (ICR, paragraph 66). c. M&E Utilization The M&E reports informed project progress. Safeguards compliance and resolution of grievances benefited from the use of M&E data. Report on land subsidence led to the adding two new activities in the 2017 Restructuring. Design changes were incorporated following field conditions and feedback from stakeholders. This led to the reduction in the number of relocated households. The significant shortcomings in the M&E system’s design, baselines, methods for monitoring the indicators, and implementation contributed to a modest rating for M&E. The lack of comparable base maps and data could not establish whether the achievement of the targets based on upstream flood data was the result of the downstream project-financed dredging and embankment works. Additional tools, Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) such as surveys, could have tested the links in the results chain and better assessed the achievement of the objective. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as a Category A project because of significant adverse environmental impacts from the project’s dredging and construction/repair of embankments. The project triggered two policies - OP/BP 4.01 Environmental Assessment and OP/BP 4.12 Involuntary Resettlement. Sludge was handled, stored, stockpiled, transported, and disposed. Physical works could remove informal settlements. Indirect project impacts included traffic disruption, reduction in air quality due to dust and would odor emissions, increased noise levels, reduction in surface water and seawater quality among others. The project complied with environmental safeguard policies (ICR, paragraph 79). Environmental Impact Assessments (EIAs) and Environmental Management Plans (EMPs) followed both Bank and Government requirements (Analisis Mengenai Dampak Lingkungan, AMDALS or Environmental Impact Assessments) supplemented by project specific supplemental EIAs. The project complied with safeguards requirements regarding dredged materials and its transport including occupational health and safety measures. Disposal within Ancol CDF was an integral part of the project but overall reclamation in Ancol was unrelated to the project. Disposal sites were not financed through the project but were regularly monitored. No hazardous materials were found following sample tests. The project adhered to the Environmental and Social Management Framework (ESMF) (ICR, paragraph 59). Two hundred and fifty six complaints that were largely construction related inconveniences and damages were resolved under the Grievance Redress System (GRS) with the support of the Construction Supervision Consultant (CSC). About 90 percent of the complaints were addressed within 3 months and 99 recent within six months. All complaints were resolved within a one year period. A preliminary social assessments study was carried out in 2008 and social impact assessments in 2009- 2010 as part of the EIA process. Specific Focus Group Discussions (FGDs) were carried out at project sites in all 57 kelurahans to communicate the project, obtain more detailed environmental and social impacts and gauge support for the project from the affected communities. The Resettlement Policy Framework (RPF) clarified principles, procedures, and organizational arrangements that would apply to Resettlement Plans (locally known as Land Acquisition and Resettlement Action Plans or LARAPs) for seven project locations. The scope of the RPF had to be made clear because of sites or activities that could be incorrectly linked to the project. The project limited its works to priority sections, reduced rights of way to a minimum, and geared its detailed engineering designs to minimize impact. As a result, the project affected persons were reduced from 2,513 households at preparation to 672 households during implementation. Most construction packages steered away from having to resettle households and affected only 328 households, relocating 22 households to public low-cost rental apartments, 44 households to surrounding areas, and compensating 262 partially affected households. According to the ICR there was no detailed resettlement costs information (ICR, paragraph 87). Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) b. Fiduciary Compliance Financial Management. The Borrower maintained adequate financial management (ICR, paragraph 89). External audit reports, except for two, received unqualified opinions: (i) the FY 2014 audit due to an Rp 8.6 billion overpayment and internal control weaknesses, later resolved; and (ii) the FY 2018 audit due to the conduct of dredging with barge and disposal work that were not in accordance with the agreed methods and was supported by incomplete documentation, which remained unresolved despite Bank reminders. The FY 2019 audit that would cover 2 months ending on Feb 28, 2019 was not yet conducted by BPK by the time the ICR was prepared. The Bank was following up on its submission. Internal audits prior to the external audits were not prepared despite Bank recommendations. Procurement: Procurement processes under the project were carried out in accordance with Bank policy even though procurement capacity varied amongst the PIUs. Not all were experienced with managing large works contracts under International Competitive Bidding procedures. A long prequalification process (17 months in the case of the first batch of contracts) shortened construction periods. Using a post qualification process for the next batch of contracts shortened the procurement process to 8 months. Contracting the Construction Supervision Consultant was delayed by four months due to the PIUs’ internal approval process. The three PIUs did not have a clear chain of command between the consultants and the contractors. These procurement delays led to extending the loan closing date. Procurement issues also failed to execute the two activities (i.e., the pumping station and review of the flood management master plan) that were added at restructuring. Pusat Penelitian dan Pengembangan Sumber Daya Air (PUSAIR or National Research Center for Water Resources) decided to finance the master plan review through a central government budget allocation. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Moderately Bank Performance Unsatisfactory Unsatisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) 12. Lessons The ICR presented 7 lessons drawn from this project. Most of them were focused on specific characteristics of this flood mitigation project. Based on these lessons one broad lesson is presented below:  Establishing relevant PDO indicators with reliable baselines at project appraisal provide the basis for sound project management as well as rigorous monitoring and evaluation of project results In this flood mitigation project, the original PDO indicators were (a) water level of inundated areas; (b) the number of hours of water logging in inundated areas; and (c) the extent of inundated areas. These indicators were acknowledged at appraisal to be vague The PAD stated that "Performance indicators are expected to be reviewed and adjusted as necessary during Mid-Term Review" (Annex 1, page 20). There were no reliable baselines because they could not be accurately mapped to neighborhood units, and most importantly they were ten years out of date because land subsidence in the project area during those ten years had been far more rapid than anticipated with serious consequences for the veracity of old baselines. These shortcomings made the original PDO indicators problematic because they provided no measurable or substantive challenge. Indeed it was not until the Mid Term Review that the PDO indicators were amended to: (a) average flood depth; (b) average length of inundation; (c) and number of neighborhoods affected by floods, which could be attributed to the project and measurable. Nevertheless the challenges of obtaining accurate baseline data continued. As this review noted "There was a lack of well distributed measurements of indicators such as flood depth and duration. Even with in-stream gauges and water level readings in some areas, data were not properly recorded. A single actual observation value was used for multiple kelurahans that had different characteristics" (Section 9b). The general lesson is that sound appraisal and M&E methodology needs, among other things, to create timely and measurable PDO indicators with accurate baselines that would contribute to generating reliable data to guide implementing agencies to track the progress of implementation. The specific lesson from this project is that an investment in generating accurate baseline data at the time of appraisal would have improved implementation and M&E performance. 13. Assessment Recommended? No 14. Comments on Quality of ICR The report was concise and followed OPCS guidelines. The report provided a detailed overview of the project - from its start as an emergency response to a standalone project dedicated to a limited aspect of flood management (O&M improvement) of a specific area of a mega city affected by the environmental pressures from sea level rise, land subsidence, rapid urbanization, and migration. Credible sources were appropriately Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Jakarta Urgent Flood Mitigation Project (P111034) referenced throughout the report, including annexes and maps. Evidence of project impact and outcome were provided but not all of them supported the weak indicators. There was reference to the theory of change to justify the ratings reached. The quality of analysis was concise and linked evidence to the findings. The report was results oriented and highlighted how the activities informed the project outcome. The report focused on the consequence of the project, providing the complex interactions of the various aspects of a complicated flood management system supported by weak indicators. The report was internally consistent with various aspects of the project highlighting the impact of a poor results framework on efficacy, efficiency, M&E and Bank performance. The report was focused on results, Lessons were based on the evidence and analysis of the data generated by the project. A minor shortcoming was in relation to the completeness of data or information regarding the sufficiency of institutional, technical, and financial planning and management capacity for O&M flood infrastructure which could have been addressed as part of the risk to development outcome. a. Quality of ICR Rating Substantial Page 19 of 19