Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) Report Number: ICRR0021736 1. Project Data Project ID Project Name P145391 Studies for Sustainable Flood Mgmt. Country Practice Area(Lead) Philippines Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-17736 31-Dec-2015 3,091,046.70 Bank Approval Date Closing Date (Actual) 12-Jan-2015 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 7,000,000.00 7,000,000.00 Revised Commitment 3,200,000.00 3,200,000.00 Actual 3,091,046.70 3,091,046.70 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan J. W. van Holst Christopher David Nelson IEGSD (Unit 4) Krishnamani Pellekaan 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The PDO as stated in the Grant Agreement (Schedule 1, page 5) and the Project Appraisal Document (PAD, page 6) was: Page 1 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) "To prepare priority projects that aim to improve flood management and resilience in the greater Metro Manila Area". The original PDO referred to preparing two project proposals for structural measures that are technically, economically, socially and environmentally sound and ready for appraisal. The revised PDO as stated in the restructuring paper dated February 17, 2016 (page 3) was: "To prepare a priority project that aims to improve flood management and resilience in the greater metro Manila Area". b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 18-Feb-2016 c. Will a split evaluation be undertaken? No d. Components There were two components (PAD, paragraphs 28 and 31). 1. Preparation of feasibility and design studies for priority flood management infrastructure. Appraisal estimate US$6.80 million. Actual cost US$3.29 million. The actual cost was lower than the appraisal estimate due to the reduction in project scope. (discussed below). There were two sub- components: preparing feasibility and design studies, including preparing environmental and social safeguard instruments for: (i) a high dam in the upper Marikina River catchment area; and (ii) flood plain developments along the shore of Laguna de Bay. The scope of this component was reduced with the project restructuring on February 17, 2016. The activity associated with design of flood plain developments along the shore of Laguna de Bay was cancelled as the government decided to pursue this activity through Public - Private Partnership (PPP) arrangements. Grant funds for this activity were cancelled (discussed in section 2e of this review). 2. Project management and administration. Appraisal estimate US$0.20 million. Actual cost US$0.00 million. This component aimed at capacity building of the Department of Public Works and Highways (DPWH) for project management. The actual cost of this component was zero as the DPWH did not use Page 2 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) funds from the grant for capacity building. The costs for for project management in DPWH were absorbed by the regular budget of DPWH. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. Appraisal estimate US$7.00 million. The Bank task team clarified that the information in the Data Sheet in the ICR regarding a Borrower contribution of US$1.2 million was incorrect (see footnote in Annex 3 of the ICR). No borrower contribution was planned at appraisal. Actual project cost was US$3.20 million. Project financing. The original project cost of US7.00 million was wholly financed by the Australia - World Bank Philippines Development Trust Fund. Following the cancellation of US$3.80 million the actual amount disbursed from the trust fund was US$3.20 million. Borrower contribution. As noted already, none was planned at appraisal. The Data Sheet in the ICR indicates that the government contributed US$1.20 million at appraisal. The Bank team clarified that this was a system error in the portal and that there was no separate borrower contribution during implementation. Dates. The project approved on January 12, 2015, became effective on February 2, 2015 and was scheduled to close on December 31, 2015. Restructuring. Amendments were made to the project through the Level 1 restructuring on February 17, 2015 as follows; (i) US$3.80 million of the grant was cancelled for activities that had not yet started and were no longer aligned with the current priorities of the Department of Foreign Affairs and Trade: (ii) the project scope was reduced given the change in the PDO and target indicators were adjusted; (iii) the covenant on establishing resettlement activities on the flood plain along the shore of Laguna de Bay was removed as this activity costing US$1.8 million was to be pursued using a Public-Private Partnership (PPP) arrangements and the remaining activities in the PDO were not expected to entail resettlement; (iv) the government would finance the remaining $2 million of the cancelled amount to conduct social and environmental studies for both the dam and flood protection works on the Marikina River; and (v) the closing date was extended by two years and ten months in order to complete all the studies. Page 3 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) Split rating. As in the ICR (paragraph 29), this assessment also does not use a split rating of objectives, given that there was no disbursement before restructuring. In line with the ICR, this assessment is based on the revised PDO. 3. Relevance of Objectives Rationale Country context. Typhoons and related flooding occur regularly in Philippines, with an average of 20 typhoons affecting the country per year. Many areas in the Greater Metro Manila Area (defined as Metro Manila and adjacent areas of Laguna de Bay), are designated as flood prone, with insufficient protection against frequent inundation. Given that Metro Manila contributes about 35% of Gross Domestic Product of the Philippines economy and is one of the most populated cities in the world with a population of around 15 million, improving the capacity for managing floods was important in the country context. Government Strategy. The PDO was highly relevant to the government strategy. The updated Philippine Development Plan for the 2011-2016 period, prioritized disaster risk reduction and climate change as themes that underlie various sectors. The Flood Management Master Plan for the Greater Metro Manila Area identified a set of priorities for safety control and managing flood events in the area, through structural and non-structural measures and institutional improvements. The government also enacted legislation, the National Disaster Risk Reduction and Management Act in 2010, aimed at shifting the focus from emergency response to disaster prevention. The project was well-aligned with the Philippine Development Plan for 2017-2022. A pillar of this plan aimed at reducing inequality through reducing the vulnerability of individuals and families due to natural hazards. Bank strategy. The PDO was well-aligned with the Country Partnership Strategy (CPS) for 2015-2018 (extended to 2019). The fourth engagement area of the CPS underscored the need for building resilience to climate change, environment and disaster risk management through (i) increasing resilience to natural disaster and climate change impacts: and (ii) improved natural resource management and sustainable development. Original and Revised PDO. The original PDO was rated as High. The revised PDO was a subset of the original PDO. It reflected the activities that were supported by the project, following the cancellation of part of the grant. Therefore, the relevance of the revised PDO to the Bank and government strategy is also rated as High. Rating Relevance TBL Page 4 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To prepare a priority project that aims to improve flood management and resilience in the greater metro Manila Area. Rationale Theory of change: Preparing feasibility and design studies (including the environmental and social safeguard instruments) for the dam in the upper Marikina River were intended for preparing priority projects that are technically, economically, socially and environmentally sound and ready for appraisal by the government and international development partners. Implementing these priority projects was expected to contribute to the development objective of improving flood management and resilience in the Greater Metro Manila Area. Outputs.  According to the ICR (paragraph 30), a feasibility study for interventions for the dam in the upper Marikina River was completed as targeted with seven main sections: (i) description of possible options and reasons for selecting the preferred option (large dam); (ii) preliminary design of the selected option; (iii) study on possible use of reservoir water for secondary purposes; (iv) description of safeguard issues; (v) economic and financial analysis; (vi) implementation plan for construction of the structure; and (vii) possible impact of climate change. The study was reviewed by the Bank task team, Project Steering Committee members, specialists from the Japan International Cooperation Agency and the Bank Panel of Experts on dams, and eventually approved by the Project Steering Committee. Following the completion of the study, the design study of the dam was completed with aspects such as, project cost estimates, construction schedule and operation and maintenance requirements. The tender documents and tender drawings were also prepared. Relevant documents for appraisal were submitted to the National Economic and Development Authority's Investment Coordination Committee (NEDA-ICC). The ICR also notes that ICC appraisal and clearance is a pre- condition for securing loans by national government agencies. The ICR expected that the clearance process would be completed by the third quarter of 2019 (paras 30 and 31). The Bank team clarified that there has been good progress and that formal project preparation is expected to start soon.  A draft Environmental and Social Impact Assessment (ESIA) of the dam was completed as targeted by the Department of Public Works and Highways. The ESIA was submitted to the Department Page 5 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) of Environment and Natural Resources as required in Philippines for obtaining an Environmental Compliance Certificate (ICR, para 32)  An Indigenous Peoples Plan (IPP) was envisioned when the dam was designed since the dam was to be located on ancestral land. The ICR notes that the IPP was being prepared when the project closed and is expected to be completed by the end of 2019 (ICR, para 33).  An evaluation of the Australia-World Bank Strategic Partnership acknowledged that the grant made an important contribution to capacity building within the Department of Public Works and Highways (ICR, paragraph 39). Outcomes (paragraph 34).  A priority project plan was prepared as targeted to mitigate the flooding risks in the Pasig-Marikina River Basin as planned. The dam design prepared was technically, socially and environmentally sound with an estimated economic internal rate of return of 20%. The dam when constructed was expected to benefit at least three million people living downstream in Metro Manila (ICR, para 34). Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The studies produced from this project were endorsed by a range of expert reviews and are expected to assist the government in general and the Department of Public Works and Highways and the Metro Manila Development Authority in particular, in addressing flood management and resilience in Metro Manila. Overall Efficacy Rating Substantial 5. Efficiency Economic analysis. The PAD (paragraph 47) notes that an economic and financial analysis was not undertaken for the studies financed under the project. There was also no economic and financial analysis of the project in the ICR. Page 6 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) Administrative and Operational Issues. There were considerable implementation delays because the procurement of consultants took much longer than expected. The scope of the project was reduced by half and yet the project closing date had to be extended by almost three years. While there were also delays in payments to consultants, payments were made after project closure (but during the grace period permissible under Bank procedures). This review agrees with the ICR that the efficiency with which this project was implemented is modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the revised PDO to the government and Bank strategy is rated as High. Efficacy of the single PDO - to prepare a priority project for improving flood management and resilience in the greater metro Manila Area - is rated as Substantial, as the expected outcome was realized. Efficiency is Modest, given that there were obvious inefficiencies in the project’s administration during implementation. In summary there were moderate shortcomings in the project’s achievements and its overall outcome is therefore rated moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Page 7 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) Social risks. There are social risks associated with complaints by population and non-governmental organizations about dam development, especially upstream of a city with 15 million people and near an earthquake fault. A good communication strategy and plan and comprehensive stakeholder engagement with downstream population, indigenous people and Non-Governmental Organizations, is hence necessary before the construction of the dam. Institutional risk. There are risks associated with the limited capacity of both the Department of Public Works and Highways and the contractor to execute dam construction works. As the ICR (paragraph 66) notes, it will be necessary to carry out pre-qualification of contractors, using stringent technical and financial qualification criteria (ICR, paragraph 66). Technical risk. As the ICR notes, the proposed large dam is among the most critical infrastructure investments to achieve sustainable flood management in Greater Manila. There is, however, a risk that dam construction will not be in line with the designs and recommendations from this project. Metro Manila will require additional water sources for domestic consumption. As the Marikina River is close to the city and is a possible source for domestic water supplies, there is the risk that government allows the private sector to build and manage the dam in that basin and use it for domestic water supply, rather than use it for managing floods (paragraph 67). 8. Assessment of Bank Performance a. Quality-at-Entry The design of technical assistance activities, which focused only on studies, was simple and straightforward (ICR, paragraph 44). The project design built on the recommendations of the Master Plan prepared by the government, with support from the Bank. The ICR (paragraph 60) notes that the task team that supervised the preparation of the Master Plan also prepared this project. The preparation team included members with expertise in fiduciary and safeguards aspects. The implementation arrangements were appropriate, with the Project Management Office - the Flood Control Unit of the Department of Public Works and Housing, in charge of day-to-day project management. Several risks were identified at appraisal, including substantial risk associated with social and environmental issues, given that 50 - 60,000 households living in the flood plain of Laguna de Bay were likely to be affected by project interventions (discussed in section 10a of this review). Mitigation measures included engaging key civil society leaders and an information campaign for stakeholders. The arrangements made at appraisal for Monitoring and Evaluation (discussed in section 9) and safeguards and fiduciary compliance were appropriate (discussed in section 10a and b). Page 8 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) The Bank preparation team underestimated the risks associated with the implementation capacity of the Department of Public Works and Highways, that contributed to delays during implementation. Quality-at-Entry Rating Satisfactory b. Quality of supervision Seven Implementation Status Results Reports were filed over a four-year implementation period (implying twice a year supervision missions). These missions were supplemented by short term review and support missions. The continuity of leadership was maintained, with the same Task Team Leader (TTL) through the implementation period (ICR, paragraph 48). Support was provided by the team, both through training sessions and regular interaction with the counterpart agencies, for strengthening the capacity of the Department of Public Works and Highways (ICR, paragraph 39). The supervision team provided technical guidance on matters such as ensuring that enough attention was paid to issues such as, the impact of the probable maximum flood on dam stability, that inundation mapping of downstream areas was carried out properly and that there were sufficient criteria to select the option that would have the least social and environmental impacts. The task team organized workshops and training sessions and provided on-the-job in financial and procurement management and this aided in fiduciary compliance (discussed in section 10b). The timing of the restructuring of the project was appropriate, with the project restructured immediately after the confirmation of the extended completion date of the parent Trust Fund. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The key outcomes of the project - proposals and detailed designs for structural measures that technically, economically, socially and environmentally sound and that are ready for appraisal - were appropriate. The indicator could easily be measured without an elaborate M&E system in place. There were no indicators for monitoring activities associated with "strengthening capacity of Department of Public Works and Highways". Page 9 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) b. M&E Implementation Following project restructuring, the indicators were revised to reflect the reduction in project scope and an intermediate indicator was added for monitoring the capacity building activities of the project. The ICR (paragraph 50) notes that there was no need for continuous monitoring, as the PDO outcome indicator could only be achieved at project closure. c. M&E Utilization The M&E was used to monitor the performance of consultants and assess the achievement of the objective. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as a Category A project. Seven safeguard policies were triggered at appraisal: Environmental Assessment (OP 4.01); Natural Habitats (OP 4.04): Safety of Dams (OP 4.37); Physical Cultural Resources (OP 4.11); Forests (OP 4.36): Indigenous People (OP 4.12); and Involuntary Resettlement (OP 4.12)) (PAD, paragraphs 52 and 58). Environmental Assessment, Natural Habitats, Safety of Dams, Physical Cultural Resources and Forests. Although the project only financed studies, these safeguards were triggered, as the studies were expected to lead to investments, that could have adverse environmental impacts (PAD, paragraph 59). The PAD (paragraph 60) notes than an Environmental and Social Impact Assessment (ESIA) was to be conducted and an Environmental and Social Management Plan (ESMP) prepared during implementation. The ICR (paragraph 53) notes that an ESIA was conducted and ESMP prepared during implementation. The ICR does not report of any non-compliance with respect to environmental issues during implementation. Page 10 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) Indigenous People and Involuntary Resettlement. These social safeguards were triggered, under the assumption that investments when implemented, could cause temporary and permanent physical and economic displacement of people occupying the plans (PAD, paragraph 53 and 54). The PAD (paragraph 5) notes that a Resettlement Policy Framework and a Resettlement Action Plan was to be prepared during implementation. The main social risk related to the resettlement of people from the flood plain of Laguna de Bay during implementation. With the removal of this activity, the social risks were minimized. The ICR does not report on any unexpected potential impacts during the design of the dam. b. Fiduciary Compliance Financial management. An assessment of the financial management capacity of the Department of Public Works and Highways (DPWH) was conducted at appraisal. DPWH had been implementing both Technical Assistance and lending operations of the Bank and was familiar with Bank procedures. The assessment concluded that the financial management arrangements were satisfactory. The ICR (paragraph 59) notes that there were delays in processing of payments by the Department of Public Works and Highways, with many payments done after project closure (but within the four month-grace period). The ICR does not indicate whether the audits were qualified or not. Procurement. An assessment of the procurement capacity of DPWH, concluded that the procurement risk was moderate (PAD, paragraph 51). The ICR (paragraph 58) notes that there were no procurement issues during implementation and that procurement issues were addressed as per Bank procedures and guidelines. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Page 11 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) Quality of ICR --- Substantial 12. Lessons The ICR (pages 19-20) draws the following main lessons from the experience of implementing this project, with some adaptation of language. (1) Capacity building activities should start as early as possible during implementation for the Bank team to play a significant role in assisting the implementing agency. This is particularly so when a government agency starts to implement a Bank-financed grant for the first time. Although the Department of Public Works and Highways had implemented Bank-financed projects, the managing unit for this study had no experience with Bank-funded projects and grants and it took time for the managing agency to become familiar with the financial, procurement and disbursement arrangements of the Bank. (2) Continuity in Bank leadership can aid in long-term engagement with the clients. This project was prepared by the same Bank team that assisted in the preparation of the government's Master plan. This continuity allowed the team to have a precise understanding of the country and the sector context for this project. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well-written and relatively concise at 20 pages. It provides a candid description of the issues encountered during implementation (such as those pertaining to the administrative details of the Trust Fund which eventually led to the reduction in project scope). The ICR draws reasonably good lessons from the experience of implementing this project. It also appropriately chooses not to do a split rating of PDOs, given that there was no disbursement when the project was restructured. There was a lack of clarity regarding the borrower contribution. This issue was, however, clarified by an explanation from the Bank team. a. Quality of ICR Rating Page 12 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Studies for Sustainable Flood Mgmt. (P145391) Substantial Page 13 of 13