ICRR 13151 Report Number : ICRR13151 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/15/2009 PROJ ID : P069491 Appraisal Actual Project Name : Lgu Urban Water And Project Costs (US$M): US$M ): 35.27 8.45 Sanitation Project Apl2 Country : Philippines Loan/ US$M): Loan /Credit (US$M): 30.00 7.48 Sector Board : WAT Cofinancing (US$M ): US$M): 0 0 Sector (s): Water supply (70%) Sanitation (20%) Sub-national government administration (6%) Flood protection (4%) Theme (s): Other financial and private sector development (25% - P) Municipal finance (25% - P) Other urban development (24% - P) Municipal governance and institution building (13% - S) Public expenditure financial management and procurement (13% - S) L/C Number : L7080 Board Approval Date : 10/18/2001 Partners involved : Closing Date : 11/30/2006 11/30/2008 Evaluator : Panel Reviewer : Group Manager : Group : Emily O'Sullivan Ridley Nelson IEGSE ICR Reviews IEGSG 2. Project Objectives and Components: a. Objectives: The project development objective (PDO) of APL2, used for this review, and as stated in the loan agreement, was to assist selected local government units (LGUs) to: (i) provide sustainable water and sanitation services; and (ii) strengthen their institutional and technical capacity for planning, budgeting and financial management of local services, including for water supply and sanitation . The ICR notes the inconsistency in the project development objectives and uses the Loan Agreement PDO in evaluating the project. The key performance indicators (KPIs) were officially revised. The project was restructured at the request of Government due to slow implementation. In the restructuring, the target values for the key performance and outcome indicators were officially lowered as follows : (i) KPI 1 was lowered from 70% to 35% of cases of conflict resolved without conflict, (ii) KPI 3 was lowered from 50% of households without sanitation facilities to 10%, (iii) KPI 6 regarding the performance in LGUs with constructed systems was added in order to monitor the users ’ satisfaction, but no targets values were provided, (iv) the intermediate outcome indicator of the water supply component for water supply systems successfully constructed was lowered from 40 to 11, and (v) the outcome indicator of additional towns with lease and management contracts by the end of APL 2 was lowered from 30 to 5. These modifications were approved by the Board on July 11, 2005. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 07/11/2005 c. Components (or Key Conditions in the case of DPLs, as appropriate): Water Supply Program (Appraisal: US$19.84 million; Actual: US$6.99 million) The purpose of the component was to finance civil works, equipment and supervision for improved water supply systems in Local Government Units (LGUs), water utilities with private operators and regional water utilities, and for the rehabilitation and expansion of existing systems. The constructed systems would be operated through leasing and management contracts with the private sector operators. Consultancy (Appraisal: US$1.65 million; Actual: US$1.46 million) The purpose of the component was to finance construction supervision consultant ($1.50 million) to assist the Development Bank of the Philippines (DBP)-Project Management Office (PMO) with construction management as well as individual consultants ($0.15 million) for hiring specialized consultants to support the LGUs and government agencies in implementing their WSS and drainage components. Sanitation Program (Appraisal: US$1.67 million; Actual: n/a) Component dropped. The purpose of the component was to finance physical improvements in household toilets, on -site sanitation facilities, including soak -away pits for septic tank effluents or the disposal of wastewater flows arising from augmented water supplies hygiene education . Funds would be on-lent to the households and be administered by the water supply operators and recovered from the water bills on an installment basis . Funding was cancelled after it was determined that households preferred to privately finance household sanitation improvements. Drainage Program (Appraisal: US$1.11 million; Actual: n/a) Component dropped. The purpose of the component was to provide financing for investments and consultant services in micro -drainage infrastructure. The selection of towns would be demand-driven except where necessary to mitigate adverse impacts of augmented water supply systems . Because the subprojects did not have any adverse impacts and the Environmental Management Plans (EMPs) did not require any drainage, there was no demand for financing . On-Lending to Private Operators (Appraisal: US$4.00 million; Actual: n/a) Component dropped. On-lending was cancelled after project restructuring . Land Acquisition (Appraisal: US$2.60 million; Actual: n/a) Component dropped. Land acquisition was not required by the project and was cancelled after project restructuring . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Many of the subprojects intended for financing by the project were cancelled . The civil works component with the highest allocation fell short of its target of constructing 11 LGU water systems, undertaking only nine with the other two still to be completed. As a result, this component accounted for only US$ 6.38 million or 48.93% of the estimated cost after loan cancellation . Financing: The loan was reduced by 50 percent from US$30 million equivalent to US$15 million and corresponding key performance and output indicators were reduced when the project was restructured on July 1, 2005. The final amount disbursed was US$7.48 million, or 24.9 percent of the appraisal value . Borrower Contribution: Appraisal, US$5.27 million; Actual, US$0.97 million. The Borrower contribution was reduced along with the Bank's in line with the restructuring of the project . Dates: The Project was extended for two years on a phased basis to give more time to complete approved and on-going subprojects. The first extension (original loan closing to September 30, 2007) allowed approved LGUs to sign loan agreements and bid out subprojects before June 30, 2007 while the second extension (October 2007 to November 2008) permitted them to complete construction and start operation . 3. Relevance of Objectives & Design: Relevance of Objectives: Substantial . The objectives of the second phase of the project remained relevant to both the country's conditions and the Bank's strategy . The most recent Country Assistance Strategy, CAS, (2010-2012) highlights the need for continued support from the Bank to "increase access to better public services for the poor by deepening the reform agendas in key public services sectors and expanding basic service delivery directly to the poor." In line with the Bank’s country governance and anti -corruption framework (CGAC), which supports more capable and accountable government at the national, local, and agency level in public financial management, procurement and decentralization, the objective to strengthen institutional capacity also remains relevant . The CAS confirms the continued commitment of the country and the Bank to promoting private sector partnerships in service delivery. Relevance of Design: Modest . APL2 was intended to build on the experiences of APL 1 particularly with regard to subproject implementation, and sustain the momentum of private sector interest in the construction and operation of water supply systems through lease and design -build-lease (DBL) contracts. While the project objectives remained the same, the design was restructured as discussed in Section 2.a. above. As a result of the restructuring in 2005, the following changes were made to the components : (i) funding for the sanitation and urban drainage components was cancelled, and (ii) funding for water supply was reduced along with the number of target LGUs . With the cancellation of the sanitation component, the project design failed to consider the objective to provide sustainable sanitation services and the restructuring did not include an amendment to the objective . The design of APL2 failed to identify the risks associated with the implementation as experienced under APL 1 and incorporate appropriate mitigating measures such as providing / requiring more active supervision, better risk management and more efforts to strengthen the capacity of the LGUs . The demand-driven approach of the project did however allow for flexibility in design to meet the needs of the LGUs and water users, though it was not entirely successful in this regard . Overall, project relevance is Modest . 4. Achievement of Objectives (Efficacy): The original PDO was not revised; however, since the key outcome targets were formally revised, ratings for the achievement of the two parts of the objective will be provided based on both the original and revised targets . (i) Provide sustainable water and sanitation services . Low . Revised : Modest . Original : Low. Based on the original target values for achieving this part of the objective, the targets were not met . Whereas the project originally intended to include 40 LGUs, only nine water supply systems were successfully completed . This also fell short of the revised target of 11 LGUs, though by a lesser margin. Thirty additional lease and management contracts were supposed to be signed by project close, however there were only five . The revised target for new leases was met. The project did meet design parameters for completed systems, which fulfilled the original target value of 100 percent. Sanitation services were not improved by the project . The original target value was to improve sanitation facilities in at least 50 percent of households without sanitation . The sanitation component was dropped from the project due to a lack of demand, however at revision the associated objective was not formally dropped from the PDO. The Contract Administration Unit (CAU)/regulatory body was expected to resolve conflicts between LGUs/customers and operators without resort to contract cancellations in at least 70 percent of private sector participation contracts. The revised target was 35 percent. Both were met with 78 percent retention; however, this percentage does not convey the small number of contracts involved - two of nine contracts were terminated . The target for consumer satisfaction - 80 percent - was the same after revision and was exceeded at 87 percent satisfaction with the water service . However, 60% reported the price of water delivery “heavy" which is not represented in the results framework . There were no target values associated with the performance indicators for service to low income households and performance of the systems, therefore an assessment of these is difficult . (ii) Strengthen their institutional and technical capacity for planning, budgeting and financial management of local services, including for water supply and sanitation . Original : Modest . Revised : Modest . Local Government Unit (LGU) capacity was strengthened in areas of planning and project implementation . In the 95 LGUs where feasibility studies were conducted, important knowledge has been gained regarding how to expand water supply systems. Further, these LGUs engaged with their communities on issues of service coverage and cost recovery thus gaining a better understanding of constituent ’s demands. While APL2 only produced nine LGUs with completed water supply systems, 36 more were financed by outside sources that had benefited from the project ’s feasibility studies. Private sector participation was a major centerpiece of this project; the project provided incentives compatible with the respective interests of the LGU, consumers and operator . Out of the 11 subprojects provided with funding, five involved private operators under mainly a design -build-lease contract. While some improvement has been made with regard to private sector involvement, problems remain related to : poor pre-qualification standards for contracting private operators; and, complexity of transaction and contractual documents exceeding the experience and capacity of LGUs leaving them at a disadvantage . The project allowed exposure of LGUs to borrowing from a government financial institution (GFI) as opposed to using their own resources in funding development projects, as well as to new contractual arrangements involving outsourcing of services to the private sector . Where private operators have been contracted, LGU capacity in monitoring and regulating services has been developed through their involvement in the Contract Administration Unit, but needs to be further strengthened . However, in many cases capacity gaps at the local level which remained throughout the project adversely affected the ability of the LGUs to effectively engage with the private sector in service delivery. Finally, because the second PDO did not have outcome indicators in the original or revised design, a valid assessment of its success is difficult for both . 5. Efficiency (not applicable to DPLs): Efficiency was Substantial . There was no economic rate of return (ERR) computed at appraisal for the project as a whole since it was a program loan where specific investments were not known in advance . However, an ex-ante ERR was estimated using feasibility studies for 11 subprojects available at the time of appraisal and this averaged 19% considering direct use benefits only (with water sales used as proxy ). The ex-post ERR was based on five of the nine completed and fully operational subprojects . ERRs were not calculated for the other four completed subprojects due to a lack of completed data. The ERR averaged 12% considering only direct use benefits . The lower ERR is attributable to the following (a) higher investment costs mainly due to failed wells, (b) low consumption levels in the initial years of operation, and (c) high operation and maintenance costs . The systems had been operating less than two years at the time the ICR was prepared, therefore it is likely that consumption will improve with time . Cost effectiveness was also analyzed for all nine subprojects . The average investment cost per capita was US$51.35, which is lower than the appraisal estimate from the 11 sample projects of US$64.31. At the time the ICR was written, operators were not remitting lease fees to the LGUs (except in one case) leading the LGUs to subsidize debt service . This situation has since changed and payments are being made . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 19.2% 0% ICR estimate Yes 12% 45% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project outcome was Moderately Unsatisfactory (see below for weighted rating calculation ). While the project objectives were relevant to the needs of the country and the Bank strategy, the design and implementation of the project were inadequate to fully realize the objectives resulting in only modest relevance . The revision of targets midway through the project created a more viable project based on demand from the local government units for water supply operations which contributed to a moderately satisfactory rating post -restructuring. Efficiency was substantial for the restructured targets, however it was based on only a portion of the subprojects . Though the program was cancelled after APL 2, support continues from government financial institutions and private financial institutions for creditworthy water utilities . The Bank continues involvement through support to less creditworthy utilities. The Local Government Support for Regional Water Supply Project (P096822), currently pipelined, is designed to bring public water service providers to creditworthy standing by improving financial and operational performance. Weighted rating: US$7.48 million disbursed total. US$2.29 million had been disbursed at restructuring . US$5.16 million disbursed after restructuring . Disbursement ratio before restructuring : 2.29/7.48 = .306 Disbursement ratio after restructuring : 5.16/7.48 = 0.690 Original PDO/targets: Unsatisfactory (value:2), 2*0.306=0 0.612 Revised PDO/targets: Moderately Satisfactory (value:4), 4*0.690=22.76 Weighted average score: 0.612+2.76=33.37 Rounded to the nearest whole number, 3, the project's weighted rating is Moderately Unsatisfactory . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to maintaining the Development Outcomes is rated High . The capacity of the LGUs is not sufficient since many have not undergone professional training to plan for and manage their water supply and sanitation systems . As mentioned, a regulatory framework was not developed to mitigate problems related to the transfer of systems to private operators and to monitor lease contracts to ensure obligations were met . Extended periods of low water consumption may affect financial sustainability . Seasonal farmers, for example, may experience a temporary lack of income necessary to pay monthly bills thus motivating a reduction in usage . One of the most critical risks to project sustainability is the disruption caused by changes in political leadership at national and local levels that can affect the use of private sector partnerships and result in LGUs not honoring previously agreed to institutional arrangements . Short-term decisions adopted to ensure election (e.g. delaying tariff increases) often thwart long-term results, as experienced by the project . Related is the ability of consumers to pay for the piped water and the time it may take for users to adjust to the introduction of water pricing . High rates of disconnection were seen during the project . These were in part attributable to inadequate assessment of demand through the Willingness-to-connect survey, taken during project preparation, which did not involve careful consideration of reasons for connecting or disconnecting . a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: Bank performance was Moderately Unsatisfactory . Quality at Entry: The preparation of APL2 was successful in supporting the Government ’s strategy for the sector and developing a market for water supply systems with the government financial institution and private financial institutions. However, as discussed under relevance of design, the Bank failed to fully incorporate the experiences of APL1 in the design of APL2 in large part because of its preparation several years prior to the close of the first phase. This resulted in the reappearance of problems in the second phase . In addition, the PDO was presented differently in the PAD and Loan Agreement by the Bank , and an M&E system was not established to fully assess the achievements of the project in terms of its development objectives . Further, the target number of LGUs to provide assistance is not made clear in the PAD . Only the ICR specifies the target of 40 LGUs. Supervision: Although the Bank country office provided all of the necessary fiduciary support required for the successful implementation of the project, had a Mid -term Review been conducted many problems may have been avoided. The MTR could have presented a consistent PDO, eliminated the sanitation element of the first PDO, and developed KPIs to measure progress regarding the second PDO . In addition, the project task team leader changed three times during implementation making continuity and coordination more difficult . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Unsatisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: The Government Performance was Moderately Unsatisfactory . At the national level, the Government performance was moderately unsatisfactory . A major factor affecting outcomes was the lack of sustained capacity building of LGUs as asset owners and private operators . The Department of Interior and Local Government (DILG) was responsible for oversight of LGUs . However, it proved unable to provide leadership and capacity building due to its own lack of institutional and technical capacity, compounded by inadequate financial resources. However, DILG remains committed to water sector policy reform at the local level . There was an absence of a functioning national level policy steering committee to represent the government as well as national oversight agencies to monitor progress on sector reforms . While an inter-agency Technical Working Group (TWG) was formed, it was not effective in filtering information to policy makers responsible for sector oversight and policy formulation. It is noted in the ICR that this group rarely met . However, the Government's commitment to the project objectives was evidenced in part by Executive Order No . 279 setting a framework for financing the sector with the associated reforms supported by the Department of Finance and National Economic Development Authority. The initiative of the central Government, started under the project, led the Government Financial Institutions (Government banks) to lend to LGUs and also reduced the roles of sectoral agencies (LWUA, DILG) as financiers. At the local level, LGU performance was moderately unsatisfactory . The political dynamics at the local level led to frequent changes in LGU decisions at various stages even when commitments had been made . Further, the lack of progress in developing an effective regulatory framework to take the place of the Contract Administration Unit (CAU) will delay the realization of the full benefits from the program and the projects . The Implementing Agency performance was Moderately Unsatisfactory . For a period of 18 months, implementation was adversely affected in terms of quality of construction and facilitating funding of subprojects when the construction supervision consultant (CSC) was replaced by the implementing agency (DBP) in order to reduce costs. The reinstatement of the original CSC was a condition for extending the project ’s loan closing date. While DBP successfully established a Project Management Office, its operation was compromised by the agency ’ s decision to replace CSC. The DBP-PMO was unable to meet the demands of project monitoring, management of loan disbursement, and program marketing . This situation was compounded by a national reorganization of the DBP midway through the project. However, it is notable that the DBP recognized it could not achieve the original targets of the project development objective and outcome, and requested project restructuring which the Bank approved. a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: Design: Modest . The monitoring and evaluation framework, as presented in the Program Design Summary of the PAD included a number of key performance indicators (KPIs) which provided output and outcome -oriented information for the first objective . However, there were no indicators for the second objective and even when the project was restructured new KPIs were not introduced to correct this . There were no baselines provided for performance indicators and two indicators did not have target values making effective evaluation difficult . The design (in the PAD) also failed to include the central output of the project – to assist 40 LGUs in addressing water supply services – in the results framework. Implementation: Negligible . The project failed to strengthen the interim regulator on dispute resolution, to contract monitoring and enforcement, and to institute a national level steering committee to monitor and evaluate results at the sector level. No Mid-term Review was conducted. A household survey was conducted in five municipalities to measure achievement of performance indicators . The ICR notes that the results of the Willingness -to-connect (WTC) survey taken during project preparation to determine acceptability of the water systems and water tariffs may have inadequately assessed demand, at least in part resulting in high rates of disconnection seen during the project . Utilization: Negligible . The ICR provides little evidence of the use of M&E to inform decisions . In the case of the WTC survey, it is noted that a lack of coordination between the social survey team and the technical /design team resulted in the installation of pipes in some areas with no nearby households . As previously discussed, the same survey did not involve a study of the myriad reasons for wanting to connect or disconnect . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards: The Project complied with the Bank ’s safeguards policies. Procedures and arrangements to ensure that the environmental safeguards were in accordance with Operational Policy (OP) 4.01 were included in the updated Operations Manual and were complied with . On the social safeguards, OP 4.12 was not triggered because there were no cases of involuntary resettlement in the subprojects funded by the Project . Fiduciary compliance: The financial management (FM) system of the Project was undertaken in accordance with the project design and loan agreement, except for the planned computerization of the project ’s books of accounts. Annual audited financial statements for the project were not always submitted on time . 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development High High Outcome : Bank Performance : Moderately Moderately Unsatisfactory Unsatisfactory Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Drawing partly from the ICR lessons, with some amendments the three main lessons are : Capacity building for local government units on contracting for services from the private sector is critical for the urban water sector. In this case, in areas where there has been a positive experience with private operators, a contributing factor to success was the capacity of local government (officials) to understand the intricacies of the institutional arrangements. This capacity can be developed by programs sponsored by DILG or by having senior officials in the LGUs with relevant experience in law, engineering, or private sector to help them to understand the technical aspects of the project . Disruptions caused by local and national elections and the associated need to provide capacity building for newly elected officials are important aspects to consider in project design . Time needs to be invested “up front� to build consensus at the national and local levels regarding critical policy and institutional issues – such as the use of private sector participation and the importance of establishing an effective regulatory framework . Financial sustainability of new water supply operations can be adversely affected where measurements of demand for piped water are not based on full cost recovery . In this case, the project's willingness -to-connect survey did not adequately capture user's ability to pay and significant disconnections resulted . 14. Assessment Recommended? Yes No Why? This project may be of interest to the IEG Water Study . Though the program was cancelled, its goals are still expected to be achieved with government financing, other donors, and new Bank support . With results of the project at high risk for sustainability, assessment of the project should be useful in the planning of future projects with similar involvement of the private sector at the local level . 15. Comments on Quality of ICR: The ICR discusses the project with candor, presenting available evidence in a clear and concise manner . The outcomes, and shortcomings, are well related to the lessons . The document could have been improved with more data related to project monitoring and evaluation as this section was brief . In addition, a better explanation for the discrepancies between the Loan Agreement and the Project Appraisal Document would have been useful . a.Quality of ICR Rating : Satisfactory