Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) Report Number : ICRR0020345 1. Project Data Project ID Project Name P101279 CO Solid Waste Management Program Projec Country Practice Area(Lead) Colombia Social, Urban, Rural and Resilience Global Practice L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-77420 31-Dec-2013 27,130,316.00 Bank Approval Date Closing Date (Actual) 04-Aug-2009 31-Dec-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 20,000,000.00 0.00 Revised Commitment 7,400,000.00 0.00 Actual 5,221,073.17 0.00 Prepared by Reviewed by ICR Review Coordinator Group Chaitri N. Hapugalle John R. Eriksson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) formulation in the Loan Agreement (LA, Schedule 1, page 5) was to support the implementation of the regional solid waste management component of the Borrower’s Program, which aims at improving the quality and coverage of integrated (1) solid waste management services in the territory of the Borrower. The PDO statement in the PAD is virtually identical but does not include the word “regional” in its reference to the solid waste management component (PAD p. 6). Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) IEG adopts the formulation of the PDO in the LA. Note*(1) "Integrated" refers to considerations of financial, environmental, and social sustainability. (ICR footnote 26, p. 5). b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components Actual costs for Component one and revised Component were based on updated numbers provided by the Region. Component 1: Development of and Investment in Solid Waste Disposal Systems. (US$20 million at appraisal; Actual US$ 4,389,564.36) (1.1) The construction of solid waste management infrastructure consisting of civil works, equipment and services in support of the rehabilitation, construction, and/or expansion of landfills and transfer stations. (1.2) Finance the rehabilitation and expansion of landfill sites that may currently be operating at sub sanitary levels. (1.3) Technical advisory services for the development of SWM Investments. Component 2: Institutional Strengthening in Solid Waste Management. (US$ 0 million at appraisal; Actual US$ 41,302.15) (2.1) Strengthening sector regulatory framework and capacity building. (2.2) Strengthening of SWM planning systems (though development of SWM plans and strategies). (2.3) National assessment of recycling markets and strategy framework. Component 3: Project Management (US$ 0 million at appraisal; Actual US$ 4,272,251) (3.1) The component would finance costs associated with project management including goods, works, services, and operating expenses associated with financial audits and midterm and end-project evaluations. (3.2) Reinforce technical, environmental, community outreach, communications, administrative, and fiduciary capacity within the. Ministry of Environment, Housing, and Regional Development (MAVDT). Revised Components Component 1: Development of and Investment in Solid Waste Disposal Systems. (US$16,510,000; Actual amount US$ 4,389,564.36) (1.1) Carrying out and supervising investments in SWM infrastructure consisting of civil works, equipment, and services in support of the rehabilitation, construction, and/or expansion of landfills, waste treatment systems in selected sites, and the closing of unsanitary open-air landfills. Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) (1.2) Provision and supervision of technical advisory services for the development of SWM investments including with respect to their regionalization, engineering designs, environmental impact assessments, social assessments, and other related studies. (1.3) Development and implementation of social inclusion, skills building, and entrepreneurial programs for waste pickers. Component 2: Institutional Strengthening in Solid Waste Management (US$1,430,000; Actual amount US$41,302.15) (2.1) Provision of advisory services to strengthen the technical, operational, and commercial capacity of SWM operators. (2.2) Carrying out studies to strengthen SWM at the national level. Component 3: Project Management (US$2,060,000); Actual amount US$ 4,272,251.130) (3.1). Support the Ministry of Housing, City and Territory (Ministerio de Vivienda Ciudad y Territorio MVCT) technical, environmental, communication, administrative, and fiduciary capacity in managing the administrative and financial aspects of the Project, through the financing of goods, consultants’ services, training, and operating costs, including the financing of project audits, evaluations, and other related project management activities. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Costs - The project cost and component table (ICR annex 1 pp 29-30) took into consideration only project costs based on the World Bank contribution. The Region confirmed the following disbursement rates but the final disbursement rate at project closure was not provided. The disbursement rates (2) at the time of restructuring undertaken in 2013 were US$ 4.56; US$ 5mn was disbursed during the second restructuring on 08/25/2014; and US$5 Mn disbursed during the third restructuring on 06/25/2015. Financing and Borrower Contribution - The Borrower’s contribution at appraisal was to be US$ 5.0 million but the actual contribution was US$0.90 million at project closure. The ICR did not provide disbursement rates, which were missing for the 2013, 2014 and 2015 restructurings. The Region was not able to provide disaggregated data (ICR annex pp 29-30, team meeting and final audited numbers provided by the Region). Dates- The first restructuring was approved on 09/09/2013. The closing date of the project was extended by 18 months to 06/30/2015. Due to slow implementation progress, the Bank requested a reimbursement of US$1.07m. A second restructuring was approved by the Bank on 08/25/2014. The key changes included a) changes to rectify design deficiencies in the project implementation arrangements, b) adjust scope of activities to more accurately respond to country demand, and c) reallocate funding so that components two and three would be financed by the Bank instead of the borrower. Moreover, a retroactive waiver to the withdrawal conditions was provided to resolve inapplicable conditions, modify project indicators and target values to simplify the results framework, clarify ambiguities and accurately Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) reflect activities financed under the project and their outcomes. (ICR pp.8-9). A third restructuring formalized a second extension, which was approved after the Borrower’s request to (i) extend the closing date of the Loan by six months, from 06/25/2015 to 12/31/2015. The total cumulative extension was 24 months and included a partial cancelation of US$ 12.6mn (ICR pp.8-9). Information provided by the team show that US$12.6 million was cancelled as of 05/13/2015. US$3,01 million was cancelled as of 08/23/2016. Note * (2) Disbursement of funds from the Bank to the special account 3. Relevance of Objectives & Design a. Relevance of Objectives Historically, the disposal of solid waste in Colombia has always been a municipal responsibility. Colombia's 1,122 municipalities produced approximately 28,800 tons of solid waste per day and was inadequately disposed in open-air dumps. Limited waste minimization, source separation, and recycling were concerns. Related to the not-in-my-backyard (NIMBY) factor, regionalization of municipalities with existing landfills was obstructed by local government and citizen groups. Vulnerable waste pickers who depended on open-air dumps for their livelihoods also lacked access to basic services (PAD pp. 1-3). The regionalization of solid waste management was a key priority for the Government of Colombia (GoC). Government initiatives had identified 130 potential regional landfill projects which were financed through public resources. The National Strategy for Infrastructure Development, Solid Waste Sector issued by the Department of National Planning (DNP) in 2014 prioritized improvement of regional landfills, expansion of the lifespan of sanitary landfills, and closure of open dumpsites. The Bank’s implementation assistance remained relevant to achieving the country's development goals. The Project was strategically aligned to support the World Bank Group’s Country Partnership Strategy (CPS) 2012-2016. The PDO reinforced a core strategic pillar; the Sustainable Growth with Enhanced Climate Change Resilience pillar. Solid waste final disposal was one of the means identified to improve sustainable urban development. (ICR p.19). Rating High b. Relevance of Design Both pre and post restructuring, the project included a clear statement of the PDO and was relevant for the challenges facing the sector. (ICR p.18). The PDO and its three sub-objectives (see Section 4 below) were to be achieved with; a) development of investments in solid waste disposal systems, b) institutional development and c) improved project management. (ICR pp.6-8). The expected intermediate results were the construction and operation of sanitary landfills, environmentally sustainable closure of open-air dumps cost recovery for final disposal in Bank financed landfills, formation of specialized utility companies and the transition to the Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) specialized operator model for solid waste disposal. (ICR p.5). However, the causal chain was of negligible to modest relevance. (ICR p.19). The original design and implementation were not relevant because the existing planning structure and subproject preparation capacity were not relevant to the unique challenges of solid waste management. The PDO indicator was weak and subsequently modified (see section 10). While two components supported institutional and capacity building elements related to program outcomes, the PDO did not reflect this linkage. (Team Interview). The 2014 restructuring addressed design deficiencies related to the inconsistency between Government and Bank and processes and to accommodate changes in demand in the sector due to increased private sector focus on medium size landfills. (See Section 8a on Bank Quality at Entry). However, the PDO remained “unchanged and achievable, due to the broader scope and regional approach of the sub-projects to be implemented.” (Report No: RES11773 p4). Under Component 1, eligible investments were expanded to include complementary investments in collection and treatment technologies and to cover investments in smaller landfills. Components 2 and 3 were no longer to be funded solely by the borrower. Component 2 included activities related to skill development for human resources, technical, and entrepreneurial strengthening for landfill operators and undertaking of studies on a national level, and Component 3 was focused on project management. (ICR p 19). The project changed the approach to regionalization of disposal services to incorporate more municipalities into regional systems. The project design re-focused on the unmet demand by building new regional landfills for medium and smaller landfills (Team Interview). Other changes included the expansion of the type of investments and reallocation of funds. (ICR pp. 7-9). “The restructured design was substantially relevant due to the following changes: the adoption of a planning and subproject execution process more adapted to the solid waste sector” (ICR p.19). The subprojects financed under the loan were not limited by a geographic department’s “PDA” (Departmental Plans for Water and Sanitation) budget quota and thus not competing with water and sanitation projects. The solid waste unit was more proactive in developing subprojects (ICR p.13), which “enabled the financing of investments in existing regional facilities with a variety of administrative models and sizes; improved management; and final disposal solutions for isolated municipalities when regional landfills were not the appropriate solution”. (ICR p.19). Overall, the relevance of the design was less than optimal to fully demonstrate the achievement of the PDO. While the modified PDO indicator (See M/E discussion) was overachieved, Bank funds had to be drawn to finance a consultancy to develop the plans in Ipiales and Grenada for social inclusion pilots. While MVCT assigned budget to works under way, no budget was available for institutional strengthening, capacity building for operators, nor for sector and regionalization studies, which were not executed (ICR p 19). The ability to draw on carbon financing in the event funding shortfalls was noted during the appraisal stage, but limited progress in tariff reform related to composting prevented the ability draw on carbon financing to cushion the shortfalls. (Team interview). Rating Modest 4. Achievement of Objectives (Efficacy) Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) PHEFFICACYTBL Objective 1 Objective The overall objective of the Project was to support the implementation of the regional solid waste management component of the Borrower’s Program which aimed at improving the quality and coverage of integrated solid waste management services in the territory of the Borrower. (LA, p.5) The overall objective is divided into three sub-objectives as shown below. To better assess the achievement of objectives, the PDO indicator was supplemented by outcome indicators developed ex-post specifically for the purposes of the ICR (ICR, p. 20). Sub-objective 1 focused on improving the quality of solid waste management services in the territory of the Borrower. Discussion in this section draws in part from the ICR (pp 20-22) Rationale For the PDO Indicator the target was for the solid waste of at least 400,000 beneficiaries to be adequately disposed of in sanitary landfills. Achievement: this target was overachieved with 683,547 people benefitting. This included investments completed in all the landfills reconstructed and upgraded and the populations of the 60 municipalities served by these landfills (ICR, p. 20). However, the indicator had shortcomings as discussed in the M&E section. Service quality was improved (in 6 of the 7 regional systems), benefitting 675,352 people through the construction of upgraded cells, treatment systems, and auxiliary infrastructure, ICR, p.22, note 58). Additional information on the improvement in the technical and environmental quality of final disposal is provided in the ICR (p.21). Capacity building for 6 operators (ICR p.22) allowed previously open dumps or landfills with operational deficiencies to operate at higher standards. Subprojects consisted of upgrading or reconstructing existing landfills rather than constructing greenfield sites. As of 2016 more municipalities ended up using landfills than anticipated, which explains why this sub-objective was achieved in spite of the low percentage of disbursement (26%). (ICR p.22, note 58). Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Sub-objective 2 focused on improving the coverage of solid waste management services in the territory of the Borrower. Rationale The project completed seven sanitary landfill subprojects that served 60 municipalities incorporating 33 municipalities into regional landfill systems, providing a population of 683,547 with adequate solid waste Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) services (ICR p.22 para 83). One target was that five landfills were to be expanded, optimized or upgraded. This was achieved (5) and 33 municipalities were incorporated into regional disposal systems. The latter represented an overachievement from a revised target of 15 municipalities and a baseline of zero. Increased service coverage was provided for 42 municipalities with a population of 465,795 through expanded capacity in six of seven systems (ICR, p. 20, note 54 and Annex 2). Sixteen municipalities benefited from an upgraded system that provided adequate disposal service not previously available. The upgraded sites also provided the opportunity for 26 municipalities that had previously used open dumps to join the regional systems. The number of inhabitants in 2016 that benefited from landfills with an increased useful life was 566,534. This resulted from construction of additional disposal capacity, namely, 1,089,000 additional tons in six landfills: Ipiales- 5 years, Guaviare 10 years, la Dorada 12 years, Aguadas 21 years, Granada 3 years and Quichia 18 year (ICR, p.20, note 55). Regarding waste collection, the results are not strictly comparable, but suggestive.(3) The target was an “increase in the coverage of proper final waste disposal (number of inhabitants and percent of waste collected).” The achievement was that 465,795 inhabitants produced one percent of waste collected in the country (see Section 10a below for further discussion). Before the investment was undertaken, four percent of the waste collected in the country was not disposed of in sanitary landfills (ICR, p.20). Note* (3) The region provided the following clarification. The indicator of reference relates to disposal rather than collection services. Colombia uses “ % of waste that is collected that is disposed in sanitary landfills” as their indicator for landfill service coverage. To allow for comparison, the average waste production per person in Colombia was multiplied by the population provided this service under the project to get the waste that is now due to the project investments, disposed in a sanitary manner. It was considered a more precise estimate than using the tons per day disposed in each of the new landfills as in some cases the population being served had already disposed in another sanitary landfill. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Sub-objective 3 focused on improving the coverage of an integrated system of solid waste management services in the territory of the Borrower, where “integrated” referred to financial, environmental, and social aspects (ICR, p. 22). Rationale The Project partially achieved the integrated systems of solid waste management (financial, environmental and social). Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) Financial-With the exception of Milaflores, the project succeeded in assuring financial sustainability through the application of the tariff system of 2015 in all the systems and the associated capacity building. However due to its size and lack of ability of the tariffs to cover costs, Milaflores had to rely partly on general municipality income to cover operational costs. According to a supplemental PDO outcome indicator reported by the ICR, the population (680,078) of six of the seven landfill sites was by 2016 served by disposal systems with adequate cost recovery (ICR, p.21). No target or baseline information was provided. Environment-The Project’s environmental outcomes included compliance with environmental assessment policies and improved operation but the operational improvements were limited only to the Project investments. Social- Five studies were completed for open-dump closure but had limited impact on the ground. One study led to a closure and one other closure was completed in parallel with the investments in Ipiales. Social inclusion plans were to incorporate informal recyclers in two landf ills (Ipiales and Granada), but due to lack of budget allocation and unclear legal mandates, the MVCT did not develop or implement these plans. To overcome the situation, the Bank provided technical assistance to develop plans for the two sites. Granada incorporated the plan in its municipal SWM plan (ICR, p.17, para 66). Two regionalization studies were conducted. (ICR p.22, p.83) Rating Modest PHREVDELTBL PHREVISEDTBL 5. Efficiency The project focus shifted to restructuring smaller landfills with capacities of 4 tons, 13 tons and 16 tons, as well as three small-medium landfills with respective capacities of 56, 57 and 96 tons. (Team Interview). Methodology-According to the PAD, the economic analysis assumed the tariff as a proxy for willingness to pay in the municipalities where appropriate final disposal service is provided and economic prices were assumed as financial prices without taxes. (ICR annex 3 p 36-39). For the ICR, a financial rate of return was estimated by measuring its costs and benefits at market prices. The ex post economic rate of return was estimated by converting financial costs into economic costs through the elimination of taxes and subsidies to remove some market distortions. Economic benefits consisted of the population's surplus resulting when the open-air dumps were closed and regional landfills were constructed and operated. Additional benefits, e.g. those related to public health and the environment, were not quantified. Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) Cost-benefit Analysis - A financial and economic analysis was undertaken for each component. Cost-benefit accounted for investment and operational costs as well as the benefits in terms of revenue (gate fee per ton times disposed tons). The same methodology was used to assess the ex-ante scenario (PAD) were used also to assess the ex post scenario. (ICR annex 3 p 36-39) Key Findings - Overall, the ICR findings in the tables below show that the combination of revenues from the tariffs and the relatively low cost but high impact investments in upgrading of existing regional sites under the restructured project contributed to the financial and economic viability of all the investments except for the smallest landfill, Miraflores (1 ton per day) located in a remote area of Colombia (ICR, p 23). Findings show that the construction of regional landfills is financially and economically feasible for landfills with a capacity higher than 40 tons’ day (see Annex 9). Moreover, economic results, as shown in Tables 1-2 below, are generally greater than financial results. Results were also tested against real-world uncertainties by conducting a sensitivity and risk analysis, which continued to show robust returns. Conclusions - The updated project cost table provided by the team included project costs broken down by the Bank and counterpart funding, but the team had difficulty identifying the actual costs and disbursement rates (ICR, p 17 and lessons learned under “Legal Framework”). During missions the Bank detected differences in total amounts of money assigned to some contracts when comparing the records of MVCD and those of the Gestores (ICR p 17 footnote 46). The Region clarified the source and costs used for economic and financial analysis. The costs used in the economic analysis were based on the costs found in official documents directly related to the contracts (final acts, agreements and contracts). These numbers were the same as those reported by the government after completion of the audit of the project in January 2017. The differences in total amounts of money assigned to some contracts when comparing the records of MVCT and those of the Gestores detected during a supervision mission (ICR p 17, footnote 46) were subsequently resolved during implementation and confirmed through audits. Table1. Financial Results-Net Present Value and Internal Rate of Return, incl. Investment and Operating Costs NPV IRR (US$)* (%) Regional landfill. Municipality of Ipiales (Nariño). 417,073 49 Municipal landfill. Municipality of Miraflores (Guaviare) - 322,277 10 La Doradita landfill. Municipality of La Dorada (Caldas) 866,007 48 Los Eucaliptos landfill. Municipality of Aguadas (Caldas) 568,557 45 Quinchia landfill expansion Municipality of Qunichia (Risaralda) Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) 10,453 15 La Guaratara landfill expansion. Municipality of Granada (Meta) 179,077 44 NPV = Net Present Value; IRR = Internal Rate of Return; Exchange rate:1 US$=COP$3.149 December 31th, 2015). Table 2: Economic Results - NPValue & Economic Rate of Return, incl. Investment & Operating Costs, excl. Taxes. Solid Waste Systems NPV IRR (%) (US$)* Regional landfill- Municipality of Ipiales (Nariño) 792,784 72 Municipal Landfill. Municipality of Miraflores (Guaviare) - 5,073 8 La Doradita landfill. Municipality of La Dorada (Caldas) 1,543,098 67 Los Eucaliptos landfill. Municipality of Aguadas (Caldas) 936,566 56 Quinchia landfill expansion Municipality of Qunichia (Risaraldra) 82,409 21 La Guaratara landfill expansion. Municipality of Granada (Meta) 397,952 73 Note: ERR = Economic Rate of Return 1] In some cases the intervention will be limited to the construction of transfer stations that will direct waste to existing landfills. In the case the financial viability of the transfer stations is ensured. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  0 Not Applicable 0 ICR Estimate  0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) 6. Outcome This rating is moderately satisfactory for the following reasons: relevance of objectives is high; relevance of design is modest; efficacy of objectives is substantial, substantial and modest, respectively; and efficiency is substantial. Therefore, Outcome is rated moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating Institutional risk - At the time of project closure, with help from national budget allocated to the MVCT, the solid waste unit was managing a small but important set of investments and activities. However, limited budget allocation to the unit and the sector, lack of World Bank (and other international institutions’) involvement may present a barrier to expanding their program to meet the country’s demand and to a lesser extent present a risk to the capacity built under the project. (ICR p24) Financial and operational risks - Both technical and financial risks are higher in Guaviare due to their lower access to resources, difficulty in supervision, and lower financial sustainability. (ICR p 25) a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry The original project design adopted similar design and institutional arrangements used in the water sector (ICR, p.1). The project appraisal conducted sector, poverty and economic analyses, identified safeguards issues, conducted financial analysis including fiduciary arrangements, and designed a results monitoring framework (ICR, p. 25). The Project aligned its preparation and objectives with the sector reform that was under way in the country at the time of appraisal and was included in the country strategy and policy documents. The design of the project took into consideration international best practices to promote the optimization of regional SWM systems. But it failed to facilitate compatibility between the following Bank and Government processes: a) the approach to project evaluation, (b) the disbursement structure through the subnational accounts and (c) the requirement to implement social inclusion plans. These were significant Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) deterrents at the outset of project implementation (ICR, p. 25). But making the implied adjustments was considered impractical considering that: (a) the water sector institutional, financial and operational framework did not adapt to investments in the solid waste management subsector; b) the project did not anticipate technical advancements in the sector nor allow flexibility to adopt them; c) a pipeline of subprojects was not ready at the time of Board approval of the project; and d) the need for capacity building was underestimated by the different government levels involved in project implementation (paraphrased from ICR, pp.10-11). The challenges of using the Ventanilla Unica (single window approach for subproject selection) (4) and the PDAs were underestimated along with the need for institutional capacity building of the MVCT and Gestores (geographic-level Department project management entities). (5) Risks were taken into account but underestimated as some risks such as the lack of pipeline of subprojects, implementation complexity through the PDA and entities not familiar with financial, management and procurement processes did materialize (ICR, p. 11). The project shifted the approach to regionalization of disposal services in order to incorporate more municipalities into regional systems. The project focus also changed to include the improvement of already existing regional facilities to reconstruct or upgrade them to allow for adequate disposal and to increase capacity to accommodate more municipalities. The program broadened its scope allowing for operation through a variety of arrangements (not just specialized operators), and investments in smaller facilities including isolated municipalities when regional facilities were not feasible. Notes * (4) The evaluation process, Ventanilla única, is an application-based process where the geographic Department-level Gestores would present projects to the MVCT who would ask for adjustments and, once approved would be presented to the Bank for No Objection (ICR, note 33, p. 9). Notes* (5)The Department-level planning and budget structures (Planes Departamentales para el Manejo Empresarial de los Servicios de Agua y Saneamiento, PDA) were established under the MAVDT to be the primary means of channeling resources from the National Government to the subnational geographic Departments for investments in water, sanitation, and solid waste. The PDA framework allowed for prioritization, investment screening, and execution of investments within a budget quota assigned to each Department (ICR, pp. 2-3). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The project was implemented over the period 2009-2015. Bank missions coordinated closely with representatives from the Minister of Housing, Cities and Territory (MVCT). “No midterm review was held; instead each mission had the elements of a midterm review and involved senior staff, management, and experts” (ICR, p. 26). The approval process for the restructuring and extensions was however delayed due to both the Bank and Government decision making and approval processes which did not affect the Gestores approval process directly. In 2011, the Bank requested reimbursement of US$1.07 million due to lack of execution, and discussed the options with the Borrower, including closure of the project and partial cancelation Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) of implementation, but agreed to restructure the project. However, agreement was delayed due to complications resulting from changes, design issues, and perceived risks, which led to an increased number of Bank requirements that were not easy to negotiate in the context of national legal requirements (ICR, pp.13, 26). The delays reduced time for implementation of the planned activities and disbursement of the loan amount and “accompaniment in technical, fiduciary and safe guard’s aspects was impactful for the project and sector” (ICR, p.26). On the positive side, key decisions regarding the design, lack of project and institutional readiness were overcome. The Bank worked with the MVCT to strengthen institutional capacity and funding. During 2012, based on the progress made, the special account was increased, including for use of Bank funds to finance social inclusion plans. (See lessons learned in section 13 for other key interventions.) The Bank supported South-South exchange visits, clients in the field by conducting field visits with help from specialists and regional experts, and conducted a solid waste sector study to improve the capacity of the Government (drawn from ICR, p.25). Owing to moderate shortcomings including delays related to complications resulting from changes, design issues, and perceive, an increased number of Bank requirements were not easy to negotiate in the context of national legal requirement the quality of supervision is Moderately Satisfactory. Pursuant to the Harmonized Criteria agreed by IEG and OPCS, with Outcome rated Moderately Satisfactory, Quality at Entry rated Moderately Unsatisfactory and Supervision rated Moderately Satisfactory, Overall Bank Performance is rated Moderately Satisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance Strong borrower ownership was evident at the Appraisal stage.The Government response to critical challenges facing the SWM subsector was comprehensive as demonstrated by its commitment to undertake legal reforms, policies, strategies and plans. MVCT was given the mandate for the development, promotion, and optimization of regulatory, technical, financial, and capacity-building instruments, including promoting regional planning and provision of services. National and sector planning was undertaken by the National Department for Planning (DNP). (ICR p 1-4). Commitment during the implementation phase was hindered by a weak legal framework, funding and capacity and working knowledge of Bank policies. The slow pace of execution of some contracts by Departments delayed the pace of disbursement and highlighted the need for further capacity building within the MVCT and at subnational levels where high rotation of personnel was a key issue. (ICR p12). Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The implementing agency was the Ministry of Housing, City and Territory ((MVCT); at the time of appraisal, the Ministry of Environment, Housing and Territorial Development (MAVDT). The MAVDT did not have the requisite stature and capacities, as well as relationships with the Departments to fulfill its role. Moreover, the solid waste unit was limited by its relatively low political profile. The activities of building this capacity were not considered as explicit activities and although Components 2 and 3 included capacity building. These activities were not focused on these capacity constraints and these activities were overlooked in implementation in part because they were not in the LA nor financed by loan resources. The MVCT staff used their own budget to enhance the regulation of operation by involving the Superintendencia and regional environment authorities; undertake low cost capacity building through exchanges with good operators; conduct supervisory technical visits by World Bank and MVCT experts; and develop capacity building consultancies through national funding (outside of the loan). The limited PDA Departmental budget quotas at the beginning of the Project and the overall limited budget allocated by the MVCT to the sector, led to subprojects that were designed or selected to be at a lower cost. A bottleneck for the Gestores was the MVCT approval process and was not affected significantly by the Bank approval process for these investments. (Comments made by the Region) Solid Waste Unit- The solid waste unit was established in the MAVDT with experienced staff but lacked the minimum staff stipulated by the LA, which was only reached by the end of 2011. The unit did not have technical professionals working exclusively for the project, but rather relied on part-time consultants with other commitments. The unit was at first limited by the lack of outreach to Departments. But over time, it was strengthened and professionalized. The unit disseminated good practices between municipalities, coordinated exchanges between service operators, and shared the Colombia experience globally. The solid waste unit steadily improved its contract supervision and monitoring of the sub-projects through field visits and periodic contact with the technical staff of Gestores Departamentales. Pursuant to the Harmonized Criteria agreed by IEG and OPCS, with Outcome rated Moderately Satisfactory, Government Performance rated Moderately Unsatisfactory and Implementing Agency Performance rated Moderately Satisfactory, Overall Borrower Performance is rated Moderately Satisfactory. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) a. M&E Design A monitoring and evaluation (M&E) framework was developed to track progress and subsequently adjusted post restructuring. (see details in PAD, pp.17-18). According to the ICCR, a key weakness of the PDO indicator [the beneficiary population] was that measures of key intermediate indicators were lacking, including the pipeline of subprojects at appraisal and of its evolution during implementation, as well as the changing populations served year-to-year by regional landfills. The absence of measurement of both factors resulted in “imprecise estimates of the project beneficiary population (the PDO indicator) which, combined with the lack of a good causal link to the intermediate indicators, provided an imprecise measure of PDO achievement." (see ICR, p. 15-16). “There was a weak causal chain between the intermediate indicators and the PDO indicator. The PDO indicator, which focused on the number of beneficiaries, did not properly reflect the intended impact of the loan or the PDO description entirely. Although the activities related to social inclusion, the financial sustainability and environmental benefits such as dumpsite closure were addressed by some intermediate indicators, they were not properly reflected in the PDO indicator." (ICR, p.15) The indicators did not, according to the ICR, properly consider several intermediate outcomes and outputs. Activities related with capacity building were not adequately reflected in the indicators, even though a significant amount of time and effort was invested in capacity building, which proved to be essential to attain the Project outcomes. Subprojects relied on studies that needed to be contracted and executed beforehand and were not reflected in the original M&E framework." b. M&E Implementation The original framework included several indicators that were not useful or no longer considered relevant post 2015 restructuring. Of the 14 original intermediate results indicators, ten were eliminated, three were adjusted (leading to more flexible ones in terms of targets or scopes) and only one intermediate results indicator was maintained. Seven new indicators were also included (three of them core indicators). The usefulness of the new set of indicators was limited by the delays in officially approving the restructuring, but increased during the last year and a half of the project when results were being achieved at a faster rate. (ICR p15). c. M&E Utilization While it was not mentioned under the M&E section, during the ICR stage, to better assess the achievement of objectives, the PDO indicator was supplemented by outcome indicators developed ex-post specifically for the purposes of the ICR. (See ICR p20-22 for specific details). Activities related with capacity building were not adequately reflected in the indicators, even though a significant amount of time and effort was invested in capacity building and it proved to be crucial to attain the project outcomes. Findings of the indicator performance were used as an input to draft country strategies and plans. (Team interview). Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) M&E Quality Rating Modest 11. Other Issues a. Safeguards Two safeguards were reported at ICR stage. OP 4.01 – Environmental Assessment and OP 4.12 – Involuntary Settlement. During the PAD and ICR stages, the environmental safeguard was assigned an “A” category. The compliance rating for OP 4.01 varied during implementation cycle. While no safeguard was triggered under OP 4.12, a resettlement plan was developed despite any displacement and social inclusion plans were developed with technical assistance provided by World Bank funds.(ICR pp16-17) b. Fiduciary Compliance The following was noted in different parts of the ICR, but the Region confirmed the disbursement rates partially. Financial Management- ICR findings show that financial management compliance ranged from Moderately Satisfactory to Unsatisfactory during the project implementation cycle. The main issues were a) the inability to justify disbursements at the beginning of the project that led to the World Bank asking for reimbursement of US$1.07 million due to lack of execution. Legal barriers prevented MVCT from undertaking the process of retrieving or substituting expenses for the small amount of funds not expended in subnational accounts, the special account was able to be closed satisfactorily. Limitations in the supervision at subnational level. (ICR p17). The second of three restructurings (August 2014) was substantial and renegotiation necessitated the incorporation of updated Bank requirements and legal measures designed to reduce risks related to the disbursements to subnational accounts. Three amendments to the disbursement letter took place (approved 03/31/2010, 12/13/2012 and 07/07/2015. with delays in the last modification which limited its utility. (ICR p15). Project reports were reviewed and found acceptable to the World Bank with some delays in contracting and presenting them (ICR p 17). Audit reports were not mentioned by the ICR. Disbursements-The ICR noted the following but the Team was not able to verify disbursement figures. The following information was pulled together from different parts of the ICR. During the first three years of the Project, there was little to no disbursement of the loan. The Bank disbursed US$2 million in December 2009 to the special account, which was then channeled into subnational accounts. Two years later, less than 4 percent of that amount had been paid against contracts. ICR findings show that disbursement arrangements necessitated adjustments during implementation, which led to significant delays. For FY12, the disbursement ratio was only 6.7 percent, while it climbed to 17 percent in FY13. The Project did not disburse the full amount of the remaining loan amount, due to the cancellation of subprojects. The limited PDA Departmental budget quotas at the beginning of the project and the overall limited budget allocated by the MVCT to the sector, led to Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) subprojects that were designed or selected to be at a lower cost. The fact that the subprojects did not have the expected size reduced the magnitude and rate of disbursement.(ICR p12, 15 and 17) Procurement-The procurement compliance rating at project closure was moderately satisfactory. At the outset, procurement challenges were cited as one of reasons for delays during project implementation. Bidding processes for US$ 3.6 million (San Gil, Choco, and Pamplona were cancelled and the time was limited for rebidding. (ICR p14). Delays were prevalent because the Bank’s procurement rules were not consistent with local procedures and practices and expectations. One of the main issues was the weak capacity of Gestores, which had no experience with Bank procurement policies and as a result, three procurement processes failed. (ICR p 17). c. Unintended impacts (Positive or Negative) Institutional change/strengthening. The Project also increased the interest and capacity of Departmental Gestores in the development of solid waste projects. Strategy and policy- As a complementary activity, a sector study was launched in December 2015. It incorporated the lessons learned from the implementation of the loan and highlighted the approach to recycling and resource recovery and informal recyclers. The findings were used by the DNP in designing the Programmatic Development Policy Loan for Sustainable Development. d. Other Other unintended outcomes and impacts (positive or negative). The Project helped increase technical standards within the sector by providing examples of good practice including examples of concession agreements, operational contracts, public management of small landfills (Quinchia) and the Colombian tariff system (see references in annex 7) that were re-disseminated in Colombia and internationally. 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Given that Relevance of Objectives is rated High, Relevance of Design is rated Moderately Moderately Outcome Modest, Efficacy of the first Unsatisfactory Satisfactory two sub-objectives are each rated Substantial while Efficacy of the third sub- Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) objective is rated Modest, and Efficiency is rated Substantial, Outcome is rated Moderately Satisfactory. Risk to Development Modest Modest --- Outcome Pursuant to the Harmonized Criteria agreed by IEG and OPCS, with Outcome rated Moderately Satisfactory, Moderately Moderately Quality at Entry rated Bank Performance Unsatisfactory Satisfactory Moderately Unsatisfactory and Supervision rated Moderately Satisfactory, Overall Bank Performance is rated Moderately Satisfactory. Pursuant to the Harmonized Criteria agreed by IEG and OPCS, with Outcome rated Moderately Satisfactory, Government Performance Moderately Moderately rated Moderately Borrower Performance Unsatisfactory Satisfactory Unsatisfactory and Implementing Agency Performance rated Moderately Satisfactory, Overall Borrower Performance is rated Moderately Satisfactory. Quality of ICR Modest --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR included seven lessons, several of which this Review adapted. Some lessons were added that appear relevant given the significant design, implementation and funding issues encountered by the project. 1. Global, and regional lessons learned from the other operations in the sector are not always suitable for replication due to specific country, sector conditions and priorities. The original design of this project adopted global and regional lessons learned (ICR pp9-11) and adopted similar institutional arrangements used in water and sanitation sector projects. However, as pointed out in the ICR (p. 27), the latter projects, owing to their long record, higher visibility and established procedures, tended to relegate solid waste investments to Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) lower budget priority in the PDA budget structure. Restructuring highlighted the need to consider the solid waste sector separately and adapt investment financing frameworks accordingly. 2. A conducive legal framework should be in place prior to project implementation. Prior to effectiveness, the Bank should understand the legal framework and be prepared to offer workable alternatives in the event that legal issues should arise during implementation. Colombia has a highly regulated public sector with public audit and control functions established in the constitution and national regulations governing government activities at all levels including basic services. (ICR p 1). A) Laws related to subnational accounts. In 2011, the Bank requested reimbursement of US$1.07 million of the loan from the Special Account due to lack of execution, and discussed the options of closure of the project and partial cancelation. But it was determined by Colombian authorities that this would have raised legal barriers due to the Constitutional provision that requires money to be used for designated purposes in designated Departments). In this project an alternative solution was developed entailing an increase to the Special Account and an agreement to work toward project restructuring rather than closure (ICR, p.13). B) Legal barriers related to social inclusion. The Government did not prioritize the social waste pickers as their activities were not supported by the prevailing law and there was strong opposition to changing the law. A subsequent Supreme Court ruling resulted in waste pickers being designated as within the public service. Near project closure, the Bank used its own funds to provide technical assistance leading to incorporation on a pilot basis of informal recyclers in two sites of the system. (Team Interview). 3. The importance of committing funds in advance of project implementation to help obtain stakeholder consensus. The regional landfill operations involved multiple municipalities and required extensive negotiation and consensus building across municipal parties which spanned a period of municipal elections and transition of administrations. Political consensus could not be obtained until funding was assured. (Team interview). 4. The role and importance of performance targets to improve accountability and improve project implementation. Project execution did not advance throughout 2010 and for a significant portion of 2011.The GoC and Bank agreed on a set of performance targets, which played a catalyst role in incentivizing the MVCDT to take action to speed up the process. (ICR p13). 5. Importance of closer coordination and continuous oversight by the Bank and Borrower to identify and mitigate risks in a timely manner. The project faced multiple risks at different levels, some of which were identified and mitigated better than others. Some risks pertaining to budget and staff allocations at higher levels were perceived but also seen as being of limited scope for influence by the Bank or implementing agency. There were two key risks that were not identified at appraisal. The high staff turnover at Departmental-level management entities for the PDA (Gestores Departmentales) was not considered in the risk analysis and in consequence, the mitigation measures regarding capacity building were not effective. The risk of using the subnational account structure and a ring-fenced project sub-account within a single national level (Patrimonio Autonomo) structure was not adequately identified and addressed. (ICR p 11) 6. Best practices related to technical standards merit dissemination. The Project helped increase technical standards within the sector by providing examples of good practice including examples of concession agreements, operational contracts, public management of small landfills (Quinchia) and the Colombian tariff Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Solid Waste Management Program Projec (P101279) system (as noted in Section 10d above; also see references in ICR, annex 7). These examples were re- disseminated in Colombia and globally. (ICR p 24, footnote 59) 14. Assessment Recommended? No 15. Comments on Quality of ICR The report was concise; but was more an implementation narrative than an outcome based report. Contrary to OPCS guidelines, a key shortcoming of the report was the substantial evidence found in different parts of the report which should have been organized under specific topics – relevance of design, quality of entry. borrower performance, government performance. Disbursement rates were partially confirmed. However, there was a lack of discussion related to operational and administrative efficiency. The indicators in the PAD were inadequate to demonstrate achievement of the PDO, but the ICR relied on new indicators identified ex post by the ICR team. An interview with the project team was useful to clarify nuances related to design, implementation, funding issues, efficacy, efficiency and M/E discussions, and project costs, as well as inconsistencies between Bank and Government processes. The comments sent by the region helped to rectify most of the discrepancies and clarify the nuances. Another round of editing would have improved clarity and readability. a. Quality of ICR Rating Modest Page 20 of 20