ICRR 13982 Report Number : ICRR13982 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/16/2013 Country : Chile Project ID : P076807 Appraisal Actual Project Name : Chile Infrastructure US$M ): Project Costs (US$M): 90.26 90.26 For Territorial Development L/C Number : L7269 Loan/ US$M): Loan /Credit (US$M): 50.26 50.26 Sector Board : Transport Cofinancing (US$M): US$M ): Cofinanciers : Board Approval Date : 12/16/2004 Closing Date : 06/30/2010 12/31/2011 Sector (s): General water sanitation and flood protection sector (30%); Roads and highways (27%); Telecommunications (22%); General public administration sector (12%); General energy sector (9%) Theme (s): Rural services and infrastructure (29% - P); Rural policies and institutions (29% - P); Participation and civic engagement (14% - S); Infrastructure services for private sector development (14% - S); Administrative and civil service reform (14% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Victoria Alexeeva Roy Gilbert Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: “To increase the effective and productive use of sustainable infrastructure services by poor rural communities in Territories of the Eligible Regions� (Loan Agreement, Schedule 2, p. 19). The Project Appraisal Document’s (PAD) formulation is the same, the only difference being that it names the participating regions as Coquimbo, Maule, BioBio, Araucania, and Los Lagos, and adds "or any other region as may be proposed by the Borrower and agreed to by the Bank " (PAD, p. 6) b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: Component 1: Participatory Territorial Planning (appraisal cost US$3.75 million; actual cost not reported by the ICR). This component was to assist local stakeholders and regional government agencies to : (i) prepare the development framework plans in prioritized rural territories in the five eligible regions; (ii) identify demands for improved infrastructure services, and (iii) monitor progress with plan implementation . Component 2: Infrastructure Service Delivery (appraisal cost US$76.47 million; actual cost not reported by the ICR). This component was to support : (i) feasibility and design studies of subproject proposals to submit for financing under this component, (ii) rehabilitation of rural roads, and construction, rehabilitation and expansion of rural water, sanitation, electricity and ICT infrastructure, (iii) supervision of the mentioned works, and (iv) studies and support for the establishment and strengthening of local service providers to operate, maintain and administer the services to achieve quality and sustainability . Component 3: Institutional Strengthening (appraisal cost US$5.26 million; actual cost not reported by the ICR ). This component was to support : (i) project coordination units at national and regional levels, (ii) implementation of studies and capacity building to strengthen policies and institutions in areas of territorial planning and rural infrastructure service delivery (including those related to the application of social and environmental safeguards), and (iii) project monitoring, evaluation and learning as a contribution to national level expansion of the program. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost : Total project cost was US$90.26 million at closure as appraised. This is the amount the ICR reports in Annex 1- Project costs and financing . There were additional resources allocated by the Borrower to the project (see Borrower contribution below). The ICR does not provide actual cost data for each project component. Financing : The World Bank Group contribution consisted of a loan in the amount of US$ 50.26 million as appraised. There was no external partner financing . Borrower contribution : The ICR reported the actual Borrower contribution of US$ 40 million as appraised in Table B, Annex 1. As a footnote to the table, the ICR adds that the Borrower allocated additional resources, i .e., the Subsecretariat of Regional Development (SUBDERE) contributed US$40.6 million from its own budget for a total contribution of US$80.6 million; sectoral ministries and the National Fund for Regional Development (FNR) contributed US$29.2 million; and 27 productive initiatives linked to the PMDTs were financed with US$ 12.9 million from other sources (ICR, page 24). Thus, total investment from local sources would amount to US$ 122.7 million. Dates : The project was extended by one year and a half from the original closing date of June 30, 2010 to December 31, 2011 as a result of implementation delays . The extension was granted during the second amendment to the Loan Agreement in September 2009. The first amendment took place in February 2009 and included the following changes : (i) expanding the definition of eligible subprojects to include the financing of small port works; (ii) allowing financing of pre-investment studies for productive subprojects (this was to help communities access additional funding, under other programs ); (iii) allowing for financing operation and maintenance; (iv) modifying the role of a national coordination body for the project ; and (v) increasing financing percentages from 50% for works and 70% for consulting services to 100%. In April 2011 the Government of Chile (GOC) and the Bank agreed to include the regions Libertador Bernardo O ’Higgins and Aysen del General Carlos Obañez del Campo for financing under the loan . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial . At the time of appraisal, despite a significant commitment of the Government of Chile expanding rural infrastructure services, there were still about 500,000 of the estimated 2,000,000 rural population that lived in poverty and lacked basic services . The project development objective (PDO) was consistent with the Government of Chile's development priorities to continue coverage expansion into the highly dispersed and hard to reach rural populations while maximizing the sustainability and productive and social impact of existing infrastructure investments. The PDO is also relevant to the key goals of World Bank Group's FY 11-16 Country Partnership Strategy for Chile (current at project closure) that include reducing economic and social inequality in Chile through enabling the emergence of economic opportunities in high value added activities and promoting human capital development. In particular, increasing the access of rural population to markets for their products, among others. The PDO was also in line with the main priorities of the Country Strategies at appraisal and during implementation, which aimed at sustaining economic growth and social progress as well as heightening inclusion, especially of rural populations and vulnerable groups . b. Relevance of Design: Substantial . The statement of development objectives was clear . The project results framework indicated a clear causal chain between the activities financed by the project and the outputs and outcomes related to the attainment of the development objectives. For example, with regard to physical investments in civil works for rural roads, water, sanitation, electricity, and ICT infrastructure (Component 2), the activities were intended to lead to an increased effective and productive use of sustainable infrastructure services by poor rural communities . Institutional strengthening activities and technical assistance (Components 1 and 3) were meant to help identify demands and prioritize the benefiting poor rural communities . A necessary selection criterion was the existence of ongoing potentially viable economic activities (ICR, page 5). As was identified by the evaluation report of the project by the Budget Office within the Ministry of Finance (DIPRES), the project was too restrictive both in terms of eligible infrastructure subprojects and the non -inclusion of financing for productive activities (ICR, page 7). As a result, in February 2009 the Loan Agreement was amended to expand the definition of eligible subprojects to include the financing of small port works and allow financing of pre-investment studies for productive subprojects . 4. Achievement of Objectives (Efficacy): Increased effective and productive use of sustainable infrastructure services by poor rural communities in selected territories : Substantial . The achievement of project objective is rated based on the achievement of outputs . Outputs 31 Territorial Development Framework Plans (PMDTs) were approved by the Regional Governments (COREs) meeting and exceeding the target of 25 (18 PMDTs correspond to the first generation and 13 to the second one). In addition, at project closure 14 PMDTs were under preparation (third generation of PMDTs) for a total of 45 PMDTs. According to the PAD, the agreed standards included (i) evidence of community, indigenous and gender participation, (ii) sign-off of communities for the identified investments, and (iii) adequate consideration of social, environmental standards (PAD, page 37). In this respect, the ICR reports that social considerations were fully embedded in project design, including both indigenous and gender issues. The consultation process was inclusive . The National Corporation for Indigenous Development considered the methodology as the most appropriate to represent indigenous points of view . For 7 subprojects, all beneficiaries were indigenous (ICR, page x). According to the evaluation of social and environmental aspects of the PMDTs by a consultancy firm in 2012, women accounted for about 40% of the members of the “Water Committees� (the local social organizations /entities that managed the operation and maintenance of the project’s rural water supply subprojects). 250 subprojects were financed under the project : 158 road subprojects, 65 water supply subprojects, 7 sewage subprojects, 12 energy subprojects, 6 ICT subprojects, and 2 port subprojects reaching about 320,000 beneficiaries in participating, poor rural communities (ICR, page 13). The annual percentage level of compliance with targets set in PMDTs was met: Water and sanitation: The target of 95% of water and sanitation connections constructed with service providers having revenues greater than cost of operation and maintenance was met. The ICR reports that "tariffs were set to fully recover marginal costs " (ICR, page x). According to a survey carried out in November 2011 (comprising 12 water subprojects), all the Water Committees reported having enough funds to undertake maintenance activities (ICR, pages 46-47). Roads: The target of 95% of kilometers of roads rehabilitated with sustained routine maintenance mechanisms established was met. The ICR reports that the secondary road network that is under the direct responsibility of the Directorate of Roads (Direction de Vialidad, DV) of MOPTT, is routinely maintained. Vialidad agreed to incorporate into their maintenance programs all the roads improved under the project, including those not formally under their responsibilities, i .e., “no enroladas �. Electricity: The target of 95% of electrification constructed with service providers having revenues greater than the cost of operation and maintenance was met . The ICR reports that "tariffs were set to fully recover marginal costs" (ICR, page x). The MIDEPLAN evaluation methodology was expanded as planned to include assessment of subprojects within the territorial development approach advanced by the project . The methodology was formally approved by MIDEPLAN in March 2010. An updated version was prepared in August 2010. The methodology was used for other than the Territorial Development Framework Plans (PMDTs), in particular it contributed to prioritize investm ents for reconstruction works after the 2010 earthquake. New approaches were adopted in water and sanitation as planned . These included (a) appropriate technologies for water for dispersed populations and rural sanitation, (b) improved financial policy to increase cost recovery, (c) legal framework that would allow providers higher management and revenue raising capacity, and (d) assignment of institutional responsibilities for rural sanitation and water for dispersed population. The project contributed to the e laboration of a comprehensive rural water manual, which is being used by SUBDERE, DOH and municipalities for the selection and implementation of appropriate low cost technologies . This work provided inputs to drafting a Rural Water Law, which was approved in 2013 as subsequently confirmed by the project team . With respect to the target of 3 quality PMDTs implemented outside the project regions using non -project resources or 25% of regional public investments program implemented with a territorial approach, outside project financing, the ICR reports that three regions outside the project were incorporated in the program in 2012 planning to begin preparation of the PMDTs (page xii). The inclusion of Antofagasta and Atacama was planned for 2013, and the regions of Valparaiso and Metropolitana were programmed to join in 2014 when the program was expected to have the national coverage . Additional resources were allocated for the project activities from local sources amounting to total investments for infrastructure services delivery (Component 2) of US$153.7 million compared to the appraisal cost of US$ 76.47 million. Outcomes PDO outcome: Increase in use of quality and sustainable services in the selected territories, as measured by the following achievement of targets : The target of 90% of population benefiting from project -supported PMDTs with sustained access and effective use of services provided by the project was met . According to a survey carried out in November 2011 (comprising 12 water subprojects), it was reported that all the systems were working and were generating enough resources to cover operational and maintenance costs . The target achievement of 20% of increased use of transport services along rehabilitation road infrastructure has not been measured . According to the PAD, the indicator was supposed t o be measured by pro-rated increase in demand six months after completion of works, for a representative sample of roads (PAD, page 35). The ICR says that according to a survey carried out in November 2011 (comprising 22 road subprojects), the beneficiaries reported an increase in the availability of transport services but it was not possible to quantify such increase (only one case reported that the availability of transport before the project was once per day, and after the project it was once per hour ). The beneficiaries also reported that they could increase their production because of improved condition of the road (e.g., production of berries that are sensitive to damage during transportation ). The target achievement of 90% of population benefiting from project PMDTs with increased use of electricity for economic activities (as was supposed to measured by surveys as per PAD ) has not been measured. With respect to this target, the ICR says that "this indicator is not consistent with the project ’s objective of supporting productive activities . In any community only a minority of the population uses electricity for economic activities " (ICR, page ix). While the ICR reports that "electricity benefitted activities such as small fisheries, olive oil production among others ", there are no sources of evidence in the ICR as the survey was undertaken only for a sample of road and water supply subprojects (ICR, Annex 5, page 45). The planned impact assessment has not been carried out, so there is no evidence on increased annual production. 5. Efficiency: An ex-ante overall rate of return was not estimated at appraisal . The PAD provides ex-ante IRRs of a sample of the initial set of subprojects identified in the five eligible regions, i .e., 9 out of 14 sub-projects or about 50% of the total cost of US$6.2 million (PAD, Annex 9, page 77). The ICR reports that an ex-post specific economic or financial rates of return analysis for the overall project was no t possible at appraisal "due to the demand-based nature and characteristics of the multiple, relatively small subprojects that were to be financed by the project " (ICR, Annex 3, page 32). The evaluation methodology of MIDELPLAN was planned to be used for the selection of sub -projects that depended on the type of subproject (water and sanitation, roads, electricity, and telecommunications ) and overall included the following steps : (a) diagnosis of the current situation in the area of intervention; (b) analysis and estimation of current and future demand and supply; (c) definition and analysis of alternatives; (d) identification of benefits (private and social, with conversion factor defined by the MIDEPLAN ); (e) identification of costs (also private and social, with conversion rates defined by the MIDEPLAN ); (f) calculation of the social (economic) NPV and other economic indicators; and (g) results of sensitivity analysis on the basis of variations to benefits, costs or both (PAD, page 72). As the ICR subsequently concludes, the MIDELPLAN approaches could not capture "investment benefits accruing to disperse populations " (ICR, page 32). This initial evaluation methodology (with some adjustments) was used in the selection of subprojects in the first generation of Territorial Development Framework Plans (PMDT1), and as the ICR acknowledges, this is "casting doubts on the quality of the selected subprojects in terms of their productive impact, i .e., social rather than productive c onsiderations in many cases may have prevailed " (ICR, page 11). A new improved evaluation methodology was approved and used for the second and third generation PMDTs . PMDT1: The first generation of PMDTs (total 18) covered 74 subterritories in 5 regions. The ICR does not provide the total number of beneficiaries . A results survey of 40 communities was carried out for a sample of 22 road subprojects and 12 water supply subprojects. Positive results identified by the beneficiaries of the road suprojects that could be attributed to the project included decrease in travel time, improved connectivity, increase in frequency of transport services, increase in tourism, and avoiding of loss of perishable goods because of better roads; negative results included increase in traffic and littering in some cases . Positive results from the water subprojects included less gastrointestinal diseases, increase in production, and issuance of sanitary permits to some businesses . PMDT 2-3: The second generation PMDT2 (total 13) covered 71 subterritories in 3 regions, and the third generation PMDT3 (total 14) covered 40 subterritories in 3 regions. The ICR does not provide data on the total number of beneficiaries. Impact assessment is being delayed as of the time of ICRR (November 2013), as confirmed with the project team. The closing date was extended for one and a half years as a result of implementation delays . As was subsequently confirmed by the project team, no cost overruns had been reported . Additional resources were allocated for the project activities from local sources (see section 2d above). Total investments under infrastructure services delivery (Component 2) amounted to US$153.7 million as compared to the appraisal cost of US$76.47 million (ICR, page 30). Efficiency is assessed as substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project remains substantially relevant to the rural development of Chile . Relevance of design is rated substantial. Sustainable infrastructure services were delivered under the project serving about 320,000 people in participating poor rural communities. There is, however, still little evidence of the increased productive use of these infrastructure services as the project had intended . Efficiency is rated substantial . Outcome is assessed as satisfactory. a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risk is considered moderate as a result of the following factors : Institutionalization of the project - developed methodology and tools . These included: the methodology for an integrated economic and social evaluation of multi -sector projects formally adopted by MIDEPLAN for the evaluation of Territorial Development Framework Plans (PMDTs); the Manual on Water and Sanitation Technical Solutions for Rural Areas served as the basis for drafting of the Water and Sewage Sector Law, which was approved in 2013 (as subsequently confirmed by the project team ). Government's continued financial commitment . The Government allocated US$33 million in the 2012 budget for the continuation of the PMDTs . Continuous expansion of the PMDT program . In 2012 the regions of Arica, Parinacota, Tarapaca and Magallanes were incorporated to the program . The inclusion of Antofagasta and Atacama was planned for 2013, and the regions of Valparaiso and Metropolitana were programmed to join in 2014 when the program was expected to have the national coverage . Commitment to maintenance of all the roads improved under the project by the Directorate of Roads (Direction de Vialidad, DV) of MOPTT. Vialidad agreed to incorporate into their maintenance programs all the roads improved under the project, including those not formally under their responsibilities, i .e., “no enroladas �. Ongoing attention to the program seems likely, but ne eds to focus more upon the productive use of the program infrastructure, and not just its delivery . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: In 2003, the Bank prepared a number of studies to assist the Government of Chile (GOC) to find new approaches to the delivery of rural infrastructure services . Among major constraints of the existing system identified was the use of social and economic evaluation methodologies that were inadequate to capture the economic merits of initiatives in highly dispersed rural areas as well as inefficiencies due to centralized decision-making based on individual sector programs and funding allocations . The dialogue with the GOC evolved to designing an operation that supported a new system for the delivery of rural infrastructure to ensure continued efficiency in resource expenditures while improving the economic development potential of the rural poor. The lessons incorporated into the project design from other Bank - supported projects included avoiding shortcomings related to a lack of inter -sector coordination of the delivery mechanisms through creation of the Project Steering Committee, incorporating participatory approaches as well as territorial approaches to improve coordination and complementarity of investments . Bank experience in other countries showed that when several infrastructure services were made available simultaneously in the same rural area, the impact was greater than the sum of the individual effects . The officials interviewed by the ICR team recognized the high quality and effectiveness of the Project Operational Manual in clearly delineating the steps necessary to select project- supported territories and subprojects . Community participation, approvals, signing of working agreements, validations, among other, were required from community representatives, municipalities, local elected legislators, central government appointed local authorities, representatives of the sector infrastructure agencies both at the central and regional levels, Ministry for Social Development (MIDEPLAN) and other agencies supporting productive infrastructure and activities . A range of risks was identified during project preparation and appropriate mitigation measures were considered. In particular, the possibly limited capacity of the different levels of government to deal with new participatory approaches to planning were mitigated though the inclusion of legal instruments clearly specifying the roles and accountability of the different agents . Flexibility was ensured to address any possible weakness in institutional capacity through technical assistance and consultancy . However, despite recognizing and explicitly stating in the PAD that the project implementation needed a substantial lead time, the expected pace of execution and disbursements was unrealistic and resulted in an early downgrading of the performance rating, which, at one point, jeopardized the government's ongoing support of the project. There was also insufficient attention to the local -procurement decision processes that slowed down implementation at the initial stage . Full acceptance of the Bank ’s agreed documents took place about 2 years after loan effectiveness (see Section 11b). at -Entry Rating : Quality -at- Satisfactory b. Quality of supervision: The ICR reports that supervision was carried out regularly; infrastructure sector experts were included in most supervision missions (ICR, pages 19-20). Throughout this initial period (2005-2008), the Bank supervision missions and SUBDERE tried to accelerate the slow pace of project implementation . The following actions were agreed to and undertaken : (i) comprehensive and closely monitored action plans to finalize PMDTs; (ii) workshops launched to improve coordination and communication between stakeholders at the political and technical levels; (iii) re-launching meetings when the unfulfilled expectations at the community level started to erode the original enthusiasm for the project; and (iv) expert consultants hired by some regions to help in the preparation of the PMDTs and strengthen procurement capabilities . As a result of these actions, in 12 months between late 2008 and 2009, the project disbursed almost as much as during the previous three years of implementation . The ICR acknowledges that there were weaknesses in the quality of supervision, in particular insufficient consideration was given to the development of the initial high quality PMDTs including the development of reliable baselines for impact assessment . Since PMDT1 were developed prior to the existence of the approved methodology, as the ICR acknowledges, this is "casting doubts on the quality of the selected subprojects in terms of their productive impact, i .e., social rather than productive considerations in many cases may have prevailed" (ICR, page 11). The approval of the PMDT1 subprojects "might have been influenced by pressures to accelerate the initially slow project implementation " (ICR, page 19). There was a lack of clear terms of reference for the PMDT 1 development, which represented a new concept to the main regional stakeholders, resulting in uneven quality of PMDT 1 among regions (ICR, page 10). The weaknesses also included inadequate follow -up on SUBDERE’s obligations for M&E, i.e., "the ground work for a meaningful impact assessment exercise " of the PMDT2 and PMDT3 subprojects started only towards the project closure (ICR, page 20). Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The ICR acknowledges the Government ’s commitment and support to the project during preparation and implementation (pages 5, 20). The GOC provided a continuous support for decentralized, demand -driven approaches, and gave local communities and agencies substantial latitude to develop their own strategies for rural development as required by the project . During the initial years of implementation, when the project performance was rated unsatisfactory, the Government (i.e. MoF, Contraloría, MIDEPLAN) thoroughly scrutinized and evaluated the project to identify remedial actions . The findings led to amendments to the Loan Agreement in 2009 that accelerated the project execution . Other actions on the part of the Government that contributed to an improved project performance included the decision to reform SUBDERE to increase its role as a coordinator of regional investments as well as a strong pressure on public entities to accelerate disbursements as part of a US$ 700 million macroeconomic stimulus package emphasizing regional infrastructure. The ICR reports that while the main effectiveness condition of creating the Project Steering Committee were complied with, the meetings of the Committee never took place. The Project Steering Committee (Directorio Nacional) was to comprise high level representatives of the Ministry of Public Works, Transport and Telecommunications (MOPT), the Ministry of Finance (MOF), MIDEPLAN, Ministry of Agriculture, and Ministry of Economy, and provide overall strategic directives for project implementation . In 2008, however, when the project started to regain momentum, a new technical committee was formed to provide the type of coordination originally expected from the Steering Committee (ICR, page 7). Also, despite the fact that the safeguards screening mechanisms were not implemented as agreed as part of the Environmental Management Framework of the project, the national environmental requirements for infrastructure projects were followed, and supervision missions did not detect any negative environmental impacts. Government Performance Rating Satisfactory b. Implementing Agency Performance: The ICR commends the implementing agency, the Sub -secretariat of Regional Development (SUBDERE), for its efforts throughout the preparatory stages of conveying the merits of the territorial planning approach to the beneficiaries in the regions and securing th eir early ownership and commitment to the project (page 7). During the early stages of implementation, SUBDERE lacked experience with project execution and project coordination that affected the project performance . These weaknesses, however, were overcome through changes in personnel and incorporation of procurement experts in line with Bank ’s recommendations. The new managers in SUBDERE showed their firm support to the project and were proactive in accelerating the slow pace of project implementation. SUBDERE was in charge of the monitoring and evaluation (M&E) system, which had only been partially implemented by project closure. The agency was supposed to assess the gradual impacts of the implementation of the infrastructure components of the PMDTs on the communities in the selected territories and on the related productive activities . A low priority given to M&E system during early years of project implementation delayed carrying -out a full impact assessment measuring the project impact on incomes . "The ground work for a meaningful impact assessment exercise " started only towards the project closure (ICR, page 20). Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The project's indicators were appropriately linked with the objectives . Project outcomes were to be measured by the increased use of sustainable and quality services by the rural population and their contribution to economic and productive activities as spelled out in Annex 3 of the PAD. The key indicators included baseline data and measurable targets set at appraisal, except for the outcome indicator measuring an increase in annual production; baseline data was to be collected at the initial stages of project implementation and the indicator was to be tracked separately by the implementing agency as it involved a broader set of actions not under the control of the project (PAD, Annex 3, page 35). SUBDERE was responsible for managing and operating the M&E system . b. M&E Implementation: The planned M&E system was only partially implemented . Readily available data, i.e., physical outputs of sub-project delivery, the number of completed PMDTs, the potential aggregated beneficiary population in a given territory, was routinely collected and reported . That was not the case of the more relevant impact assessments, however. As the ICR reports, the Borrower and the Bank claimed that the continuous struggle "to keep the project alive" during the first years of implementation reduced the priority conferred upon M&E (ICR, page 10). Later into the project implementation period, and as the government confirmed the PIRDT ’s importance as a full-fledged tool for the delivery of rural infrastructure, there was renewed interest in evaluating its impact on productivity and rural incomes . Project results for the initial first generation PMDTs (or PMDT1) were inferred from simple community surveys . For the second and third generation of PMDTs, the definition of the baseline had been completed by project closure . Impact assessment is being delayed as of the time of ICRR (November 2013), as confirmed with the project team . c. M&E Utilization: The M&E system was used to support recommendations or inform management decisions during the course of implementation. M&E Quality Rating : Modest 11. Other Issues a. Safeguards: This was a Category B project that triggered four safeguards policies – OP4.01 Environmental Assessment, OP 4.12 Involuntary Resettlement. OP4.10 Indigenous People, and OP4.11 Cultural Resources. Environmental Safeguards and Management : The project preparation with respect to environmental assessment was carried out in accordance with the World Bank Operational Policy 4.01 on Environmental Assessment. An Environmental Management Framework (EMF) provided procedures to screen infrastructure subprojects and mitigate potential impacts that went beyond the national requirements . This procedure included: (i) a preliminary environmental assessment, (ii) subproject categorization, (iii) selection of the environmental assessment instrument, and (iv) implementation of a given environmental assessment instrument . However, with regard to preparation of the Territorial Development Framework Plans (PMDTs), the EMF prepared by the Borrower did not include information and guidance for mainstreaming of environmental considerations into formulation of PMDTs. Furthermore, the project’s Operations Manual did not incorporate the aforementioned procedure nor the environmental guidelines . Even though the procedures agreed with the Bank during project preparation were not implemented, SUBDERE followed the national environmental requirements for infrastructure projects. The ICR reports that "supervision missions did not detect any negative environmental impacts" (ICR, page 12). Indigenous People : "The attention and documentation in regard to social issues in general, and social safeguards specifically, was considered outstanding by the Quality Assurance Team (Currently Safeguards Advisory Team (SAT))" (ICR, page 5). The specific regions targeted by the project had about 42% of Chile’s Mapuche Indian population, according to the 1992 census of adults. A Social Assessment and an Indigenous Peoples Plan (IPP) were prepared to comply with OD 4.10 and the National Corporation for Indigenous Development (Corporación Nacional de Desarrollo Indígena – CONADI) was consulted during project preparation. During project supervision, the Bank observed that the project delivered significant benefits to communities of Indigenous Peoples in the regions of Araucan ía, Maule and Bío Bío and that negative social impacts were minor or nonexistent. Subprojects responded to needs expressed by the community and facilitated productive activities. Community participation in the planning and implementation of subprojects was a critical ingredient of their success, particularly in the case of subprojects to provide rural water supply to indigenous communities. Involuntary Resettlement : A Resettlement Policy Framework was prepared to comply with OP 4.12 on Involuntary Resettlement. In practice, the project did not cause physical displacement (relocation of people). Land for subprojects was acquired through voluntary transactions, except in the case of a road rehabilitation subproject that required the expropriation of narrow strips of land adjacent to the road . Expropriation became necessary in this subproject due to changes in its original design . The expropriation was carried out by the Roads Department of the Ministry of Public Works (MOP), which allocated funds for the compensation payments; however, some of the payments had to be deposited in the courts because the expropriated owners did not have title to the land affected . These persons would be entitled to the payments after they legalize their tenure status . The ICR does not address compliance with OP 4.11 Cultural Resources, a safeguard likely to be triggered by the project according to the PAD. b. Fiduciary Compliance: Procurement : Harmonization of the Bank's and the GOC's procurement requirements, i.e., bidding documents of the Ministry of Public Works, Transport and Telecommunications (MOPTT), took time and contributed to the initial slow pace of project implementation . The review of the existing legislation supporting adherence to international agreements was completed in October 2005. In mid 2006, MOPTT sent to its regional representatives the order to use the new bidding documents . Training and hiring of procurement experts helped to incorporate the new documents, although there were still some regions reluctant to change . Full acceptance of the Bank’s agreed documents with the consent of Contraloria, the government's principal auditing agency, took place in a formal act in 2007, about 2 years after loan effectiveness . Financial management . According to the ICR, the project was characterized by a satisfactory performance and a low risk in FM (page 11). For the project pre-financed expenditures, SUDBERE put in place a mechanism to validate information from regional governments and interact permanently with regional governments to assure expenditures eligibility. The project used the Integrated Financial Management System SIGFE complemented by SUBDERE’s own system to produce financial information . Interim un-audited reports (IFRs) were sent on timely basis, except for the IFR for the second semester 2011, which was overdue and expected to arrive before the end of April 2012. Audit reports were generally submitted on time, with unqualified audit opinions and considered acceptable by the Bank. c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Moderately Due to the fact that the Quality of Satisfactory Supervision rating is moderately satisfactory. Insufficient attention was given to the development of initially high quality Territorial Development Framework Plans (PMDTs) that should have included the development of reliable baselines for future impact assessments. Also, the Bank should have been more diligent on following up on M&E system and impact assessment. Borrower Performance : Satisfactory Moderately Due to the fact that Implementing Satisfactory Agency Performance Rating is moderately satisfactory. The M&E system had only been partially implemented by project closure. A low priority given to M&E system during early years of project implementation delayed carrying -out a full impact assessment measuring the project impact on incomes, and thus achievement of the objective . Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR has identified a number of lessons and the most important ones are listed below with some adaptation: The design of technical assistance for planning in projects with national coverage in poor rural areas might require a thorough assessment of existing community needs and available optimal social delivery systems . Rural development projects need to diagnose the quantity and quality of technical assistance services at an early stage of project preparation and include appropriate measures for their improvement, which may include public /private arrangements, piloted in selected territories, and strategies for ensuring the delivery of services consistent with the type of expected product . Focus on realistic intended outcomes right from the beginning should become an integral part of projects, which are pioneering substantive and complex changes in servicing rural communities . The project showed the importance of (i) having realistic estimates of intermediate and final targets and the lead time to achieve them; (ii) having in place continuously open channels of communication with beneficiaries to discuss progress /problems/remedial actions to reach the agreed targets; and (iii) creating synergies, disseminating information and building participatory networks which promote direct interface between government officials of different regions and beneficiaries of different communities . M&E should remain a high priority during both project design and implementation, and weaknesses in the original arrangements and responsibilities should be addressed as soon as detected . This is even more important in pilot projects having to prove their effect for the purpose of future replication . Project design should consider establishing M&E as a legally binding obligation that if necessary could persist beyond project closing so as to ensure reliable measurement of impacts with substantial lags . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is informative. Lessons are based on evidence and analysis . There are a number of weaknesses of the report, however. It did not provide the actual component costs meaning that we cannot know what the project invested in each of them, or how efficient that investment was . The ICR's reporting on safeguard compliance should have been more complete . As an exercise in self-evaluation, the ICR is weakened by the use of unsubstantiated statements about results, such as affirming that project ’s objectives were highly relevant to country priorities, without specifying what those priorities were or providing a referenced source (ICR p. 13). The ICR assessment of the Risk to Development Outcome lacked substantive references and evidence . More seriously, the ICR concluded that the project objective of increasing the use of infrastructure was achieved while providing little evidence that the infrastructure provided under the project had indeed been used more intensively. a.Quality of ICR Rating : Unsatisfactory