ICRR 13806 Report Number : ICRR13806 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/21/2012 Country : Bolivia Project ID : P073367 Appraisal Actual Project Name : Decentralized US$M ): Project Costs (US$M): 38.8 21.18 Infrastructure For Rural Transformation L/C Number : C3788 Loan/ US$M): Loan /Credit (US$M): 20.0 20.78 Sector Board : Energy and Mining Cofinancing (US$M): US$M ): Cofinanciers : Board Approval Date : 06/17/2003 Closing Date : 12/31/2007 05/27/2011 Sector (s): Renewable energy (34%); Power (33%); Information technology (33%) Theme (s): Other rural development (40% - P); Other urban development (20% - S); Poverty strategy; analysis and monitoring (20% - S); Infrastructure services for private sector development (20% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Fernando Manibog Soniya Carvalho Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: Program Objective The project was originally intended to support the first of a three phase program backed by three Adaptable Program Credits (APCs) over a ten year period. According to the Project Appraisal Document (PAD, page 3), the purpose of the program was "to assist the Government of Bolivia in expanding and improving the delivery of infrastructure services through private sector led mechanisms as a catalyst for the development of rural areas in Bolivia ." Original Project Development Objective The PAD (page 4) states that the project development objective is “to expand and improve the delivery of electricity and ICT services through private-sector led mechanisms as a catalyst for the development of rural areas in Bolivia ." The Development Credit Agreement (DCA) reiterates the APC program objectives as the objectives for the project : "The objectives of the Project are to assist the Borrower to expand and improve the delivery of infrastructure services through private sector-led mechanisms as a catalyst for the development of rural areas in the Borrower's territory ." Formally Revised Objective The project underwent a first-order restructuring that was approved by the Board on August 9, 2007, which removed ICT from the project (except for some small activities, such as identification of productive uses of ICT in rural areas ). The project development objective according to the July 2007 Project Paper (page 7) was as follows: " to expand and improve the delivery of electricity services through private sector -led mechanisms as a catalyst for the development of rural areas in Bolivia." Outcome and impact indicators were adjusted accordingly (Table 3, page 7 of the July 2007 Project Paper). Basis for this ICR Review This ICR Review is based on the original statement of project objectives in the PAD and the revised statement of objectives in the Project Paper . In accordance with OPCS/IEG guidelines, an overall outcome rating will be derived from separate outcome ratings against the original and revised PDOs, which will be weighted in proportion to the share of actual loan/credit disbursements made in the period before and after Board approval of the revision . This procedure was not followed in the ICR . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 08/09/2007 c. Components: I: National Strategies for Rural Electrification and ICT (US$3.4 million appraisal cost; US$0.8 million at restructuring; US$0.76 million actual) The component would assist the Vice Ministries of Energy and Communications in developing effective strategies to improve and accelerate electricity and ICT coverage expansion in rural areas . II. Rural Coverage Expansion (US$25.25 million appraisal cost; US$19.94 million at restructuring; US$17.97 million actual) The component would support (1) Solar photovoltaic market development for a target of 15,000–20,000 users, (2) Grid densification strategy and funding of a pilot project to reduce the cost of new connections, (3) financing, together with the Government and the private sector, of around 25,000 new cellular lines to benefit a population of about one million, and (4) other energy and ICT small interventions, such as pico -hydro stations and limited financing for the expansion of coverage of TV and local radio stations . III. Productive and Social Uses (US$2.59 million appraisal cost; US$0.3 million at restructuring; US$0.3 million actual) The component would identify, develop, facilitate and promote productive and social uses of the installed infrastructure: (1) Productive and Social Uses of Solar Energy, oriented towards promoting non -obvious uses and those that require lesser known types of equipment, as well as adaptation of existing processes, (2) Productive and Social use of ICT (together with a Development Gateway ) including the establishment of an internet platform, local content development, a community radio program, and a capacity building program . IV. Promotion, Communication and Capacity-Building (US$ 2.27 million appraisal cost; US$0.1 million at restructuring; US$0.1 million actual) This component was designed to complement the others by providing a common platform for community consultations, promotion campaigns and training of users in using the new infrastructure efficiently and productively . V. Project Management, Monitoring and Evaluation (US$3.52 million appraisal cost; US$2.00 million at restructuring; US$2.05 million actual) In addition to management, this component was expected to focus on M&E by developing indicators and following their evolution throughout project execution . After restructuring, the following investments were dropped : (a) new cellular phone connections in rural areas (25,000 targeted), and (b) rural population covered by the new cellular phone services (1 million targeted). The funding for the ICT component was reallocated to the component for new electricity users connected through grid densification, for which the target was increased from 300 to 10,000 users. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost . Actual project cost was US$21.18 million, 55% of the appraisal estimate of US$38.80 million. The reduction was due mainly to the removal of the planned ICT investments at restructuring, although the Solar PV component was also reduced from US$ 13.93 million US$8.75 million. There was a reallocation of US$8.12 million to the Grid Densification component, which rose from US$ 2.47 million at appraisal to US$10.59 million at closure. Financing . Annex 1(b) of the ICR states actual IDA funding to have been US$ 21.3 million. However, this is inconsistent with the total project cost figure at closure of US$ 21.18 million. This inconsistency was not explained either in the ICR or subsequently by the project team . Taking account of the Borrower contribution of US$ 0.40 million (there were no external financing sources other than IDA ), IDA’s funding is estimated at US$20.78 million. Borrower Contribution . At US$0.40 million, this was substantially below the US$ 2.40 million estimated at appraisal. Local sources of borrowing that were anticipated at appraisal did not materialize . Dates . There were three extensions to the closing date, totaling about 3.5 years. At project restructuring in August 2007, the closing date was extended by about 26 months, from December 31, 2007 to June 30, 2009. A second extension was granted to November 27, and following this an interim two month extension was agreed through January 27, 2010. At that time, a third and final extension was granted for an additional 17 months, leading to a final closing date of May 27, 2011. The extensions were granted to enable the completion of works related to the solar home system and grid densification components . 3. Relevance of Objectives & Design: a. Relevance of Objectives: The relevance of project objectives is substantial , for both the original and revised objectives . Both statements of objectives are fully consistent with the Government's current programs focusing on better welfare for rural and particularly indigenous peoples . This emphasis is reflected in the Bank's 2009 Interim Strategy Note and the World Bank Group's (WBG) Country Partnership Strategy (CPS) dated November 1, 2011, which indicates on page 18 that "During the CPS timeframe, improving access to productive infrastructure in rural areas will be a priority ." In terms of areas of WBG support, the CPS reiterates on page 23 that "Expanding access to basic services will continue to be a priority." The removal of most of the ICT-related activities (subsequently financed from the Government's own resources) at restructuring does not alter the continued priority of improving access to basic infrastructure services for the rural poor in Bolivia, which for the restructured project focused on the electricity sector . b. Relevance of Design: The relevance of the original project design is modest . The Results Framework (PAD, pages 46 to 54) contained unrealistic "stretch" targets for the entire anticipated APC series, rather than focusing on the first Phase covered by this project. The relevance of the revised project design is substantial . There was a clear and logical causal link between the activities financed and the achievement of the revised project development objectives, which focused mostly on the provision of electricity services . For instance, the implementation of a regulatory framework for rural electrification and the establishment of a transparent and competitive financing mechanism were essential underpinnings for the expected investments in photovoltaic solar home systems to serve the targeted number of new consumers . The Solar PV investments were to be implemented and maintained through Service Management Contracts financed by the project. 4. Achievement of Objectives (Efficacy): To expand and improve the delivery of electricity services through private -sector led mechanisms as a catalyst for the development of rural areas in Bolivia (Original and Revised Objective). Modest . Outputs A strategy for rural electrification was adopted . In December 2005, the Government issued a decree establishing the Rural Electrification Rule Book and the norms for photovoltaic installations . This decree established a new regulatory framework for rural electrification . 10,174 photovoltaic solar home systems were installed (target 15,000). Four management contracts are in place for solar home services (target six). 20,073 consumers have been connected to the grid under the so -called Grid Densification component (target 15,000). 356 users have been given access to photovoltaic systems for economic activities (wool shearing, poultry rearing, store lighting and machine tools ) – no target. Community workshops and training were organized on the care and maintenance of equipment and on the use of improved cooking stoves . Outcomes Both the number of solar home systems installed and the management contracts for such systems were below target. The number of users connected to the grid exceeded the target . However, the revised development objectives state that the expansion and improvement of electricity services would be a catalyst for "the development of rural areas in Bolivia." It is unclear from the ICR that the beneficiaries of the Grid Densification component in fact dwell in rural, rather than in peri-urban or urban, areas. It would not be economically viable to connect remote and isolated rural communities to the grid . The project team responded to IEG's request for clarification by stating that users linked to the grid were "located near the electricity company ’s grid (i.e. they do not need major transmission investments, substations etc .), but beyond the limit for which the electricity company is obliged to connect a user." In the IEG Project Performance Assessment Report (PPAR) that is being recommended (see Section 14 below), one of the primary lines of inquiry would be to obtain adequate evidence of the extent to which project beneficiaries (especially those of the grid densification component ) were in fact rural dwellers, and thus to assess the extent to which the project catalyzed "the development of rural areas in Bolivia ." While a new regulatory framework for rural electrification was adopted, there is no evidence provided in the ICR that it has been implemented or tested in the market or that it is "efficient", "effective", "competitive", and successful in "facilitating private investment" as stipulated in the triggers contained in the restructured project . To expand and improve the delivery of ICT services through private sector -led mechanisms as a catalyst for the development of rural areas in Bolivia . (Original Objective) Negligible . Outputs No strategy for rural services was developed, nor was any regulatory framework to enable private investment in rural telecommunications adopted . There was no expansion of telephone coverage . A first bidding round for output-based aid investments in rural telecommunications was unsuccessful . A second round was declared unresponsive by the new Government which took office in January, 2006. The Bank and the Government agreed that it would be impossible to organize a third round of bidding and to implement the ICT components before the closing date of December 31, 2007 (ICR, page 6). ICT investments were dropped from the project at the August 2007 restructuring. Outcomes There were no project outcomes in the ICT sector . 5. Efficiency: Efficiency is rated modest . The ICR on page 11 reports an economic rate of return (ERR) of 15 percent for the Solar Homes System component (representing about 33 percent of total disbursements ), compared to the ERR of 29 percent estimated at appraisal . For the Grid Densification component (representing about 47 percent of total disbursements), the ICR reports an ERR of 50 percent for the first phase and 36 percent for the second. However, these figures are not based on robust assumptions about the willingness to pay for superior luminance compared to traditional lighting sources. Moreover, the successive changes in Government created an unstable institutional environment and had serious negative impacts on the project . There were considerable implementation and procurement delays, which necessitated a project restructuring, and a three -and-a-half year extension of the closing date ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 36% 47% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Outcome under the original objectives is rated unsatisfactory , based on negligible efficacy of one objective (the ICT component did not perform) and modest efficacy of the other, modest efficiency and design relevance, and substantially relevant project objectives . Outcome under the revised objectives is moderately unsatisfactory , based on ratings of substantial for relevance of objectives and design, and modest efficacy and efficiency . While intended physical outputs for electricity were partially completed, there is considerable uncertainty about the extent to which rural, rather than urban and peri-urban dwellers, benefited from the Grid Densification component, and therefore whether the project achieved its development objective of acting as "a catalyst for the development of rural areas in Bolivia .� The derivation of the final outcome rating is presented in the table below, following the "Rating The Outcome of Projects With Formally Revised Objectives : Joint Guidelines for Use in ICRs and in OED Assessments ." Against Original PDOs Against Revised PDO Overall Rating Unsatisfactory Moderately Unsatisfactory Rating Value* 2 3 Weight ** 16% 84% 100% Weighted Value 0.32 2.52 2.84 (rounded to 3.0) Final Rating Moderately Unsatisfactory * Based on: Highly Satisfactory= 6; Satisfactory=5; Moderately Satisfactory=4; Moderately Unsatisfactory=3; Unsatisfactory=2; Highly Unsatisfactory=1 ** % disbursed before and after PDO change a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: On the basis of what the ICR describes as the relatively good performance record of power distribution companies in Bolivia, risks to the outcome of the grid densification component are considered as low . However,risks to the solar home systems are moderate. The establishment of the basic legal conditions for rural electrification have not been fully subjected to market tests, particularly with respect to attracting and keeping sufficient private sector participation in the rural electricity sector. The overall political context has also proven to be highly unpredictable . Overall, risks to development outcome are rated as moderate . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The project's performance indicators, as well as its assumptions regarding the Government's commitment, political and regulatory stability, and private sector interest, were unrealistic . As the ICR correctly indicates on page 13: "The project design was too complex to be achieved in a 5-year period. In addition, the Output-Based Aid contracts included an unrealistic 4-year post-installation maintenance guarantee that could not be achieved during the project's implementation." The latter, which required amendment of the solar home systems contracts, led to delays. Another major source of delay was the failure to ensure that the project was in the 2004 National Budget, which was under preparation by the Government as the project was being appraised, approved, and made effective in March, June and December 2003, respectively. at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: As indicated on page 13 of the ICR, the project was adequately supervised, in terms of mission frequency and resources. The Bank team was responsive to issues as they arose and provided constructive advice . However, the specialist members of the team rotated frequently, which disrupted continuity in dialogue, and made the team incomplete at times. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The successive changes in Government created an unstable institutional environment and had some negative impacts on the project. After two unsuccessful bidding attempts, the Government awarded a contract for the ICT component on a single source basis to a local company ENTEL . Shortly thereafter, ENTEL was taken into public ownership, and the regulatory framework for the telecommunications sector was abolished . The Government’s counterpart contribution was reduced to US$ 400,000 from an appraisal estimate of US$2.4 million, so that the project was funded almost entirely with scarce IDA resources . The Government, nonetheless, was able to fund the ICT component itself after it had been dropped from the project and to take the telecommunications company into public ownership. These aspects need to be reviewed in the recommended IEG PPAR . Government Performance Rating Unsatisfactory b. Implementing Agency Performance: The implementing agency was the Project Coordination Unit, housed in the Ministry of Hydrocarbons and Energy. There were many rotations at the Director level, and in the fiduciary and procurement staff . Project procedures were cumbersome. There was weak knowledge and capacity in contract processing, as well as in managing relationships with the different units of the Ministry . These factors led to significant implementation delays, which translated into serious disbursement lags . However, during the last 15 months of implementation, the Unit (specifically the Director) accomplished much more progress for the project compared to the entire preceding period. Field supervision was intensified . Towards the end, the Unit was able to retain a core of qualified staff and streamline its procedures . Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: According to the PAD (page 6), progress towards meeting indicators of the impact of enhanced infrastructure services in rural areas would be “measured through a comprehensive monitoring and evaluation (M&E) program, which will be based on the active participation of users and communities and will be facilitated by the new ICT tools developed in the project. For this purpose, an M&E expert will be hired as part of the [Project Coordination Unit] staff.� The list of indicators included a user satisfaction index, reduced turnover of rural teachers and health care worker, measures of economic development (new investment, employment and income generation ), and increased indoor illumination quality in rural households The PAD also states (page 22) that M&E was a specific component of the project (together with project management), and that its activities would include “(i) development of M&E indicators, baseline and methodology; (ii) periodic evaluation of progress toward achieving the identified targets; and (iii) final evaluation upon project completion.� It is not specifically stated in the ICR who would be responsible for M&E, but by implication it would be the Project Coordination Unit. The project team subsequently clarified that a "monitoring office was created by the Project . All the staff was financed by the Project ." b. M&E Implementation: According to page 7 of the ICR, a monitoring office was established to oversee contracts, as well as monitor risks and performance indicators. The monitoring office was staffed with a unit director, a PV /Solar Homes Systems specialist, a grid densification specialist, an information specialist, and eight field supervisors . It is not stated whether this entire staff was in place from the outset, or expanded during implementation . The monitoring office verified proposals of contractors applying for subsidies, in line with the Output -Based Aid* approach of the project. The subsidies were granted in accordance with the number of households supplied by the contractor with electricity services . The monitoring office also collected project data and administered contracts . There were important weaknesses in M&E performance . None of the economic and social impact indicators mentioned under Design above is discussed, or even mentioned in the ICR, suggesting that they were not measured . No baseline data appear in the PAD, while the Results Framework in the Data Sheet of the ICR contains no baseline data either, again suggesting they were not established . An ex-post assessment in 2008 of the first phase of the Grid Densification component concluded that the monitoring reports did not contain the data required under the project's Environmental Management Plan (EMP). *Under the Output-Based Aid approach, service providers bid to supply a certain number of consumers with services, the prices of which would be insufficient to cover costs (usually because the consumers are poor or isolated or both). The winner of the contract would (ceteris paribus) be the provider who offers the service to the largest number of consumers and asks for the lowest subsidy, either per head or in total. c. M&E Utilization: The ICR provided little information on M&E utilization, except to indicate that the monitoring office "ensured that the Project was present and visible in the field, ensuring channels of communication with the users, and helping to strengthen communication with local officials ." The monitoring office continues to operate . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: Environmental. The project was assigned a Category "B" for Environmental Assessment (OP 4.01). An Environmental Management Plan (EMP) was prepared. An ex-post assessment in 2008 of the first phase of the Grid Densification component concluded that the monitoring reports did not contain the data required under the EMP . The companies involved provided the missing information later, which showed that the environmental impacts were minor and had been adequately mitigated . A Bank safeguards specialist joined the Bank team in January 2010. In the second phase of grid densification, environmental procedures were included at all stages, and monitoring and compliance occurred in line with the EMP, according to page 8 of the ICR. Social. The project also triggered the Bank's Indigenous Peoples (OP 4.20) and Involuntary Resettlement (OP 4.12) policies. The ICR on page 8 states that "during project supervision, the Bank did not find evidence of any serious noncompliance..." However, supervision aide-memoires did raise the issue of lack of adequate documentation on (i) consultations prior to grid densification sub -projects, and (ii) the implementation of the Indigenous Peoples Plan . The ICR indicates that "the documentation was reconstructed after the fact " and training was conducted to improve documentation related to social safeguards compliance . With the apparently uneven performance on safeguards, these aspects would be reviewed in the recommended IEG PPAR. b. Fiduciary Compliance: According to page 8 of the ICR: "Lack of experienced and qualified staff for prolonged periods resulted in outdated and inaccurate accounting records thereby affecting the quality of financial information and timely compliance with audit requirements." It seems that financial management performance improved only towards the end of the project implementation period. There is no indication in the ICR as to whether the opinions of the project ’s external auditors were qualified. The ICR (page 9) states that “the Project Coordination Unit was responsible for the procurement activities and the contracting process; responsibilities were handled in a satisfactory manner . Nonetheless, “given the specificity of the procurement methods applied by the Project Unit, difficulties were faced when interacting with the Ministry authorities, not familiarized with World Bank Procurement Guidelines . Therefore, the signing of contracts was cumbersome.� Supervision reports rated procurement performance as moderately satisfactory . Given the uneven record, financial management compliance would also be reviewed in the recommended IEG PPAR. c. Unintended Impacts (positive or negative): d. Other: 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately The efficacy of the original objective to Satisfactory Unsatisfactory expand the delivery of ICT services was negligible, as was the relevance of design. While several physical outputs were achieved in terms of delivery of electrical services, it is unclear whether most beneficiaries were bona fide rural dwellers and hence whether the project achieved its objective of serving as "a catalyst for the development of rural areas in Bolivia." Restructuring improved the relevance of design and improved the overall outcome rating by dropping the ICT objective. However, even after restructuring, both efficacy and efficiency were modest, equating with significant shortcomings and a moderately unsatisfactory outcome rating. Risk to Development Negligible to Low Moderate The political environment, the uncertain Outcome : market reaction to the Government's regulatory framework, and the moderate risks to the solar home systems component, point to an overall moderate risk rating. Bank Performance : Moderately Moderately There was a lack of realism and Satisfactory Unsatisfactory attention to critical inputs at entry, and while major issues were addressed during supervision, frequent team rotations resulted in delays and lack of continuity in dialogue. Borrower Performance : Moderately Unsatisfactory Government changes created an Satisfactory unstable environment that caused considerable delays. Government actions indicate weak commitment to the project concept of relying on the private sector to supply infrastructure services in rural areas. Government performance with regard to the ICT component, and in respect to procurement and fiduciary aspects in general, was weak. The implementing agency suffered from a high rate of key staff turnover. Financial and procurement management improved only towards the end of the implementation period. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The most important lesson from the experience of this project is the following : It is important to ensure that the intended beneficiaries reflected in the statement of objectives are, indeed, reached by the project’s activities. If the Government changes, and the new Government demonstrates, at best, diminished commitment to the project’s original approach, and restructuring rather than cancelation is chosen, then the restructuring should be openly geared towards the new Government ’s strategic goals. In this case, the objectives could have been modified to incorporate poor beneficiaries who do not live in the rural areas designated in the original statement of objectives . As it was, even after restructuring, rural dwellers were cited as the main beneficiaries and the project ’s goal continued to be to act as a catalyst for rural development . Nonetheless, the allocation of funds to photovoltaic solar home systems, which would indeed have reached rural dwellers, were reduced and reallocated to a greatly expanded “Grid Densification� component, which would more likely benefit those living in peri-urban or urban areas. 14. Assessment Recommended? Yes No Why? A Project Performance Assessment Report (PPAR) by IEG is recommended to address the following issues : Assess available evidence on the extent to which the project (i) contributed as a catalyst for the development of rural areas in Bolivia, or (ii) benefited mostly urban and peri -urban areas in the grid densification component, which accounted for almost half of actual project disbursements . Obtain and assess data on the medium -term field performance of the photovoltaic and solar home systems, as well as their benefits to rural areas . Gain a more comprehensive understanding of the justification for canceling the ICT component, in light of the Government's actions and the APC series that was at stake . Assess compliance with the Bank's environmental and social safeguards, and fiduciary requirements . The assessment would also be an important input into IEG ’s forthcoming Energy Evaluation . 15. Comments on Quality of ICR: The ICR is analytical and candid in its assessment of the project's achievements and shortfalls . It provided a lot of substance within the 15-page limit, which is a challenge given the project's complex history . However, the ICR did not follow the approach for restructured projects that is required by OPCS guidelines,i .e., to derive an overall outcome rating from separate outcome ratings against the original and revised development objectives . There were other shortcomings. There is no discussion of whether the substantial expansion of the so -called Grid Densification component was compatible with the rural orientation of the project objectives . The reported actual credit disbursement total is greater than the project cost . There is no mention of project external audits . a.Quality of ICR Rating : Satisfactory