ICRR 11965 Report Number : ICRR11965 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 01/14/2005 PROJ ID : P035743 Appraisal Actual Project Name : Environmental Project Costs 45.0 45.0 Management And US$M ) (US$M) Cartography Project Country : Venezuela Loan/ US$M ) 28.0 Loan /Credit (US$M) 28.0 Sector (s): Board: ENV - Central Cofinancing 0 0 government administration US$M ) (US$M) (81%), Sub-national government administration (19%) L/C Number : L4253 Board Approval 98 FY ) (FY) Partners involved : Closing Date 12/31/2002 02/27/2004 Prepared by : Reviewed by : Group Manager : Group : Fernando Manibog Roy Gilbert Alain A. Barbu OEDSG 2. Project Objectives and Components a. Objectives The project objectives as stated in the ICR do not fully reflect the original objectives in the PAD . The PAD states: First, to modernize Venezuela's institutional capacity for environmental management at the national, state and municipal levels by supporting the development of national policies, legislation, regulations, economic incentives and information systems for (i) environmental protection, monitoring and evaluation, (ii) integrated environmental & economic planning, and (iii) decentralized environmental management in the country . Second, to provide improved, high quality and timely services to the private sector (e.g., issuance of permits; environmental impact assessment and certification), and foster the development of private environmental services and industry sector in Venezuela . Third, to provide map data for the whole country, for realizing full economic, social and cultural growth as well as national security, environmental conservation and protection . The ICR states that the objectives remained the same although the project activities were trimmed down during the midterm review. However, this OED review concludes that there was a de facto amendment of project objectives . As a result of the midterm review, the project's key development objective (i.e., the overall strengthening environmental management institutions in Venezuela ) was effectively eliminated because core activities designed to achieve it were removed, as follows : (i) improvement of the Environmental Assessment system and certification; (ii) improvement in the Permit System for industrial plants; (iii) improvements in the environmental legislative and regulatory framework; (iv) preparation of a National Environment Plan; (v) the design of an autonomous environmental management service; and (vi) professional development of the Ministry of Environment and Natural Resources (MARN). While this may have made sense -- given the recognition that the project was overdesigned and unimplementable -- the Bank did not restructure the project and modify the objectives . Instead of this original objective, what the project really pursued after the mid -term review were selective improvements in the building and computer facilities of MARN, the "consolidation" of a laboratory, small pilots, and laying of the basic technical groundwork for monitoring environmental compliance -- a far cry from the project's original objectives and tangentially related to the project's environmental institutional reform components . This is evident from Annex 1 of the ICR on the project's performance indicators, almost 90% of which were not achieved (monitoring was also weak, as discussed in Section 5 below). b. Components There were three original components (the appraised costs and the re -allocated costs after the midterm review shown below): Component Cost at Appraisal (US$ million) Actual Cost (US$million) 1. MARN Modernization 27.78 24.15 2. Environmental Management at 6.35 8.44 State and Municipal Levels 3. Cartography Modernization 10.87 12.41 TOTAL PROJECT COST 45.00 45.00 c. Comments on Project Cost, Financing and Dates While there was some reallocation among the three components at the midterm review (see table above), the total cost at project closing remained the same as the appraised US$ 45 million, for which the Bank provided a loan of US$28 million while the Government provided local counterpart funding of US$ 17 million. The project's original closing date of December 31, 2002 was extended by 14 months to February 27, 2004. 3. Achievement of Relevant Objectives: (1) Modernize the institutional capacity for environmental management at the national, state and municipal levels by supporting the development of national policies, legislation, regulations, economic incentives and information 54 % of project costs ) - Minimally Achieved . systems (54% The original objective was only minimally achieved when measured against the paltry project results . Although intended, no indigenous lands and habitat were demarcated . The planned Center for Promotion of Clean Technologies was never implemented . The Ecobuses to deliver environmental education programs in remote areas was also not implemented due to the delays in refurbishing old buses . The partial achievements are: (i) the already existing Hatillo laboratory was "consolidated" as the National Reference Laboratory; (ii) a Coastal Law was drafted; (iii) a land management plan was prepared for one state, although the original coverage was national; (iv) three state maps were updated and environmental diagnostics were carried out in 7 regions; and (iv) MARN computerization was implemented, as well as some building improvements, although even this was partially achieved . (2) Environmental management at the state and municipal levels 19 % of project costs ) - Minimally Achieved . (19% There were only very minimal achievements with respect to this component's intent of developing the institutional capacity to "hold states and municipalities accountable for fulfilling their environmental management responsibilities " (e.g., through environmental action plans and reviews of environmental regulations ). As in the previous component, the national coverage of this objective was also reduced to just the states south of the Orinoco during the midterm review. The technical capabilities of the Environment Directorate in the Amazonas State was improved, albeit by only 60% of targets, and vehicular and ICT equipment were also provided . Similar strengthening occurred in the Bolivar State. Two Municipal Environment Offices were created in Nueva Esparta State, with some MARN functions transferred there. A water quality monitoring network was designed (but not implemented as it was meant to be ) in the Aragua State. 27 % of project costs ) - Partially Achieved . (3) Cartography modernization (27% Only the Map of Risks of Avila was completed, up to the point of marketing and sale . The Cartosur II was completed but not yet available to the public at project closing . Only 10% (i.e., the conceptual design) of the important activity to create a geography, cartography and census database for Venezuela was implemented . The Government seemed to have been only interested in this cartography component, which in the end was the only one with measurable achievements (partial as they are). 4. Significant Outcomes/Impacts: Modernization of MARN's ICT facilities and part of its building . Consolidation of MARN's existing Hatillo Environmental Laboratory into a National Reference Laboratory . Completion of the Map of Risks of Avila and Cartosur II maps, although only the first one is available for sale . 5. Significant Shortcomings (including non-compliance with safeguard policies): This high risk, Bank-designed project was overly ambitious (it had 22 themes and 495 discrete activities), was poorly prepared and did not have the commitment or support from either the senior management or the personnel ranks of the environmental institutions that had to implement it . At least 10 of the ICR's Lessons (page 17) have to do with Bank weaknesses during preparation and appraisal . The Bank lost much leverage by agreeing (during the midterm review) to eliminate the critical activities that really mattered in Venezuela's environment -- notably much-needed improvements in the environmental assessment system, the process for industrial environmental permits, the application of economic incentives, environmental regulations, and planning . The Bank-designed project had two major design flaws : (1) The Autonomous Environmental Management Service (SAGA) was given the singular authority to grant environmental permits to private industry, and -- through MARN -- recover its costs from those receiving permits . This could create a conflict of interest and provoke rent-seeking behavior. (2) The project focussed exclusively on MARN and failed to see cross - sectoral environmental "mainstreaming" as the real issue. Government agencies for the energy, water, transport, urban and education sectors -- all with a clear interest in the issues raised by the project -- had little or no involvement. Despite its complexity, the project had no effective monitoring and evaluation system . The "ad hoc evaluation system" that was established was for procurement and budget tracking only, and had little to do with monitoring and evaluating the project's development outcomes . The Bank made no effort to correct this during implementation: despite the elimination of important project activities, the performance indicators were not revised during the midterm review. The ICR's Annex 1 (Log Frame) candidly shows empty boxes throughout the matrix, stating simply that "Monitoring of indicators was not implemented ". 6. Ratings : ICR OED Review Reason for Disagreement /Comments Outcome : Satisfactory Unsatisfactory Although the project's relevance was substantial, its efficacy was modest and its efficiency was negligible. Institutional Dev .: Modest Negligible The project did not lead to efficient use of Venezuela's resources through better definition, stability, transparency and predictability in the country's environmental institutions and their management functions. This resulted from the elimination at the midterm review of critical activities to improve planning, environmental assessment, regulations, market-based incentives, and permits . Sustainability : Likely Unlikely In line with the Bank's own Environment Strategy, the environment is to be "mainstreamed" across sectors and not just concentrated on one single environment authority (MARN in a centralized command-and-control role in this case). The absence of mainstreaming in this project points to the unlikely sustainability of its limited achievements. Bank Performance : Satisfactory Unsatisfactory In addition to the Bank's shortcomings in Section 5 above, it also reflects poorly on the Bank's accountability to have maintained original objectives without restructuring the project after a midterm review, even though the drastically trimmed set of activities really correspond to a much-scaled down, new objective. This is a bad precedent, as the Bank's Board could be misled into approving a project with substantial objectives while only a fraction would be actually implemented. Borrower Perf .: Satisfactory Unsatisfactory Despite formal pronouncements from higher Government levels during the early stages of the project cycle, there was little commitment and ownership from the management and personnel ranks who had to implement the project. The Government was unwilling to carry out core institutional reform activities, which the Government eventually eliminated during the midterm review. Key staff changes were very frequent, and the lack of budget planning and the slowness of the Finance Ministry's financial allocations resulted in work stoppages and severe delays. Quality of ICR : Unsatisfactory NOTE: NOTE ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness. 7. Lessons of Broad Applicability: 1. Future environmental institutions projects should be designed to mainstream the environment and diverse environmental management instruments across sectors and national /subnational levels (including the private sector ), rather than to focus on consolidating a central environment ministry and command -and-control approaches. (On a positive note, environmental mainstreaming was integrated in more recent LAC projects, notably the Mexico-Environment SAL I.) 2. In this type of project that seeks to reform and strengthen a country's environmental management capacity, the participation and ownership of the personnel and senior officials from involved Government agencies is critical, throughout the project cycle but particularly at the project conceptualization and preparation stages . 8. Assessment Recommended? Yes No Why? There is substantial disagreement between the ICR and OED ratings, and the accountability issue of whether the project should have been restructured (including cancellation of the unimplementable parts of the project) needs gtto be assessed . Project completion also needs to be evaluated, because of important weaknesses in the ICR, as explained below. 9. Comments on Quality of ICR: The ICR is unsatisfactory, for reasons explained below . Lack or Absence of Analysis . The ICR's focus of reporting on the project's outputs has limited value as a learning tool and source of guidance for designing future environmental management strengthening projects . In particular, it did not address the following key questions, among others : (1) To what extent did the elimination of key activities affect the likelihood of achieving the project's original objectives, which remained unchanged? What was at stake during preparation /appraisal and at project closing, what were the costs of not having implemented those critical activities? (2) What are the Government's intentions during the mid -term review, and what are the obstacles regarding the future implementation of the critical activities that were removed from the project? (3) In addition to inputs and outputs, what were the development outcomes of the (limited) project achievements? The ICR did not provide any evidence of the creation of a private environmental services industry in Venezuela, although this was a key project objective . The Bank's failure to restructure the project although major parts were taken out should have been accounted for by the ICR . Lack of Evidence . The ICR provides little or no evidence to its many qualitative assertions . For example, in para 4.5, MARN is described as "transformed", "much stronger", "very well positioned as head of its sector ", having an "improved efficiency", etc., with no evidence. Yet these assertions are negated by the project's limited achievements . The ICR's analysis of the Government's and implementing agency's "satisfactory" performance was justified primarily on the availability or shortage of counterpart funds (inputs), rather project outcomes. MANR's performance was not analyzed at all, as the implementing agency analysis in paragraph 7.6 is an ad verbatim repetition of paragraph 7.5 on the Government's performance. Too much focus on process, not results . In addition to many examples above, the lessons given are keyed to process, not results and development outcomes . In its first lesson, the ICR equates the mere appointment of an MANR Vice Minister as evidence of Government commitment, which may show project support at the official level, yet the ICR does not analyze what the elimination of the project's core activities indicates with respect to the Government's operational level of commitment. Strangely, the ICR recommends that a Bank institutional expert should train country project personnel in the preparation of the PCD and the PAD, which are internal Bank documents prepared by the Task Manager . Need for Quality Control .The report should have been carefully reviewed (for substance, to address the foregoing issues) and thoroughly edited before its release (the cover page shows the project simultaneously as a loan, a credit and a grant; the acronyms have many errors; there are no page numbers in the Table of Contents; there are many errors of grammar and syntax). The ICR often uses the future conditional tense (as in a PAD) thus leading to the confusion whether outcomes really occurred or are projected to happen in the future . The ICR has potentially many useful lessons, but they are confusing and need to be better phrased and clustered thematically .