Report No. 8971 Annual Report on Operations Evaluation (FY90) August 15, 1990 Office of the Director-General Operations Evaluation FOR OFFICIAL USE ONLY Document of the World Bank This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. ABBREVIATIONS ARIS - Annual Report on Implementation and Supervision DGO - Director-General, Operations Evaluation ECDP - Evaluation Capability Development Program EDI - Economic Development Institute IFC - International Finance Corporation IAR - Investment Assessment Report (IFC) JAC - Joint Audit Committee OED - Operations Evaluation Department OEU - Operations Evaluation Unit (IFC) PCR - Project Completion Report PPAR - Project Performance Audit Report PRE - Policy, Research and External Affairs SAL - Structural Adjustment Loan SAR - Staff Appraisal Report SECAL - Sector Adjustment Loan THE WORLD BANK FOR OFFICIAL USE ONLY Washington. DC 20433 USA 001Ce of 01ectar-e~ra1 Opertens tvlutum August 15, 1990 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Annual Report on Operations Evaluation Attached, for information, is this year's *Annual Report on Operations Evaluation." The report provides an overview of the major evaluation activities during FY90, and reviews the effectiveness of dissemination and feedback processes. It also discusses issues relevant to future programs of operations evaluation. Attachment This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. FOR OFFICIAL USE ONLY ANNUAL REPORT ON OPERATIONS EVALUATION, FY90 Page No. INTRODUCTION . . . . . . . . . . . . . . . . . . 1 MAJOR EVALUATION ACTIVITIES . . . . . . . . . . . 2 Project Completion Reports . . . . . . . . . . 2 PCR Reviews and Audits . . . . . . . . . . . . 3 Annual Review and Evaluation Studies . . . . . 4 DISSEMINATION AND USE OF EVALUATION FINDINGS . . 5 Absorbing the Lessons from Experience . . . . 6 Management Responses . . . . . . . . . . . . . 8 OED's Initiatives . . . . . . . . . . . . . . 9 Access to OED Findings . . . . . . . . . . . . 11 External Dissemination . . . . . . . . . . . . 11 OPERATIONS EVALUATION DEPARTMENT . . . . . . . . 12 Staffing . . . . . . . . . . . . . . . . . . . 12 Use f Resources . . . . . . . . . . . . . . . 13 Use of Non-Bank Resources . . . . . . . . . . 15 OPERATIONS EVALUATION IN IFC . . . . . . . . . . 15 Investment Assessment Reports . . . . . . . . 15 Special Studies . . . . . . . . . . . . . . . 16 Dissemination . . . . . . . . . . . . . . . . 16 CONCLUSIONS . . . . . . . . . . . . . . . . . . . 18 Annexes A - Operations Evaluation in the World Bank 19 B - Operations Evaluated During FY90 . . . . . 23 C - Approach Papers for Ongoing Studies Approved by the JAC . . . . . . 36 D - Monitoring and Evaluation in Projects (M&E) . . . . . . . . . 37 E - Report on Dissemination and Feedback . . . 44 F - IFC Investment Assessment Reports for FY90 Filed in the Office of the Director-General, Operations Evaluation 87 G - IFC Special Evaluation Studies . . . . . . 88 This document has a restricted distribution and may be used by recipients only in the performanceT of their official duties. Its contents may not otherwise be disclosed without World Bank authoriration ANNURL REPORT ON OPERATIONS EVALUATION (FY90) INTRODUCTION 1. Operations Evaluation in the World Bank is now twenty years old. Each one of these years has helped to establish this function as a means of accounting to the Bank's shareholders for the effectiveness of its operational policies and programs, and as a source of experience to be shared across the Bank, with borrowers and increasingly with donors, and to be applied in the design and implementation of new operations. 2. Progress has continued in the three years since the Bank's reorganization of 1987. While reaffirming the need for evaluation and the mandate of the Operations Evaluation Staff, these years have created opportunities to breathe new life into evaluation approaches, products, their dissemination and use, and to further enhance the transparency of both the evaluation process and its results. 3. FY90, the focus of this Annual Report, was particularly successful. Operations produced 295 Project Completion Reports (PCRs), the highest number ever, exceeding both the number of PCRs falling due in the year and the number that had been programmed. The Operations Evaluation Department (OED), which faced a sudden large inflow of PCRs at the end of FY89, adjusted its progrars accordingly and with help of a mid-year budget increase was able to exceed all its objectives for PCR review, investment and adjustment audits, while maintaining its output of studies. 4. FY90 also saw a growing interest across the Bank in evaluation and operational experience from all sources, including OED. As well as the more formal managers' responses to OED studies, there has been a surge in informal comments on--and discussions of--OED output: audits, studies, and approach papers as well as the work program outlined in April 1990 and the Annual Review of Evaluation Results for 1989. OED staff comments have been invited more frequently, not only on policy papers but also to contribute experience relevant to the choice and design of new operations. OED staff also participate, along with a broad section of Bank staff, in the current review of issues in the economic appraisal of projects, itself stemming in part from concerns raised in recent Annual Reviews. 5. More emphasis is also being placed on developing actionable conclusions and recommendations from evaluation work, so as to contribute more effectively to the quality of operational policies and programs. Attention continues to be given to the views of all parties, the quality of analysis, and the transparency of processes and outputs to ensure the fairness and relevance of judgments made. 2 6. Major evaluation activities undertaken during FY90 are discussed in the next section of this report, which is followed by a review of dissemination and feedback processes. Also included are reviews of OED's staffing and budget situation, and of operations evaluation in the International Finance Corporation (IFC). A description of the Bank's operations evaluation system is given in Annex A. MAJOR EVALUATION ACTIVITIES Project Completion Reports 7. Self-evaluation through PCRs is the first major step to account for the outcome of completed projects, and to disseminate the operational experience to both Bank staff and borrowers. PCRs are also a primary source of information for project performance audits, and are essential building blocks for the Annual Review of Evaluation Results as well as for evaluation studies undertaken by OED and elsewhere in the Bank. 8. To improve the content, timeliness, and cost-effectiveness of PCRs, new guidelines were issued to operational staff in June 1989, reflecting the findings of an earlier review by a Bankwide working group. They provide for completion reports to be prepared in conjunction with the last supervision mission--a procedure that is designed to ensure staff continuity and timeliness of production. A new PCR format requires an assessment of the project experience from the perspective of both the Bank and the borrower. 9. The new procedure coincides with a major effort by operational managers to clear up the backlog of PCRs which had accumulated during much of the 1980s. As a result of this, OED received PCRs for 295 operations in FY90, exceeding the number of PCRs that came due within the year (227) by a significant margin. Nineteen PCRs were for structural adjustment operations. 10. Of the 295 PCRs received: 142 helped reduce the backlog of 281 operations outstanding at the end of FY89 (leaving 139 more than one year overdue); 102 helped offset the 227 which became due in FY90 (adding 125 operations overdue by less than a year); and 51 PCRs were prepared in advance of the FY91 program. However, within the slightly decreased backlog, adjustment operations actually increased from 13 at end of FY89 to 19 at the end of FY90. 11. The flow of PCRs from operational departments to OED continued to be heavily bunched in the last quarter of the fiscal year, when 57 percent of the annual total were received. With 118 PCRs the month of June alone accounted for 40 percent of all PCRs for the year. 3 12. The quality of PCRs continues to be monitored by OED staff with particular care in the transition to the new format and while the backlog is being reduced. About 60 percent of the PCRs received between May 1, 1989 and April 30, 1990 were in the new format. Twenty seven percent of the PCRs received during that period were judged to be unsatisfactory and had to be formally returned to the operational staff. This was less than in FY89 (35 percent). In most of the cases the rejected PCRs had presentational shortcomings such as missing prefaces or evaluation summaries, incomplete data sheets, or inappropriate format. Other reports showed substantive deficiencies which required clarification or additional analysis. It should be noted, however, that a large portion of the PCRs received in the above-mentioned period were submitted at the end of FY89 before the current guidelines had been issued, and that further instructions were issued by Operations in December 1989 when issues of noncompliance with the new guidelines had been recognized. As was indicated in the FY91 Work Program document, a joint review has been started with Operations to assess the experience with the new-style PCRs and the revised guidelines. PCR Reviews and Audits 13. PCRs are reviewed by OED staff within four weeks of receipt to ascertain their completeness and adherence to formal requirements as spelled out in the guidelines. They are then sent to the borrower for comment and issued to the Board, usually before a project performance audit is initiated. This delinking of PCR reviews from the audit process was started in January 1989, and became fully operational in FY90. OED's experience to date with the new procedure has on the whole been positive. To operational staff and the Board the process provides quick feedback of PCR findings. For OED it offers more flexibility for planning its audit work, and greater scope for clustering projects in meaningful sets. These benefits come at a price, however, as OED spends now about four staffyears annually on PCR reviews, compared to half that level two years ago. 14. During FY90, PCRs covering 283 projects were issued to the Board, of which 277 were on investment projects and 6 on sector adjustment operations. In addition, OED completed 134 performance audits covering 105 investment projects and 29 adjustment loans (Annex B). Those results exceeded, in varying proportions, all targets as revised at mid-year (from, originally, 225 PCR reviews, 90 investment, and 23 adjustment audits). As PCR reviews exceeded the revised target by a higher margin than audits, the audit ratio for investment projects works out to be 38 percent in FY90, slightly less than the minimum of 40 percent stipulated by the JAC. To make up for this shortfall, OED plans to achieve a higher audit ratio in the current year. 4 Table la PROJECT EVALUATIONS FY86 PY87 FY88 FY89 FY90 a/ PCRs received during PY 197 222 180 287 295 (272) of which adjustment operations 4 6 6 31 19 (30) PCR reviews by ORD 94 111 104 136 283 (250) investment projects 94 111 104 136 277 (250) adjustment operations 0 0 0 0 6 (0) Performance audits by OED 125 109 71 92 134 (125) investment projects 119 103 67 83 105 (104) adjustment operations 6 6 4 9 29 (21) Memo item: Audit ratio (excluding adjustment operations) 56 48 39 38 38 (42) a/ Figures in parenthesis are targets revised upwards in connection with the mid-year review. 15. All adjustment operations continue to be audited. The 29 audits completed in FY90 covered 13 SALs and 16 SECALs (see Annex B). This was three times as many as in FY89, bringing the total number of adjustment audits by the end of FY90 to 63. Another 10 SAL/SECAL audits are presently underway. 16. Fifty-seven percent of the projects were audited in clusters (the same proportion as last year). Many of the cluster audits were in agriculture and transport, DFC operations, and adjustment lending; fewer audits were clustered in energy, industry, public utilities, and urban development, where there were smaller numbers of operations within individual countries. Annual Review and Evaluation Studies 17. The Annual Review of Evaluation Results for 1988 was issued in November 1989; the Review for 1989 is being issued together with this report. In addition, OED completed nine evaluation studies, thus meeting the target set in the revised work program for FY90: * The World Bank and Senegal, 1960-87 5 * World Bank/Tanzania Relations, 1961-87 * Free-standing Technical Assistance for Institutional Development in Sub-Saharan Africa * Colombiat The Power Sector and the World Bank, 1970-87 * Pakistan The Aga Khan Rural Support Program, Second Interim Evaluation * Jamaica: Overview of Structural Adjustment Loans * Bangladesh: Review of the Experience with Policy Reforms in the 1980s * A Review of Procurement in Policy-Based Lending * World Bank Experience with Irrigation Development (Philippines and Thailand): Socioeconomic, Institutional and Technical Impact and Lessons. 18. Approach papers for ongoing studies approved by the JAC are listed in Annex C. DISSEMINATION AED USE OF EVALUATION FINDINGS 19. Reporting on the dissemination and use of OED findings, one of the functions of this Annual Report each year, is always a difficult task if one attempts to address substance rather than merely describing processes: there is no way for OED to account for all the initiatives that take place across the Bank to disseminate experience from all sources, and it would be improper as well as make little sense for OED to assess "its impact" on the quality of operations. What matters is that the cumulative experience of the Bank flow into new operations, not where it came from. 20. The impression that this report wishes to convey, by sharing the evidence available of good initiatives and practices, is that there is a growing awareness across the Bank of the need to reflect experience in new lending, as well as an increase in initiatives by users to collect experience and by others to arrange for its interpretation and dissemination across countries and regions. 6 21. FY90 was the first full year in whi-h the new operational guidelines on dissemination and feedback were applied. The guidelines reflect the recommendations of th5 Task Force on Dissemination and Utilization of OED Findings, chaired by PRE with representatives from Operations and OED, which e reconfirmed that operational staff should seek out OED's findings, cite them in project documents, and use them in future operations; * reinstituted formal management responses to major issues raised in OED's Annual Reviews and evaluation studies; * suggested the strengthening of informal networks that disseminate OED findings across the Bank; and * called on OED to further develop its evaluation products and make its findings more accessible to potential users. Absorbing the Lessons from Experience 22. Country department staff have increasingly called on OED staff to comment on papers and participate in discussions concerning sector strategies or issues and the early phases of project identification and preparation, and even in some cases to participate in peer reviews of new operations. These practices are welcome, at least on a trial basis, provided that they do not divert time and resources from evaluation work or other dissemination efforts with a potentially higher payoff. 23. Typical of the latter is the recent initiative of some country departments, as reportee in the OED Work Program document for FY91, to collect and use more systematically in their operations the collective experience of the Bank. In the Indonesia Department, particularly, experience relevant to new projects is drawn out in a background note at or before the inception of the project, and used to verify and adjust the project concept and its contribution to the sector and country assistance strategy. The experience statement is eventually reflected in a separate chapter of the SAR, enabling the management and Board to satisfy themselves about these aspects of the operation's quality. 1 Operational Directive 13.60, "Dissemination and Utilization of OED Findings", May 1989. 2 .Applying the Lessons of Experience: Report of the Task Force on Dissemination and Utilization of OED Finuings", JAC89-1, January 23, 1989. 7 24. Several such statements have been prepared since this practice was introduced last December, and two of them were used in the design of operations presented to the Board in FY90. The preliminary results of this experiment were reviewed by the Indonesia Department and OED. More work and lead time will be needed to fully develop the framework and to capture experience from all sources. OED is committed to help advise on design and to contribute experience from other countries and appropriate sectors. The implications for designing the agenda of issues to be reviewed in PCRs and audits prepared by the two departments will also be drawn out. The Indonesia initiative is an innovative and worthwhile pilot scheme. To what extent and in what form it can be replicated elsewhere in the Bank remains to be seen. 25. PRE's use of OED findings has been on the increase, as have its suggestions concerning the choice of topics for OED studies, and the design of approach papers. The Second Review of Adjustment Lending benefited from source material and comments provided by OED. This exchange of views also applies to a number of other studies carried out independently by OED and PRE, helping to enhance their complementarity (see para. 33). This cooperation can be particularly fruitful in areas where the experience of the Bank is still scant, such as the environment, and the role of women in development. In irrigation where, by contrast, the Bank has considerable experience, cooperation has proven highly valuable in bringing this experience to light and disseminating it. PRE-AGR has provided technical support to OED's impact studies in the sector and has taken the lead in disseminating their results as work progressed. AGR also led discussions of the findings of the current Annual Review of Evaluation Results in agriculture and initiated follow-up to the review with further analyses of the irrigation record. 26. There are numerous other channels of dissemination and feedback. The technical department in the LAC Region, for example, conducts annual reviews in which sector divisions summarize the principal findings and lessons from PCRs prepared during the previous calendar year. A meeting is then held to share these findings with regional staff and an OED representative. The department also discusses with country sector divisions the recommendations of OED impact evaluations that have broad regional relevance. Similarly, a country department in the Asia region reviews annually all PPARs pertaining to their projects, identifying issues and lessons that have general application for future operations. Other examples of dissemination and feedback in regional departments can be found in Annex E of this report. 27. Cooperation between the Economic Development Institute (EDI) and OED. As part of a new experiment, EDI staff have joined OED missions dealing with agricultural marketing and pricing, rural development, irrigation and structural adjustment to facilitate preparation of case study material for training. OED staff 8 continued to participate in EDI seminars on monitoring and evaluation and on project sustainability. EDI, for its part, is starting to provide training support to OED's Evaluation Capability Development Program. A regional seminar for the Maghreb countries, focused entirely on ex post evaluation, will be held under Moroccan auspices in Rabat with help from both EDI and OED. Management Responses 28. The new guidelines on dissemination and feedback provide for specific management responses to major OED studies, including Annual Reviews. These responses consist of (1) an assessment of the validity, significance, and relevance of the studies' findings, and (2) a plan of action for their use. The response to last year's tnnual Review took the form of a supplement to the ARIS report, establishing a direct link between the evaluation of past and ongoing operations. 29. One of the issues raised in last year's Annual Review that has been followed up subsequently is the Bank's methodology and practice of economic appraisal and evaluation of projects. During the second half of FY90 a series of steps were taken by the responsible Bank units to take an introspective look at the theory and practice of project analysis. A commitment was made by management to carry out this review in collaboration with OED and this was affirmed in the supplementary note to the ARIS report. Since then special reviews have been conducted, culminating in a workshop designed to (1) form a collective judgment on the major issues related to project analysis in the Bank, (2) develop a prioritized work program to improve the Bank's project analysis methods, and (3) specify responsibilities. The ultimate objective of this broad exercise is to enhance the quality of ex-ante and ex- post work at the Bank and by borrowers thereby strengthening the foundation on which expected project outcomes can be based and on which performance measures can rely. 30. Other management responses are reproduced in Annex E. For the third consecutive year, Bank managers have been asked to repo.; on the follow-up actions taken in response to specific OED evaluations of operational activities. The answers address issues in adjustment lending, procurement, irrigation development, country reviews, agricultural pricing policies, management of renewable resources in agriculture, and agricultural marketing. They give testimony to an active feedback process at work throughout the Bank with managers and staff responding to the findings from studies, audits, and PCRs. As in the previcus years, the issues raised in 3 "Supplementary Note to Fifteenth Annual Report on Implementation and Supervision, FY891, R90-17/1, February 2, 1990. 9 the responses have been carefully reviewed by OED and will be followed up with operational managers or within OED. ORD's Initiatives 31. OED is continuing its long-standing efforts to publicize its output, strengthen the dissemination value and analytical content of its products, improve their presentation and prcvide easier access to the growing stock of experience. 32. The OED Work Program document, which is discussed widely in the course of its preparation as well as at the Board, is also used increasingly year round to collect views for future topics of relevance and for opportunities for cooperation not only from managers and staff in the Bank but also with other evaluators in the donor community and in developing countries. 33. Approach papers for OED studies, before being submitted to the JAC, are circulated for comments to all concerned staff in Operations and PRE. Meetings are arranged routinely. In preparation for the approach paper for OED's second SAL overview, a series of seminars with staff and meetings with Bank managers were arranged, with the participation of a panel of senior consultants, to enhance the focus and design of the exercise and ensure complementarity with the past and future work of PRE. 34. Debriefing meetings are held and comments are sought on OED studies as they progress and well before they are finalized. Recent examples were the reviews of technical assistance in Africa, for which some follow-up was already reported by Regional staff; the review of the power sector in Colombia, widely circulated among country officials by Regional management in fall 1989, while being debated with staff internally, before it was finalized in June 1990; the study of the environmental impact of Bank projects in Brazil, which--in addition to the advice and support of PRE and the Region, including the Resident Mission--has benefited from technical support and active cooperation from the Brazilian authorities. The latter are preparing a separate report in parallel to OED's and have begun to take follow-up action on findings to date. This cooperation, under the aegis of SEPLAN, which is the agency responsible for reinstituting an evaluation system in Brazil, is also one of the practical ways in which OED plans to share experience with evaluation agencies as part of the Evaluation Capability Development Program (ECDP) (para. 47 below). The list above would be incomplete if it did not refer back to the active cooperation of AGR in the field of irrigation, as described in para. 25. 35. The Senegal country study was discussed with government officials and Bank staff, both of whom concurred with OED's findings and recommendations. At the suggestion of the JAC, the report was also reviewed by a donor conference in Paris arranged 10 by the country department. Subsequent Bank operations in Senegal, especially SAL IV, fully reflected the study's recommendations in their design. Similarly, the OED study on structural adjustment lending to Jamaica was widely disseminated among government officials, Bank staff, and other donors. 36. Audits have primarily served the purpose of accountability. Many drew lessons that were of relevance for repeater operations in the same sector and country, but not more general lessoy suitable for wider dissemination. The purpose of dissemination I s been served largely by OED studies (including the Annual Review), which aggregate and analyze the findings of audits and PCRs and supplement them with further study of the record. 37. Recently, OED has sought to draw more dissemination value out of the growing number of staff years that must be spent on audits--close to 19 in FY90, compared with the less-than 17 spent on studies and the Annual Review in the same year. Undertaking audits in clusters (explained in para 16 above) has contributed successfully to this aim. Some clusters have developed into studies. (Examples include the Jamaica study just referred to.) In fact, the longer lead time introduced into the work program and facilitatel by the delinking of PCRs also allows single as well as cluster audits to be planned as the building blocks of future studies, enriching the content of these studies and lowering their cost. This is the case with several FY90 audits, as well as the recent clusters of agricultural credits in Mexico and of agricultural extension and research projects in India. 38. The audit of fruit and vegetable projects in four Mediterranean countries was a unique attempt at clustering projects across countries. It was planned to complement a regional strategy paper on the subsector. Its focus on marketing was well received by the countries and two have decided to follow up on the audits' findings. Other audits that had an impact on the Bank's operations included clusters of industrial finance projects in the Philippines, and transport projects in Cameroon. This year's Subcommittee of the JAC has included two clusters in its review of PCRs and audits. 39. The preparation of this year's Annual Review evoked an unusually large number of comments both on the cohort of operations evaluated in 1989 and on the three main topics selected for detailed analysis this year. Meetings held on sector background papers prepared in connection with this Review were particularly well-attended and are being followed up with further analyses initiated by PRE, AGR and the Asia Region. 40. Two recent initiatives aim to continue improving the presentation of OED's output. Internal guidelines were recently issued to OED staff which are designed to upgrade the presentation 11 of evaluation summaries for audits and facilitate their reading by Executive Directors and staff. 41. Late in the fiscal year OED introduced a series of abstracts of evaluation findings, called OED Precis, for distribution to the Board and staff. The first topics are Trade Adjustment in Indonesia; Structural Adjustment in Jamaica; and Free-Standing Technical Assistance for Institutional Development in Africa. Access to ORD Findinus 42. OED is responsible for maintaining part of the Bank's institutional memory, and making it accessible to potential users. The department's library keeps a complete set of all evaluation reports; those can be read there (in their original form or on microfiche) by any Bank staff member. The reports are catalogued by sector, region, and report number in the Directory of OED Reports (fourth edition, September 1990) issued to Bank managers and staff. 43. OED also maintains a computerized text data base that contains summaries of all PCRs and PPARs. The existing system has been little used, however, partly because it requires cumbersome procedures both to input the summaries and to search for information contained in them. OED is exploring options for a more user friendly system that would give quick access to evaluation summaries to Bank staff without requiring them to be specially trained. This is in response to questions from the JAC and to increasing demand from operational staff for a ready supply of information that is potentially relevant to new operations. OED also aims to take advantage of advances in technology that have made text-based information systems cheaper and easier to create and use. External Dissemination 44. OED continues to make some of its major findings available to the development community at large, and to support borrower countries in building their own evaluation capability. 45. OED's Annual Reviews of Evaluation Results, which synthesize the findings of PCRs, PPARs, and special studies, are regularly published. The latest one, covering the projects evaluated in 1988, was published in April 1990; it is also available in French translation. In addition, the department published studies on renewable resource management in agriculture (November 1989) and on agricultural marketing (May 1990). A recent report to the Group of 24 on the role of the World Bank in agricultural development in the 1990s, relies extensively on the findings of OED evaluations. 12 46. At the donor level, OED participates, as an observer, in the deliberations of the OECD/DAC Expert Group on Aid Evaluation, whose role and impact in the international community has been steadily expanding in recent years. To help disseminate experience to developing countries, the Expert Group has sponsored a seminar on evaluation with the participation of representatives of Africa, Asia, and Latin America. A regional seminar on ex post evaluation in Africa was held in May 1990 in Cote d'Ivoire with the co- sponsorship of the African Development Bank. In addition, OED has started holding workshops and seminars with other evaluators in the international community to compare work programs and develop plans for cooperation. 47. OED has for many years responded to ad hoc requests for advice by borrowing countries and their agencies. In response to a perceived growth in demand it was decided in 1987 to develop a more structured approach and a capacity for following up on requests and supporting them over time. This Evaluation Capability Development Program (ECDP) has two broad goals: to help interested countries with ex post evaluation system development and to advise and support implementing agencies which form part of the system in preparing evaluation reports, including those for projects that the Bank has supported. During FY90, OED supported programs in Brazil, Morocco, and the West African Development Bank (BOAD) in Lomd. OPERATIONS EVALUATION DEPARTMENT Staffing 48. The department operated at near full strength for the entire year. The three vacancies which arose in FY90 (1 transfer and two retirements) were filled with broadly experienced staff. Two were recruited from within the Bank, and the third, an expert evaluator and manager, came from another international organization. 49. The new position authorized at mid-year, as well as two new positions in FY91, were promptly committed to new staff who will join OED by October, 1990. The new staff will strengthen OED in the areas of greatest recent growth--adjustment and agriculture operations. 13 Table 2: OPERATIONS EVALUATION STAFF FY86 FY87 FY88 FY89 FY90 Higher level staff Authorized positions 39 39 40 40 41 Staff beginning FY 34 38 35 38 40 Recruitment 9 8 4 4 3 Departures 5 8 1 2 3 Staff end FY 38 38 38 40 40 Vacancies end FY 1 1 2 0 1 Support staff Authorized positions 21 21 21 21 21 Staff end FY 20 21 20 21 20 Vacancies end FY 1 0 1 0 1 Use of Resources 50. OED's work plans had to be substantially revised at the beginning of FY90, reflecting the unexpected surge in PCRs in June 1989, well after the FY90 Work Program and Budget had been finalized. Priority was then given to increasing the number of PCR reviews and audits of investment and adjustment operations at the expense of the postponement of new initiatives--such as the abstract program--and studies still in their early stages. A mid- year budget adjustment, offsetting the costs of handling the higher volume of PCRs, was also used to catch up on the studies programs and other activities, including dissemination, guidelines and methodology. Plans for the Annual Review 1989 were adjusted to a lower level of expected resources--by limiting analyses to operations evaluated in calendar 1989, and by selecting three main topics (impact of irrigation, trade adjustment, and country reviews) on which OED had already done substantial study work, expecting a saving of close to one staffyear or 18 percent. 51. The revised budget total of $6.4 million was 9 percent higher than actual expenditures in FY89 (see Table 3). All of this was fully utilized. In addition, a higher use of staff and lower average consultant fees further increased the department's real resources so that in the end the equivalent of 56.7 staff-years had been used, eleven percent more than in FY89 (51.0 staff-years). 52. At year end, all major output objectives for FY90 had been achieved or exceeded. In the wake of the substantial disruptions 14 Table 3: ORD RESOURCES AND OUTPUT (Higher Level Staff-year Equivalent) FY87 FY88 FY89 FY90 Pavioed B-dge---- ---Act ual---------- Actual Budget Budget Sources Positions 39 40 40 41 41 Staff Years 37.2 36.2 39.5 41.0 40.0 Consultant Years 11.3 12.5 11.5 15.7 14.7 Total Resources 48.5 48.7 51.0 56.7 54.7 Financial Budget (in FY90 $000) (5,517) (5,888) (6,416) (6,417) Applications Adjustment Audits 1.7 0.3 3.9 5.3 6.8 In.estment Audits 8.6 7.7 8.3 13.6 9.6 PCR Reviews 0.9 1.3 2.4 3.9 4.3 Total Evaluations 11.2 9.3 14.6 22.8 20.7 Annual Rev. & Rept. 3.9 4.4 3.9 3.1 3.8 Studies 14.4 18.1 17.3 13.6 13.6 Dissemination 1.5 2.6 1.6 2.9 2.0 ECDP 0.3 0.7 0.4 0.9 1.0 Other activities a/ 3.2 2.7 2.2 2.2 2.5 Management & Admin. 5.7 3.7 3.9 4.0 4.0 Overhead b/ 8.3 7.2 7.1 7.2 7.0 Output Adjustment audits 6 4 9 29 21 Investment Audits 103 67 83 105 104 PCR Reviews 111 104 136 283 250 Studies 6 8 8 9 9 ---------------m- a/ Evaluation support to Operations, PPR and EDI, departmental retreats, methodology development and data management. b/ Leave and training. 15 during the year, however, the status of the pipeline of ongoing operations still requires -nalysis and some anomalies in unit costs (both low and high) need to be clarified, for the sake of good housekeeping rather than any serious implication for FY91. Use of Non-Bank Resources 53. The three-year Technical Cooperation Programme between OED and the Canadian International Development Agency (CIDA) is now approaching the end of its second year; two-thirds of the agreed resources (about US$525,000 equivalent) have been committed and almost US$400,000 have been disbursed. A wide variety of OED activities have benefited from the deepening and broadening support provided by the Programme including three audits (two of them adjustment operations) and eleven studies. The programme to date has been most helpful in helping increase the number of case studies or the amount of socioeconomic work in support of studies in OED's program contributing to the joint objective of CIDA and OED of adding new insights on the social and environmental impact of operations. The use of funds has been consistent with the guidelines and criteria agreed with the JAC to help maintain the independence of judgment of OED and the oversight of the JAC. It is proposed to continue the Programme for its third year. A final assessment will be prepared well before the end of the Programme's third year in connection with OED's FY92 Work Program. Consultations will then be held with CIDA and the JAC to determine whether the Programme will be continued or not. 54. Other inputs from non-Bank sources came from bilateral evaluation units who provided and funded their own staff for joint audits of six adjustment operations. No activities were funded under the Operations consultant's trust funds. OPERATIONS EVALUATION IN IFC 55. IFC's Operations Evaluation Unit (OEU), comprising two operations evaluation officers supported by two research assistants, exercises oversight over the preparation of Investment Assessment Reports by IFC, conducts special studies, and disseminates the findings of the operations evaluation system to IFC staff and others. Investment Assessment Reports 56. IFC's procedures call for preparation of Investment Assessment Reports (IARs) on at least 50 percent of its 16 investments. Pursuant to these procedures, IFC's Portfolio Committee discussed 32 IARs during FY90, four more than in FY89. OEU participated actively in these discussions and, in addition, sent to the Portfolio Committee and the DGO one-page notes on the comprehensiveness of the IARs reviewed in the preparation of special studies as well as a sample of other IARs. After discussion, the IARs are filed in the Office of the DGO, where interested Directors may review them (Annex F). 57. The FY90 IARs were generally useful. Most covered ventures that have had to deal, successfully or unsuccessfully, with major problems, and many will facilitate the preparation of OEU special studies. The IARs generally provided a good assessment of the performance of the ventures reviewed and identified some important lessons. Special Studies 58. OEU focuses its efforts on studies of groups of projects to identify underlying patterns and lessons that may be of broad application. The studies are intended to assist IFC in the selection, appraisal, structuring and supervision of its investments. During FY90, OEU submitted to the JAC and the Board a report on IFC's experience in the textile sector, and made substantial progress on two other studies to be completed in FY91- -one involving a second review of IFC's experience with hotel investments and the other covering IFC's experience with intermediaries providing financing to private business enterprises (see Annex G). Dissemination 59. To facilitate use of the findings of IFC's IARs (and earlier PCRs), OEU distributes to all IFC higher-level staff a list of these documents, classified by country and business sector, together with a summary of lessons suggested in the most recent year's IARs. Moreover, to encourage awareness of the findings of current IARs among the staff most likely to be able to use them, OEU notifies investment officers working on new proposals of Portfolio Committee meetings on IARs covering similar operations. The combination of the notices and the annual list has encouraged increasing enquiries by investment staff about the availability of relevant IARs and special studies. 60. To disseminate the findings of the special studies, OEU distributes to the Board, all IFC higher-level staff, and interested Bank staff, copies of the abridged versions of these studies (deleting company names and the individual case studies). The abridged versions are available for release to member governments and their agencies. OEU meets with individual departments at IFC and interested groups at the Bank to discuss the findings of these studies. 17 61. To disseminate the broader lessons that cut across sectors, OEU circulates to all higher-level IFC staff a comparative summary of findings of the completed studies with respect to factors contributing to good or poor management performance, assessments of markets and marketing, and project financing. In addition, OEU participates regularly in IFC's program of seminars for new staff members and has begun to provide new staff members with the list of documents and comparative summary of findings mentioned above. Furthermore, in FY90, as part of an expanded training program for new staff, OEU supervised the transformation of two OEU company reviews into teaching cases and the subsequent teaching of these cases by a consultant from a well known business school. This program was highly successful, and IFC is adding a third case for the next round of the training program. 18 CONCLUSIONS 62. During the past year, evaluation in the Bank has been further strengthened. Record numbers of PCRs and audits were prepared and the backlog of PCRs started being reduced. New initiatives for disseminating and using experience could be observed across the Bank. 63. In the current year, attention will be given to the following issues: * During the transition to new style PCRs and as the backlog continues to be drawn down, OED will continue to: monitor and provide feedback on quality to operational managers; and to participate in a joint review with Operations of the experience with the new PCR guidelines. * To make up for the recent shortfall in the audit ratio, OED aims to exceed the mandatory 40 percent for investment projects in the current year. * To further strengthen dissemination and feedback processes, ORD will continue tos - support country departments initiatives to make more systematic use of experience in new operations; - improve the quality of evaluation summaries; - prepare additional abstracts of audits and studies in the context of its Precis series; - contribute to improvements in the economic analysis of projects, in cooperation with Operations and PRE. 19 ANNEX A OPERATIONS EVALUATION IN THE WORLD BANK 1. Established in 1970, the evaluation function has evolved with the Bank's operational activities. Its basic purpose is to assess the effectiveness and efficiency of Bank-supported development projects and programs, and to feed the evaluation findings back into operational objectives, policies and procedures. In its present form, the operations evaluation system is firmly rooted in the Bank's institutional fabric, with clear allocations of responsibilities and proven processes and instruments. The system provides the Bank's shareholders with full accountability on past operations - strengthened by the involvement of independent evaluators - while at the same time assisting operational managers in sharpening program objectives and informing them about relevant experience. The Director-General 2. The Director-General, Operations Evaluation (DGO) oversees the Bank's evaluation function. He is supported by the Operations Evaluation Department (OED). The DGO is directly responsible to the Executive Directors of the Bank fort * appraising the Bank's operations evaluation system and reporting on its adequacy for the Bank and member governments; * assessing whether the Bank's programs and activities are producing the expected results; * distilling lessons from the growing stock of experience and translating them into actionable recommendations; * reporting to the Board on the effectiveness of dissemination and feedback processes, and * assisting member countries to develop their own operations evaluation systems. 20 Annex A 3. The DGO reports to the Board and has an administrative link to the President. He is directly appointed by the Board for a fixed time period. To further ensure his independence, he may not join the Bank staff upon termination, except in unusual circumstances. The Operations Evaluation Department 4. The Operations Evaluation Department (OED) is the staff arm of the Director-General. It is headed by a director appointed by the DGO. The principal functions of OED are to: * assist the Director-General in making independent assessments of the adequacy and effectiveness of the operations evaluation system in the light of the objectives and programs of the World Bank; * carry out performance audits on completed projects and conduct evaluation studies focusing on selected issues and sectors; * maintain the Bank's institutional memory and make it accessible to potential users within and outside the Bank. 5. To discharge these functions, the Operations Evaluation Department has unrestricted access to the staff and records of the Bank. Without compromising its independence, OED staff endeavors at all times to maintain close and continuing contact with Bank staff so that their views are adequately considered in OED reports, and the analyses and findings of these reports are fully understood. International Finance Corporation (IFC) 6. IFC's Economics Department is responsible for the operations evaluation function within IFC. It carries out this function through an Operations Evaluation Unit (OEU), under the general oversight of the DGO. OEU maintains close relations with OED, consulting with OED staff at various stages of the evaluation process and observing OED's criteria for independent evaluation. 21 Annex A 7. The purpose of operations evaluation in IFC is broadly: * assessing whether IFC's programs and investments are producing the expected re3ults; and * feeding these findings back into recommendations which will help improve the performance of IFC's investments. Joint Audit Committee 8. The Joint Audit Committee (JAC) of the Board of Executive Directors oversees, inter alia, the operations evaluation procedures and activities and reports annually to the full Board on evaluation performance and output. It reviews OED's and OEU's annual work programs, approach papers for proposed studies, the DGO's Annual Report on Operations Evaluation and all studies prepared by OED and OEU. In addition, a subcommittee of the JAC reviews a select number of project evaluation reports and assesses the adequacy of the project evaluation system. Processes and Instruments 9. Operations evaluation in the Bank is conducted in three stages: (i) evaluation of ongoing operations through project supervision, built-in monitoring and evaluation (M&E), and the Annual Review of Implementation and Supervision (ARIS); (ii) ex- post evaluation through PCRs, performance audits, and special studies, including OED's Annual Review of Evaluation Results; and (iii) dissemination and use of evaluation findings. 10. The Bank's project supervision system provides the first opportunity for self-evaluation by the relevant operational units. In addition, most loan/credit agreements now include provisions that commit borrowers to monitor the progress of projects and to evaluate their results and impact. Experience gathered through supervision and M&E is then periodically assembled and analyzed under the ADIS umbrella, and its results are reported annually to the Board. 4 Fifteenth Annual Report on Implementation and Supervision - Fiscal Year 1989; R90-17; January 30, 1990. 22 Annex A 11. Ex-post evaluation starts with the PCR which is now prepared by the last supervision mission, in collaboration with the borrower, to ensure currency of evaluation findings and participation of responsible Bank/borrower staff. All PCRs are then reviewed by evaluation officers in OED to ascertain their consistency and completeness, and submitted to the Board by the DGO. Subsequently, a selected number of operations are audited by OED staff, preferably in clusters to enhance the value from comparability. Special topics are pursued in greater depth through evaluation studies, most of which are conducted by OED# some by PRE or Operations. OED's Annual Review synthesizes the results of operations evaluation, thus providing a comprehensive assessment of the Bank's operational effectiveness. 12. Learning from experience through dissemination and feedback involves three distinct phases: first, the evaluation findings are conveyed to potential users (dissemination); subsequently, these findings are incorporated into the Bank's policies, procedures and operations (feedback); finally, the actions taken in response to evaluation findings are reported back to the Board, thus closing the loop of the lesson learning process. 23 Annex 8 OPERATIONS EVALUATED DURING FY96 I. PROJECT PERFORMANCE AUDIT REPORTS Loan/ Report Report Credit No. No. Date AdJustment Lendtna 1.* Jamaica Structural Adjustment Loan II Ln2815 9018 11-Aug-69 2. Structural Adjustment Loan III Ln2478 8. Brazil Export Development Project Ln2847 827 12-Jan-90 4. Senegal Second Structural Adjustment Credit Cr1656 695 81-May-9U S. Third Structural Adjustment Credit Crs02 0. Pakistan Energy Sector Loan Ln2552 6726 12-Jun-96 7. Uganda Second Reconstruction Program Cr1252 6751 18-Jun-9 8. Third Reconstruction Program Cr1474 9. Uruguay Agricultural Sector Plan Ln2488 86 29-Jun-90 10. Guinea First Structural Adjustment Credit Cr1859 6826 25-Jun-90 11. Gamble First Structural Adjustment Credit Cr1780/A022 8627 25-Jun-90 12. Togo Second Structural Adjustment Cr1599/A02 682 25-Jun-90 18. Zaire Industrial Sector Adjustment Cr1768/AO19 88 28-Jun-90 14. Chile Structural Adjustment Loan I Ln2825 6651 26-Jun-9 15. Structural Adjustment Loan II Ln2787 18. Structural Adjustment Loan III LN2892 17. Kenya Agriculture Sector Adjustment Cr1717/A921 6662 29-Jun-90 16. Bangladesh Imports Credit IX Cr1@71 6671 29-Jun-90 19. Imports Credit X Cr1194 20. Imports Credit XI Cr18a8 21. Imports Credit XII Cr471 22. Imports Credit XIII Cr1655 28. Cote d*Ivoire Structural Adjustment Loan II Ln2882 8872 29-Jun-90 24. Structural Adjustment Loan III Ln2711 25. Pakistan Export Development Ln2701 8878 29-Jun-90 * Numbers count operations evaluated. Clustered evaluations can be determined from the number of operations listed for a given report number. 24 Annex B Loan/ Report Report Credit No. No. Date 28. Mexico First Export Development La2881 8876 29-Jun-90 27. Second Export Development Ln2777 28. Malawi Structural Adjustment Loan III Cr1S44/A909 8888 29-Jun-90 29. Zimbabwe Manufacturing Export Promotion Ln2289 8888 29-Jun-90 Agriculture 80. Kenya Smaltholder Coffee Improvement Project Cr914 9814 29-Dec-49 81. Kenya Wildlife and Tourism Project Ln1864 8448 16-Mar-9 82. Nigeria 8ida Agricultural Development Project Ln167 8557 28-Apr-96 83 Ekiti-Akoko Agricultural Development Project Ln1854 34. Bangladesh Muhurl Irrigation Project Cr725/1504 8854 16-May-96 35 Low Lift Pump Project Cr999 86. Nepal Grain Storage Project Crl162 8669 29-May-90 87. Nepal Sunsarl Morang Irrigation and Drainage Project Cr812 8670 29-May-9 88. Nigeria Imo Oil Palm Project nI191 8672 25-May-96 89. Rivers Oil Palm Project ln591 40. Cameroon Zapi Intergrated Rural Development Project Cr778 8696 01-Jun-90 41. Western Highlands Rural Development Project Cr784 42. Northern Province Rural Development Project Cr1078/Lnl919 48. Kenya Narok Agricultural Development Cr858 8796 29-Jun-96 44. Baringo Pilot Semi-arid Zones Cr982 45. Bangladesh Drainage A Flood Control Cr864 8865 29-Jun-96 48. India Bengal Agricultural Extension and Research Project Cr896 888 29-Jun-96 47. Bihar Agricultural Extension and Research Project Cr781 48. National Agricultural Research Project Cr855 49. Kerala Agricultural Extension Project Cr1028 so. Tamil Nadu Agricultural Extension Project Cr1187 51. Maharashtra Agricultural Extension Project Cr1186 52. India Utter Pradesh Social Forestry Project Cr925 8809 28-Jun-96 58. India Gujarat Community Forestry Project Cr961 8810 28-Jun-96 54. Zimbabwe Small Fare Credit Cr1291 8856 29-Jun-96 25 Annex B Loan/ Report Report Credit No. No. Date 5. Mexico Fifth Agriculture Credit Ln1217 8869 29-Jun-96 Se. Sixth Agriculture Credit Ln1569 67. Seventh Agriculture Credit Ln1891 58. Eighth Agriculture Credit Ln2454 59. Agricaltural Credit Ln2616 89. Kenya Agrieviture Technical Assistance Cr1277 8862 29-Jun-90 81. Kenya Bure Irrigation Settlement Cr722/Lnl449 8865 29-Jun-90 82. Mexico Rainfed Agricultural Development LP1946 8880 29-Jun-98 63. Lao Dem. Rep. Agricultural Rehabilitation Cr760 8884 29-Jun-98 64. Second Agricultural Rehabilitation Cr924 65. Third Agricultural Rehabilitation Cr1921 66. Turkey Second Fruit & Vegetable U1967 8885 29-Jun-96 67. Morocco Vegetable Marketing & Production Ln1767 68. Greece Vegetable Production & Marketing W1588 69. Cyprus Fruit & Vegetable Export Ln1988 Human Resources 79. Bolivia Emergency Social Fund Project Cr1829 8449 29-Mar-86 71. Malawi Third Education Project Cr910 8489 30-Mar-90 72. Fourth Education Project Cr1123 78. Philippines Fishery Training Project Ln1786 8788 25-Jun-90 74. Lesotho Third Education Project Cr1148 8855 29-Jun-90 75. Pakistan Third Education Project CrG78 8859 29-Jun-9 76. Primary Education Project Cr892 77. Vocational Training Project Cr11@9 78. Morocco Fourth Education Project Ln1881 8863 29-Jun-98 79. Tunisia Education IV Ln1961 8864 29-Jun-98 80. Korea Firet Population Ln1774 8895 29-Jun-90 81. India Second Population Cr981 8898 29-Jun-98 DFC 82. Zambia Second Development Bank of Zambia Project Ln1923 8959 18-Sep-89 88. Morocco Ninth Loan to Banque Nationale pour le Ln2637 8132 28-Oct-09 Developpement Economique 26 Lean/ Report Report Credit -.. NO. Date 64. Turkey Deviet. Yatirle Bankast II 1,1879 8614 04-May-90 65. Devlet Yaetirim Bankast III LW1998 88. Philippines Private Development Corporation of the Philippines IV LWlO2 6761 14-Jun-90 8?. Private Development Corporation of the Phlippinee V L1S14 88. Development Bank of the Philippines III Ln1572 89. Second Small and Medium Industries Ln1727 9. Industrial Finance Lnl984 91. Third Small and Medium Industries Ln21G9 92. Korea Development Bank Project II Lal8 849 2W-Jun-90 98. Development Bank Project III Ln1574 94. Development Bank Project IV LW1983 95. India Industrial Credit and Investment Corporation Ln48 885U 28-Jun-90 of India XIII 96. Industrial Credit and Investment Corporation Ln2051 of India XIV Industry 97. Tanzania Morogoro Textile Project LM667/Cr888 6698 81-May-96 98. Morocco Pilot Project for Small-Scale Mining Ln2199 6755 15-Jun-90 99. Tanzania Morogoro Industrial Complex Ln1865/1888 868 29-Jun-90 186. Brazil Carajes Iron Ore Ln2196 8869 29-Jun-9 Eneray 101. Pakistan Second Toot Oil and Gas Development Project Ln2874 6468 16-Mar-90 162. Morocco Petroleum Exploration and Appraisal Project Ln2271 640 64-May-90 18. Tunisia Second Natural Gas Pipeline Project LA64 6754 18-Jun-90 184. Brazil Alcohol A Biomass Energy Development Ln1989 6667 29-Jun-90 power 165. Philippines Seventh Power Lnt40 86574 20-Apr-90 166. Botswana Power I Ln2209 6621 26-Jun-9U I8. Nigeria Lagos Power Distribution Project Ln1766 622 29-Jun-90 168. Power Transmission and Distribution Project n2086 169. Sweatland Third Power Project LaO 8823 29-Jun-, 27 Annex 8 Loan/ Report Report Credit No. No. Date Technical Assistance 110. Uganda First Technical Assistance Project Cr177 6751 13-Jun-g0 111. Cot. d'Ivoire Technical Aseeitance Project Ln2@59 6626 25-Jun-90 Transportation 112. Thailand Provincial Roads Project Ln2085 6447 18-Mar-90 118. Brazil Sixth Highway Project La1667 8461 28-Mar-9U 114. Cent. Africa Rep. Third Highway Cr647/174 8589 18-Apr-9 116. Fourth Highway Cr1256 118. Zimbabwe Transport Rehabilitation Importe Program Ln1994 8619 04-May-9U 117. Togo Third Highway CrO98 868 29-Jun-90 116. Feeder Roads Cr610 119. Fourth Highway Cr1189 126. Yugoslavia Fourth Railway Lnit28 6694 29-Jun-U 121. Fifth Railway Ln1584 122. Kosovo Railway Ln1977 128. Congo Second Railway Lo1228 669 29-Jun-U9 Urban 124. Brazi Medium-Sized Cities Project Ln1720 6802 29-Dec-89 125. Cot. d'Ivoire Second Urban Development Project Ln2648 6757 68-Jun-9 126. Nigeria (Bauchi) Urban Development Project Ln1767 8756 98-Jun-90 127. Botewane Francletown Urban Development Project Cr471 6629 25-Jun-90 128. Second Urban Development Project Lnl584 129. Cameroo Fi ret Urban Project La2244 681 25-Jun-9 Water SuDDl & Sewerae 18. Korea Firet Water Supply Project Ln2072 8174 86-Nov-89 181. Indonesia Second Water Supply Project La17G9 8622 89-Apr-9 182. Morocco Second Water Supply Project Ln1724 6725 8-Jun-90 188. Algeria Algiers Sewerage Project Lnl45 6772 21-Jun-9 184. Madagascar Antananarivo Water and Sanitation Project CrI02 6679 29-Jun-9 28 Annex 8 II. PROJECT COMPLETION REPORTS Loan/ Report Report Credit No. No. Date Adlustment Londing 1.* India Drought Assistance Project Ln2886/Cr1852 8145 31-Oct-89 2. Nigeria Fertilizer Loan Ln2845 8499 80-Mar-90 3. Mail Small Holder Fertilizer Project Cr1852 8534 18-Apr-90 4. Philippines Agricultural Sector/Inputs Project Ln2489 8680 28-May-90 5. Kenya Agricultural Sector Adjustment Operation Cr1717/AG21 8797 22-Jun-90 6. Uruguay Agricultural Sector Loan Ln2488 8808 29-Jun-90 Agriculture 7. Yugoslavia Bosnia-Herzegovina Agricultural Development Project Ln2136 7989 24-Jul-89 8. Laos Second Agricultural Rehabilitation and Devel. Project Cr924 8084 18-Oct-89 9. Papua New Guinea Enga Provincial Development Project Cr1227/Ln2125 6104 20-Oct-89 10. Indonesia Smallholder Coconut Development Project Ln1898 8185 24-Oct-89 11. Yugoslavia Croatia Sava Drainage Project Ln1756 8136 24-Oct-89 12. Nepal Grain Storage Project Crl162 8187 25-Oct-89 13. Turkey Second Fruit and Vegetable Project Lnl987 8157 08-Nov-89 14. Brazil Lower Sao Francisco Irrigation Project Ln1729 8158 8-Nov-89 15. Brazil Second Agricultural Research Project Ln2016 8162 08-Nov-89 18. Thailand Second Tree Crops Project Ln2078 8178 15-Nov-89 17. Indonesia Nucleus Estates and Smaltholders III Project Ln1751 8188 21-Nov-89 18. India Jamme and Kashmir Horticulture Project Cr806 8187 21-Nov-89 19. Indonesia Sixteenth Irrigation Project Ln2118 8199 27-Nov-89 20. Viet Nam Dau Tiong Irrigation Project Cr845 8239 11-Dec-99 21. Thailand Mae Klong Irrigation Project-Malaiman Phase I Ln2022 8240 11-Dec-89 22. Indonesia Transmigration III Project Ln2248 6241 31-Dec-89 23. India Maharashtra Agricultural Extenaion Project Cr1135 8258 29-Dec-89 24. Kerala Agricultural Extension Project Cr1S28 25. Tamil Nadu Agricultural Extension Project Cr1a7 26. Sri Lanka Kurunegala Rural Development Project Cr891 8276 81-Dec-89 27. Turkey Northern Forestry Project Lnes 8274 22-Dec-89 28. Korea Fourth Agricultural Credit Project Ln2649 8275 22-Dec-89 29. Mexico Rainfed Agricultural Development Project Ln1945 8835 02-Feb-90 80. Indonesia Second Provincial Irrigation Development Project Ln2875 8338 26-Jan-90 31. Zaire Kwango-Kwilu Technical Asaistance Project Cr1152 8867 07-Feb-90 82. Burkina Faso Forestry Project Cr982 8858 SS-Feb-90 3. Kenya Second Intergrated Agricultural Development Project Cr959 8361 03-May-90 84. Yemen Arab Rep. Southern Upland Regional Development Project Cr1687 8364 81-Jan-90 35. India Kandi Watershed and Area Development Project (Punjab) Ln1897 8366 09-Feb-90 86. Nigeria Agricultural and Rural Management Training Institute Ln1719 838 14-Feb-90 * Numbers count operations evaluated. Clustered evaluations can be determined from the number of operations listed for a given report number. 29 Annex 8 Loan/ Report Report Credit No. No. Date 87. Brazil Mines Gerals Second Rural Development Project Ln1877 388 09-Feb-90 88. Ceara Second Rural Development Project Ln1924 89. Philippines National Irrigation Systems Improvement Project I Ln1414 8890 28-Feb-99 48. Syrian Arab Rep. Southern Regional Agricultural Development Project Ln2124 8418 28-Feb-90 41. Syrian Arab Rep. Lower Euphrates Drainage Project LnI682 8419 28-Feb-90 42. Cameroon Northern Province Rural Development Project Cr175/Lnl919 8420 28-Feb-90 48. China Hellong]lang Land Reclamation Project Ln2281/Crl347 8448 08-Mar-90 44. Pakistan Agricultural Research Project Cr1158 8444 28-Feb-90 45. India Second Nat'I Cooperative Development Corporation Project Cr1146 4568 15-Mar-90 48. Korea Second Agricultural Products Processing Project LnlSl 6459 15-Mar-90 47. Madagascar Mangoky Agricultural Development Project Cr861 8460 21-Mar-90 48. Colombia Upper Magdalena Pilot Watershed Management Project L2069 6481 30-Mar-90 49. Brazil Plaul Rural Development Project LnelS 8466 26-Mar-90 50. Kenya Bure Irrigation Settlement Project Cr722/Lnl449 8498 30-Mar-90 51. Romania Orchards Project Ln176 8588 18-Apr-90 52. Fourth Livestock Project Ln1987 58. Moldova Agriculture Credit Project Ln2677 54. Romania Mostistes and Calmatul Irrigation and Drainage Project LnIS70 8509 18-Apr-90 5. Covurlui Irrigation Project Ln1795 56. Buceant-Buzav-Siret-Prut (BBSP) Irrigation Project Ln1988 57. Caracal-Titu Irrigation Projects Ln1971 568. Tanzania Small Holder Tea Consolidation Project Cr1687 8510 18-Apr-90 69. Indonesia Nucleus Estate and Smallholders IV Project Lnl8S 6511 10-Apr-90 60. Turkey Fourth Livestock Development Project Ln56 8512 04-Apr-90 61. Nigeria Ekiti-Akoko Agricultural Development Project Ln1SS4 8528 10-Apr-90 62. China Agricultural Education and Research Project Cr1297 8580 18-Apr-90 68. Peru Sixth Agricultural Credit Project Ln2898 8581 16-Apr-90 64. Turkey Fifth Livestock Development Project Ln1862 6588 12-Apr-90 65. Malawi Smallholder Fertilizer Project Cr1852 8584 18-Apr-90 8. India Subernarekha Irrigation Project Cr1289 8585 18-Apr-90 87. Colombia Irrigation Rehabilitation I Project LnI988 6542 17-Apr-90 68. Papua New Guinea Agricultural Support Services Project Ln2276 6552 24-Apr-90 89. Bangladesh Muhuri Irrigation Project Cr725 8555 28-Apr-90 70. Korea Agricultural Wholesale Marketing Project Ln2lll 8556 28-Apr-90 71. Somalia Agricultural Extension A Farm Mgt. Training Project Cr905 8579 30-Apr-90 72. Nepal Second Rural Development Project Mahakall Hills Cr989 8580 89-Apr-90 78. Bangladesh Agricultural Credit Project Cr1147 8581 3-Apr-90 74. Zambia Eastern Province Agricultural Development Project Ln2691 8562 30-Apr-90 75. Thailand National Agricultural Extension Project - Phase II Ln1752 8616 04-May-90 76. Tanzania Second Cashownut Development Project Cr601 025 08-May-90 77. Tanzania Coconut Pilot Project CrI070 8626 08-May-90 78. Ethiopia Grain Storage and Marketing Project Cr789 6827 08-May-90 79. India Orissa Irrigation II Project Cr1897 8628 08-May-90 80. Kenya Second Integrated Agricultural Development Project Cr959 8681 11-May-90 61. Egypt First Agroindustries Project Cr986 887 14-May-90 82. Malawi National Rural Development Program (NRDP)-Phase III Cr1188 8688 22-May-90 88. Ethiopia Drought Recovery Program Cr1576 6889 21-May-90 64. Malaysia FELCRA I Project Ln2618 6658 18-May-90 30 Annex 8 Loon/ Report Report Credit No. No. Date 6. Yemen, POR Wadi Selhan Agricultural Development Project Cr1145 8681 17-May-99 86. India Kernataka Tanks Irrigation Project Cr1116 8668 29-May-99 87. Nepal Sunsarl Morang Irrigation and Drainage Project Cr812 8670 29-May-96 88. Honduras Third Agricultural Credit Project Ln2284 8707 31-May-91 89. Ethiopia Revised Amib*ra Irrigation Project Cr767 8708 31-May-96 90. Cyprus Agriculture Research and Extension Project Ln25 9 8752 88-Jun-90 91. India National Agricultural Research Project Cr855 8768 16-Jun-96 92. Ghana Volta Region Agricultural Development Project Cr1089 8765 10-Jun-91 98. Togo Second Cocoa/Coffee Development Project Cr945 8789 20-Jun-90 94. Pakistan Irrigation Systems Rehabilitation Project Cr1289 8794 25-Jun-90 95. Kenya Narok Agricultural Development Cr858 8795 29-Jun-90 98. Baringo Pilot Seml-arid Zones Cr962 97. Western Samoa Agricultural Development II Project Cr1080 8791 28-Jun-91 98. Peru Puno Rural Development Project W1812 8802 29-Jun-90 99. India Andhra Pradesh Irrigation and CAD Composite Project Lnl261 8804 26-Jun-90 100. Zimbabwe Small Farm Credit Cr1291 8856 29-Jun-986 Human Resources 101. Korea First Population Project Lnl774 8114 25-Oct-89 102. Syrian Arab Rep. Second Education Project Ln1976 8182 17-Nov-89 108. Egypt Third Education Project Cr1069 8188 17-Nov-89 104. Malawi Fourth Education Project Cr1128 8195 27-Nov-89 106. Malawi Third Education Project Cr910 8198 27-Nov-89 106. Pakistan Vocational Training Project Cr1109 8212 11-Dec-89 187. Botswana Second Education Project Ln828 82386 22-Dec-89 108. Botswana Third Education Project Ln2O57 6287 22-Dec-69 109. Lesotho Third Education Project Cr1148 8288 22-Dec-89 110. Yemen Arab Rep. Third Education Project Cr915 8257 29-Dec-69 111. Brazil Northeast Basic Education Project Ln18S7 826 29-Dec-89 112. Comores First Education Project Cr1195 8816 05-Jan-90 118. Bangladesh First Vocational Project Cr912 6354 66-Feb-90 114. Bangladesh Fourth (Primary) Education Project CrI54 8855 06-Feb-90 115. Thailand Sixth Education Project Ln2178 8868 12-Feb-90 116. Indonesia First Phase of a University Development Program W1904 8421 8-Mar-90 117. Tunisia Fourth Education Project O1981 8441 16-Mar-91 118. Morocco Fourth Education Project Ln1881 8442 16-Mar-98 119. Bolivia Emergency Social Fund Project Cr1829 8449 20-Mar-90 120. Brazil Northwest Reg. Int. Dev. Prog.-First Phase-Health Proj. Ln20681 8489 19-Mar-98 121. Sudan Third Education Project Cr1451 8529 18-Apr-90 122. China Africa Education and Research Cr1297 8589 18-Apr-91 128. Peru Higher Agricultural Education Project Ln2208 8658 25-May-91 124. El Salvador Fourth Education Project Ln1788 8859 25-May-96 126. Jamaica Third Education Project (Technical and Vocational) LW2079 6718 08-Jun-90 126. Barbados Education Project Ln1842 8787 25-Jun-91 127. Philippines Fishery Training Project LnI786 8788 26-Jun-91 128. Mauritania Second Education Project Cr1214 8861 29-Jun-90 129. India Second Population Cr981 8896 29-Jun-90 DFC 180. Philippines Fifth Private Devel. Corp. of the Philippines Project LWl614 7997 28-Jul-89 181. Haiti Industrial Credit Project Cr1181 6061 22-Sep-89 31 Annex B Loan/ Report Report Credit No. No. Wi- 182. Argentina Industrial Credit Project Ln48 8229 29-Dec-89 188. Nicaragua Financier* do Proinverslon Preinvest. Fund Project Cr1061 888 26-Jan-90 184. Mali Industrial Development Project Cr986 8401 28-Feb-90 186. Liberia Small and Medium-Scale Enterprise Project Cr1l6 6471 28-Mar-00 180. Bangladesh Third Small Scale Industry Project CrIe65 8478 28-Mar-960 187. Indonesia Third Small Enterprise Development Project Ln248O 8525 16-Apr-90 188. Morocco Second Small-Scale Industry Project Ln2O88 8548 28-Apr-960 189. Egypt Mier Iran Development Bank Project Ln1842 8618 64-May-90 140. Korea Fourth Small and Medium Industry Bank Project Ln2OO4 8616 04-May-960 141. Second Citizens National Bank Project Ln2144 688 14-May-960 142. Ghana Second National Investment Bank Project Cr901 8717 6-Jun-90 148. China First Industrial Credit Project Ln2226/Cr1818 8780 14-Jun-60 144. Rwanda Third Banque Rwandale do Developpement Project Cr1844 8799 28-Jun-90 145. Second Banque Rwandalso de Developpement Project Cr898 148. Tunisia Small Scale Industries I Ln1969 8817 28-Jun-90 147. Benin Industrial Development Cr997 8887 28-Jun-90 148. Korea Development Bank Project II Ln1888 849 26-Jun-90 149. Development Bank Project III Ln1574 150. Development Bank Project IV Ln1988 161. Ecuador Fifth Development Banking Project n26 6670 29-Jun-960 Industry 152. Philippines Textile Sector Restructuring Project Ln2127 8167 06-Oct-89 158. Egypt Agricultural and Industrial Imports Project Cr524/Lnl0S2 8118 20-Oct-89 164. Bangladesh Fertilizer Industry Rehabilitation Project CrlO28 818 20-Nov-89 155. Bangladesh Chittagong Urea Fertilizer Project Cr1204 8189 26-Nov-89 156. Spain Industrial Research, Development and Engineering Project Ln1425 8816 29-Dec-89 157. Senegal Phosphate Industry Development Engineering Project Cr1889 8807 29-Dec-89 158. Myanmar Construction Industry Project Cr1258 8878 12-Feb-90 159. Tunisia Industry IV (SOFOMECA Foundry) Project LaWaWl 8458 21-Mar-90 160. Portugal Mechanical Industries Project Ln1874/1875 6524 16-Apr-90 161. Mexico Small and Medium Scale Mining Development Project Lnl820 8688 17-May-960 162. Egypt EI-Dikholls Reinforcing Bar Project Ln2280 8677 29-May-90 168. Korea Small and Medium Machinery Industry Project kn2215 8678 29-May-90 184. Malawi Wood Industries Restructuring Project Ln2486 8744 18-Jun-90 165. Brazil Carajas Iron Ore Ln219S 869 29-Jun-90 Energ' 166. Honduras Petroleum Exploration Promotion Project W1881 8010 01-Aug-89 167. Cote d' Ivoire Petroleum Exploration and Development Project Ln2189 8021 20-Nov-89 18. Jamaica Petroleum Exploration Project Ln2O17 89 28-Sep-89 169. Panama Energy Planning and Petroleum Exploration Project LW1954 8125 17-Oct-89 178. Liberia Petroleum Exploration Promotion Project Ln1907 8126 17-Oct-89 171. Petroleum Sector Technical Assistance Project .r168S 172. Bolivia Gas and Oil Engineering Project CrSO25 8127 17-Oct-89 178. Zambia Petroleum Exploration Project L2152 8175 6-Nov-89 174. Zimbabwe Petroleum Fuels Supply Technical Assistance Project Cr1296 81 29-Dec-89 175. Pakistan Second Toot Oil and Gas Development Project Ln2374 8806 29-Dec-89 176. Benin Petroleum Sector Technical Assistance Project Cr1207 8877 68-Feb-90 177. Thailand Mae Moh Lignite Project Ln1852 6620 8W-Apr-90 178. Yemen Arab Rep. Petroleum and Geothermal Exploration Promotion Project Cr1216 8664 17-May-90 32 Annex B Loan/ Report Report Credit No. No. Date 179. Costa Rica Petroleum Sector Technical Assistance Project kn2619 8874 26-May-90 189. Indonesia Coal Mining 2ad Transportation Engineering Project LnSO49 8820 29-Jun-90 181. Bukit Assa Coal Mining Devel. & Transportation Project Ln2O79 162. Brazil Alcohol A Biomass Energy Development Project LnOS 8887 29-Jun-980 188. Philippines Petroleum Exploration Promotion Project Ln2262 8890 29-Jun-90 184. Philippines Petroleum Exploration Promotion Project L2201 8691 29-Jun-90 Power 185. Lao POR Nam Ngue Hydroelectric Project Cr1197 8874 N-Feb-9 188. Pana"a Fourth Power Project Lal47O 8878 06-Feb-90 187. Gamble First Energy Project Cr1187 8378 06-Feb-90 188. Korea Churgj1 Multipurpose Project LWN680 879 18-Feb-90 189. Portugal Seventh Power Project Ln2240 8472 28-Mar-9 190. Philippines Seventh Power Lnl486 6574 20-Apr-90 191. Barbados Power Project Lnl940 8821 8S-Apr-9 192. India Ramagundam Thermal Power Project Cr874/Lnl848 8841 07-May-9 198. Indonesia Eleventh Power Project L12658 8701 01-Jun-90 194. Egypt Shoubrah El Khele Thermal Power Cr985/Ln1788 8819 26-Jun-90 195. Third Power Cr1052/Ln1888 198. Power Supplement Ln1888-1 197. Botswana Power I Ln2209 8821 28-Jun-9 198. Nigoria Power Transmission and Distribution Project LW2685 8822 29-Jun-9 199. Lagos Power Distribution Project Ln1788 200. Swaziland Third Power Project Ln2OO9 8828 29-Jun-98 201. Philippines Geothermal Exploration Ln2208 8889 16-Jun-90 202. Kenya Olkarla Geothermal Engineering LnSO12 8892 29-Jun-9 203. Olkaria Geothermal Power Ln1799 204. Olkaria Geothermal Power Expansion Lnt287 Technical Assistance 205. Papua New Guinea Petroleum Exploration Technical Aseistance Project Cr1279 S869 19-Sep-89 206. Sudan Second Technical Assistance Project Cr1158 8158 08-Nov-89 207. Bangladesh Fourth Technical Assistance Project Cr1124 6154 08-Nov-69 208. Tanzania Third Technical Assistance Project Cr1206 8822 12-Jan-90 209. Tanzania Second Technical Assistance Project Cr106O 8462 19-Mar-90 210. China Technical Cooperation Cr1412 8825 25-Jun-9 211. Cameroon Second Technical Cooperation Project Cr1168 8888 25-Jun-90 212. Congo Technical Assistance Ln22865 8884 25-Jun-90 Telecommunications 218. El Salvador Third Telecommunications Project Lnl670 8875 08-Feb-9U 214. Tanzania First Telecommunications Project Cr1178 8682 17-May-90 215. Mail Second Telecommunications Cr1206 8882 29-Jun-9U 33 Annex a Loan/ Report Report Credit No. . oate Transportation and Tourism 216. Sri Lanka Road Maintenance Project CrMas 800 07-Aug-89 217. Sri Lanka Road Passenger Transport Project Cr94 8811 07-Aug-90 216. Honduras Tourism Development Project LM1878 652 97-Sep-89 219. Philippines Fourth Highway Project LAl681 8058 07-Sep-89 220. Zimbabwe Transport Rehabilitation Imports Program Ln1994 8666 21-Sep-89 221. China Three Ports Project La2267 885 6-0ct-89 222. Mauritania Fourth Highway Project Cr1231 8109 17-Oct-89 228. Gman Second Highway Maintenance Project Ln2084 8119 17-Oct-69 224. Congo Second Railway Project Lnl228/CrlS47 8117 26-Oct-89 225. Brazl I Second Feeder Roads Project L"1780 8160 08-Nov-89 220. Ecuador Fifth Highway Project Lnl429 8161 06-Nov-89 227. Indonesia Rural Roads Development Project LW2O8S 3170 89-Nov-89 228. Senegal Fourth Highway Project Cr993/Ln1810 8171 11-Nov-89 229. Fifth Highway Project Cr1448 280. Portugal Second Highway Project Ln70 8184 12-Nov-89 281. Argentina Fourth Highway Project 1884 8211 11-Dec-89 232. India Bihar Rural Roads Project Cr1672 8258 29-Dec-89 288. Korea Coal and Cement Distribution Project Ln2267 8265 26-Dec-89 284. Uruguay Montevideo Port Project Ln1798 8358 07-Feb-90 235. Guinea Conakry Port Project Cr1382 8869 14-Feb-90 286. Korea Provincial and Country Roads Project Ln2228 8872 18-Fb-90 237. Sierra Leone Second Highway Project Cr1129 8891 21-Feb-90 238. Yugoslavia Kosovo Railway Project Lnl977 8407 28-Feb-90 289. Thailand Inland Waterways and Coastal Ports Project 1889 8408 27-Feb-90 240. Turkey Highway Rehabilitation Project Ln2187 8488 10-Mar-90 241. Thailand Provincial Roads Project Ln28 8447 16-Mar-90 242. Yemen Arab Rep. Fourth Highway Project Cr1267 8540 20-Apr-90 248. Comoros Second Highway (Maintenance) Project Cr1188 8541 17-Apr-90 244. Costa Rica Fifth Highway Project Ln1845 8692 4-May-90 245. Tunisia Fourth Highway Project Lnl841 8648 18-May-90 246. Bolivia Highway Maintenance Project Ln187 8878 29-May-96 247. Morocco Fourth Hotel Development (CIN-I) Ln1948 8800 29-Jun-90 248. Togo Fourth Highway Project Cr1189 8857 29-Jun-96 Urban 249. Ecuador Quayaquil Urban Development Project Ln1776 8262 29-Dec-89 2560. Botswana Second Urban Development Project Lul584 8308 29-Dec-89 251. Korea Second Gwangju Regional Project Ln1758 8804 29-Dec-89 262. Panama Colon Urban Development Project Lnl878 83805 29-Dec-89 253. Brazil Prep. of Met. Dev. Programs for Fortaleza and Salvador kn2198 840 28-Feb-90 254. India Kanpur Urban Development Project Cr1185 8402 28-Feb-90 255. Dominican Rep. Sites and Services Project Ln2194 8458 19-Mar-90 256. Indonesia Third Urban Development Project L,1658 8588 27-Apr-90 257. Fourth Urban Development Project Lnl972 34 Loan/ Report Report Credit No. No. Date 268. Mexico Deconcentration Program for the Mexico City Region Ln294 8685 28-May-90 259. Mauritius Urban Rehabilitation and Development Project Ln1928 888 28-May-96 288. Morocco Second Urban Development Project Ln1944 887 23-May-90 281. India Modras Urban Development Project II CrU82 8688 28-May-90 202. Korea First & Second Nat'l Urban Land Devel. A Housing Project Lnsl989/2216 8089 28-May-90 283. Cote d'Ivolre Second Urban Development Project Ln2648 8757 O8-Jun-9 204. Nigeria (Bauchi) Urban Development Project Ln1787 8768 08-Jun-90 265. Botewana Francistown Urban Development Project Cr471 8829 25-Jun-90 28. Second Urban Development Project Ln1584 267. Camercon First Urban Project Ln2244 8881 25-Jun-90 Water Supply & Sewerae 268. Nepel Third Water Supply and Sewerage Project Cr1059 8089 29-Sep-89 289. India Rajasthan Water Supply and Sewerage Project Cr1W84 8099 27-Sep-89 270. Madagascar Water and Sanitation Project Cr102 8091 28-Sep-89 271. Jordan Eight Cities Water Supply and Sewerage Project Ln2425 8092 28-Sep-89 272. Greater Amman Water Supply and Sewerage Project Ln2488 278. Tunisia National Rural Water Supply Project Ln2184 8128 18-Oct-89 274. Morocco Second Water Supply Project Ln1724 8129 18-Oct-89 275. Brazil Third Mines Gerals Water Supply and Sewerage Project Ln18s 8410 28-Feb-90 276. Somalia Mogadishu Second Water Supply Project Cr1288 8454 91-Jun-9 277. Indonesia Second Water Supply Project Ln1709 8622 8-Apr-9 278. Malawi Lilongwe Water Supply Engineering Project Cr1272 8842 07-May-9 279. Korea Second Water Supply Project Ln2859 8675 25-May-9 289. Bolivia Santa Cruz Water Supply and Sewerage Project Cr948 8718 81-May-9 281. Algeria Algiers Sewerage Project LnlS46 8772 21-Jun-9 282. Philippines Manila Sewerage and Sanitation Project Ln1814 8779 22-Jun-90 288. Jordan Zarqa/Ruseife Water Supply and Sewerage Project Ln2218 8818 25-Jun-9 35 Annex B EVAU.ATIO STW-IES ISSUED DURING FS Report Report No. LUaLO 0E Studies 1. Jamaica: Structural Adjustment Loans II and III 818 11-Aug-89 (Loans 281S-JM and 2478-JM) and Overview of Structural Adjustment Loans I - III 2. The World Bank and Senegal, 109-87 8941 a1-Aug-89 8. Tanzania: World Bank/Tanzania Relations, 1981-1987 8829 18-Jan-98 4. A Review of Procurement in Policy-Based Lending 8841 19-Jan-90 5. Pakistan: The Age Khan Rural Support Program 8448 01-Mar-90 Second Interim Evaiuation 8. World Bank Experience with Irrigation Development 8494 28-Mar-90 Socio-Economic, Institutional and Technical Impact and Lessons Overview (Volume IV) Philippines (Volume V) Thailand (Volume VI) 7. Free-Standing Technical Assistance for Institutional 8578 19-Apr-9 Development in Sub-Saharan Africa 8. Bangladesh: Review of the Experience with Policy Reforms 874 24-Jun-90 in the 1980. 9. Colombia: The Power Sector and the World Bank, 1970-1987 8898 28-Jun-91 Overview (Volume 1) Technical Report (Volume II) Annexes (Volume III) Annual Report 10. Annual Report on Operations Evaluation (FY89) 8131 88-OCT-89 11. Annual Review of Evaluation Results for 1988 8184 80-Oct-89 36 ANNEX C APPROACH PAPERS FOR ONGOING STUDIES APPROVED BY THE JAC Ap&roach Paper Approval Date Sustainability of Irrigation Projects (JAC88-26) 09/14/88 A Review of Bank Lending for Agricultural Credit (1951-1987) (JAC88-35) 11117/88 A Study of World Bank Supported Rural Development in Northeast Brazil (JAC88-33) 12/07/88 A Study of the World Bank's Role in Human Resource Development in Sub-Saharan Africa (JAC89-11) 04/26/89 The Effectiveness of Bank Project Supervision (JAC89-19) 06/21/89 The Bank's Role in the Environment in Brazil (JAC89-20) 06/21/89 Indonesian Education and the World Bank: An Assessment of Two Decades of Lending (JAC 89-33) 10/11/89 Population and the World Bank: A Review of Activities and Impacts (JAC 89-34) 12/18/89 Overview of Trade-Oriented SECALs (JAC 90-1) 02/02/90 Study of the Effectiveness of SAL Supervision and Monitoring (JAC 90-12) 06/08/90 37 ANNEX D MONITORING AND EVALUATION IN PROJECTS (MSE) 1. This annex reviews the recent experience with M&E covenants. It is based on contributions from the Economic Advisory Staff in Operations and from sector policy departments in PRE. ENERGY 2. The 18 energy projects in FY90 all included provision for monitoring during project implementation, typically in the form of project progress reports, annual financial reports and project completion reports. In addition, several projects included provision for action programs to increase operational performance with built-in monitoring systems. 3. For example, the Nigeria Power System Maintenance and Rehabilitation Project included a comprehensive commercialization plan which included (1) a performance contract between government and the utility, (2) a utility assistance program with a developed country utility, and (3) a pilot program for efficiency improvements. 4. Similarly, a performance contract between government and the Electricity Corporation of Ghana was included in the Third Power Project under a corporate plan aimed at developing an effective and financially sound organization. 5. The Mexico Transmission and Distribution Project included a Financial Rehabilitation Agreement between the government and the utility under which specific financial targets and institution building steps were scheduled using performance indicators as monitors. 6. Under the Yemen PDR Third Power Project the monitoring system covered operational data in areas such as plant forced outages and quality of supply to consumers. 7. A unique approach to monitoring project effectiveness was adopted in the Indonesia Rural Electrification Project which included components to assess and accelerate the productive use of electricity in rural areas and also monitor the effectiveness of village organization in rural electrification. 38 ANNEX D TRANSPORT 8. Project monitoring and evaluation in transport projects relies mostly on management information systems (MIS) developed and used by modal agencies. There is, however, a wide range in the degree of sophistication and quality of the information systems. Revenue-earning entities such as railways and ports for the most part possess a basic information system related to their financial and operational performance. This information is routinely collected although it is not always used very effectively for planning, monitoring and evaluation purposes. The existing system in many cases does not meet the requirements of modern financial and operational management; available information is poorly processed and evaluated for decision-making, mainly due to lack of qualified staff and appropriate data processing and management technology. For these reasons, most projects now include MIS components, and significant attention is directed to improving borrowers' access to MIS. A basic organizing aid and checklist for acquiring or improving railway MIS capabilities under Bank-financed operations is provided in Modular Management Information Systems for Railways, Report INU-30 issued by the Bank's Infrastructure Department. 9. In highways, the main focus of monitoring and evaluation is on traffic surveys, the size and composition of vehicle fleet, evaluations, road condition and management of maintenance equipment. A Bank Policy study entitled, Road Deterioration in Developing Countries, recommends the establishment of an adequate road management system or a phases monitorable program for establishing one as a pre-condition for Bank highway lending. Assistance is being provided under the majority of Bank-financed projects with good examples completed in Niger, nearing completion in Indonesia, and underway in Yugoslavia and China. In FY89, PPR started development of guidelines on road management systems in support of this recommendation. The first output, a guide on information systems and data management covering all monitoring methods for highways, is under preparation. An earlier departmental discussion paper (TRP-5) provided guidance on improving equipment management in developing countries. 10. The availability of microcomputer and database software in developing countries has provided a strong stimulus to monitoring and evaluation activities in transport agencies in developing countries. Bank projects have played a catalytic role in introducing appropriate management information technology to these agencies. Financial and technical support for information systems and related computer hardware and software is envisaged in future Bank lending in all transport modes. 39 ANNEX D URBAN PROJECTS 11. Several different monitoring and evaluation mechanisms were incorporated into the design of the urban projects that were approved by the Board in FY90. The predominant mechanism to facilitate feedback from operations was through the activities of a Project Management Unit, typically situated under the jurisdiction of a Ministry. The implementation of almost all urban investment and emergency reconstruction projects was coordinated by such entities also called Project Implementation Units, Special Projects Units, and Emergency Reconstruction Units. Although typically established under a Ministry, the Project Units had varying degrees of autonomy depending on their funding and staffing. In the case of the Burkina Faso Second Urban Project for example, special funds (7% of total project costs) were allocated for the establishment of a Project Unit within the Ministry of Finance to ensure an adequate level of funding for monitoring the project. 12. In cases where more than one ministry is involved in the implementation of a project, an Inter-ministerial Committee, a high level Project Advisory Committee, a Project Coordination Committee or a Special Project Steering Committee has usually been set-up to oversee the operations of the various components of the project. This organization structure is also used in multi-sectoral and multi-city projects. For example, in the Ghana Urban II Project an Inter-ministerial Committee will meet bi-annually to review and provide policy guidance for the project during the first two years and annually thereafter. Operations of Project Units will be coordinated by the Project Coordination Committee which will meet quarterly during the first two years of the project for resolving operational problems and project assessment. 13. Urban projects involving financial institutions used different monitoring and evaluation techniques compares to investment projects. For example, in the Morocco Second Housing Finance project, CIH (Credit Immobilier et Hotelier) --which had a previous working relationship with the Bank- will monitor its own lending operations by random checks to ensure that the credits are targeted as intended. Such random checks will serve as deterrents to the misuse of funds. In the Brazil Municipal Development Project in the State of Rio Grande Do Sul, in addition to routine monthly reporting activities by the eligible sub- borrowers, BADESUL's (Rio Grande do Sul State Development Bank) Project Unit will conduct random checks of the performance of municipalities. In the Ethiopia Second Addis Ababa Urban Development Project, the Housing and Savings Bank (HSB) will establish a Project Implementation Unit reporting to the Deputy General Manager of HSB. 40 ANNEX D 14. In the FY90 urban projects, pertinent ministries, local banks and other institutions (such as the Housing Authority and the Low Income Housing Fund) were given the responsibilities of preparing periodic assessments of project performance, and reporting their findings to the Bank. In this context, continuing training of staff in project evaluation and coordination was a crucial component of projects which will also ensure constructive communication with the Bank. 15. Annual Reporting activities encourage the assessment of the performance of the projects, and of related policy issues, with the Bank and local policy makers. These reports will typically be prepared bi-annually during the initial phases of the project (first two years), and yearly thereafter. In the Mexico Second Low Income Housing Project, the Low Income Housing Fund's Evaluation and Monitoring Unit will prepare annual reports and assess the impact of its programs. 16. Finally, the two emergency reconstruction projects approved in FY90, the China Earthquake Reconstruction Project and the Yemen PDR Emergency Flood Reconstruction Project contain separate and temporary Emergency Reconstruction Units to manage the reconstruction activities, which also have the responsibility to prepare annual implementation reports. WATER AND SANITATION 17. Bankwide, about 6% of projects under implementation are in the Water Supply and Sanitation (WSS) sector. These are distributed almost evenly worldwide, with 24-30 projects in each region. All 13 projects approved by the Board in FY90 have built- in provisions for evaluating and monitoring progress, including expected institutional performance indices. Monitoring indicators have been made part of periodic reporting requirements and are to be updated (usually twice a year) by the borrower for review by project supervision teams. Standard indices in the WSS sector include: % of non-revenue water, additional water/sewerage connections, % population served, volume billed per person and managerial and financial ratios. 18. An emerging feature in the monitoring components of WSS projects us concern with documenting changes in environmental quality. The Abidjan Environmental Protection Project describes 41 ANNEX D a detailed system for the quarterly monitoring of the wastewater receiving water bodies. Other projects (including Nairobi, Hyderabad, Uganda, and Jabotabek (Indonesia)) also prescribe periodic reports on those environmental aspects which are identified as important in the respective Environmental Action Plans. These include population resettlements, dam safety and quality control of water bodies used as wastewater disposal sites. 19. Also of note in this year's projects is the first ever Bank water sector adjustment loan (to Cote d'Ivoire). As is standard for adjustment loans, the President's Report specifies the creation of a Supervisory Board and a Calendar of Actions and Implementation of Adjustment schedule for the Board's guidance. The indices to be used in monitoring the efficiency of operations in the sector are specified in the Concession Contract between the Government and the water company (SODECI). Compliance with the specified efficiency improvements are a condition for release of the second tranche of the loan. POPULATION, HEALTH AND NUTRITION 20. Twelve PHN projects have been approved in FY90. All twelve recognize the importance of monitoring and evaluation in assuring the attainment of project objectives and the sustainability of the projects. As in previous years, the projects include monitoring and evaluation activities in each of the components, and have mechanisms for monitoring and evaluating the project overall. 21. Every project includes components for institutional development and the strengthening of management and administrative capacity, which in turn contain monitoring and evaluation activities. The Nigeria Essential Drugs Project, for example, aims to strengthen institutional capacity at the federal, state and local levels of governments to develop and implement the essential drugs program and to foster institutional restructuring and consolidation for effective service delivery. This is accompanied by evaluation activities to measure the success of the changes. 22. All of the projects include most if not all of the following factorst compiling baseline data, conducting household and community-level surveys, developing management information systems, ensuring community participation in project planning and decision- making, and supporting operations research and the design of 42 ANNEX D innovative activities to test new interventions or modifications. The Tanzania Health and Nutrition Project is one that includes components to strengthen rural and urban primary health care with built-in monitoring and evaluation mechanisms for feedback to primary health care policy formulation. These mechanisms included day-to-day management monitoring through monthly reports and quarterly field supervision, regular progress evaluations and a mid-project external evaluation. 23. The built-in monitoring and evaluation activities for the entire project will establish timetables for reports and will develop process, impact and outcome indicators for assessment purposes. Every project includes semi-annual (and usually quarterly) progress reports from the project implementing agency or unit to the central coordinating body and to IDA. Indicators will be simple and as easily measurable as possible, and quantifiable targets are to be set. Biannual and frequently annual surveys will be conducted to determine the impact of the projects. Special studies will be undertaken to assess the effects of different components on the knowledge, attitude and practices of the target population, such as the three studies planned in the Lesotho Second PHN project. In the Morocco Health Sector Investment Project, the project coordinating committee will organize a project launching workshop at the beginning of the project to facilitate implementation. 24. Each of the projects have taken steps to secure government cooperation by agreeing on monitoring and evaluation activities during negotiation. The content and frequency of progress reports and supervision activities are spelled out in advance. Provisions are made to ensure the success of monitoring and evaluation. In the Tamil Nadu project, for example, funds are committed in advance for an additional vehicle and for the equipment and salaries incurred during monitoring, including computers and training for operators. Although it is difficult to predict the problems that these projects will face, component-oriented monitoring and evaluation activities and mutually agreed upon schedules are important contributors to their success and sustainability. EDUCATION 25. All 21 of the education sector projects approved in FY90 included monitoring and evaluation components. This continues a trend begun in FY86 of monitoring and evaluation activity being 43 ANNEX D found in all Bank supported projects in the sector. Seventy percent of the projects called for explicit evaluation research and studies to provide baseline data and appropriate comparators for assessing the implementation and impact of the project. Nine projects included specific monitoring indicators (costs and quantitative indicators of participation). 26. Seventy one percent of the FY90 projects included investments designed to strengthen the monitoring and evaluation capacity of Borrower agencies through training and fellowships, technical assistance and purchase of computer hardware and software. Of these, 11 projects are notable for including components designed to improve the quality and efficiency of national examination systems. Twelve of the projects included support for the development and installation of management information systems, typically as part of a program to improve sector management and administration. 27. The Education and Employment Division in PHR provided support for monitoring and evaluation through advice to project staff and publication of policy-oriented evaluation studies. Among these are studies that analyze the public role in private post- secondary education, investigate the effect of job training on women's employment and wages, compare school effects and costs for private and public schools, examine a multi-level model of school effectiveness, and present new evidence and priorities concerning the relationship of education and development. 28. The Division has initiated a program designed to build research, evaluation and assessment capacity in developing countries. Specifically, this program will assist ministries of education to: (i) assess the impact of national education policies and instructional resource availability at the school level on student learning achievement and (ii) monitor changes in student learning achievement over time. 44 ANNEX 8 REPORT ON DISSEMINATION AND FEEDBACK This Annex reproduces the responses of operational managers who reported to the Director-General, Operations Evaluation (DGO) on the effectiveness of dissemination and feedback in the context of a selected number of evaluation studies, audits and PCRs. No attempt has been made to change the text of these memoranda; only introductory or transmittal statements were removed. The responses are arranged in three parts: the first part reports about the follow-up to unresolved issues flagged in last year's Annual Report; the second part highlights actions taken by Operations and PRE in response to recent OED studies; while the third part contains regional responses to selected audits and PCRs. I. POLLUi-UP TO ISSUES FLAGGED IN LAST YEAR'S REPORT Several unresolved issues were flagged in last year's Annual Report (Annex E) which required further follow-up during FY90. This section reports on the subsequent actions taken by PRE and Operations to deal with these issues. The relevant pages in last year's report are shown in parenthesis for easier reference. Cost Recovery in Irrigation (Annex E, p. 40) AGR staff, in cooperation with Operations, has completed a study on this issue. The findings of the study were presented to the Vice-President, PRS and later on AGR published the report as PRE Working Paper No. 420. Land Settlement (Annex E, p. 40) The new Operational Directive 4.30 on Involuntary Resettlement was issued in June 1990. Monitoring and Evaluation (Annex E, p. 41) The new Operational Directive 10.70 on Project Monitoring and Evaluation was issued in September 1989. 45 ANNEX B Technical Paper on Livestock Projects (Annex E, p. 40) This study is still underway. Due to early retirement of the livestock adviser, a new schedule was set for completing this study in cooperation with Winrock International Research Center, International Center for Livestock in Africa, and other concerned international institutes. The study would be completed in about two years. Indonesia Technical Assistance (Annex E, p. 43) I am pleased to inform you that we have indeed undertaken the comprehensive study on Technical Assistance Components in Bank- Assisted Projects in Indonesia as we planned last year. A special Bank mission was sent to Indonesia in March 1990 for this purpose and the draft report of the mission is currently under review within the Department. After our internal review is completed, we intend to submit the report to the Indonesian Government for discussion. The draft under review clearly shows that there are several similarities between OED findings in the "Review of Bank-Assisted Free-Standing Technical Assistance Credits in Indonesia" (No. 7156, dated March 10, 1988) and those formulated by our special mission referred to above. In addition, the mission has made a number of specific recommendations for the Government and the Bank staff about improving management, monitoring, supervision and coordination of technical assistance programs supported under Bank- financed projects in Indonesia. Included in these specific recommendations are the following: * Closer participation of government officials in identifying the needs and drafting TORs for consultancy services. * More effective supervision, monitoring and evaluation of consultants' work by government agencies. * Better coordination within the government of the technical assistance proposals from various parts of the government. 46 ANNEX B e Tailoring the scope and complexity of technical assistance components taking into account the absorptive capacity in government agencies. * Devoting increased attention by government officials to the reporting arrangements for consultants and to ensuring closer interaction with them. * Adhering to TORs of consultants as in their contracts unless there is a strong justification for amendments thereto. * Larger resource inputs than currently budgeted in the Bank to supervise technical assistance components of Bank-assisted projects in Indonesia. II. FOLLOW-UP TO RECENT OED STUDIES This section reproduces the responses of operations managers to selected OED studies issued in FY89 and FY90. The managers were requested to report on (a) the validity, significance and implications of the OED findings, and (b) the nature of the follow- up taken or planned by the unit responsible. Jamaica Overview of Structural Adjustment Lending in Jamaica; ORD Report No. 8018, August 11, 1989 We found the analysis and assessment of the above report fair and accurate, bearing in mind that the relevant PCRs and the PPAR focused primarily on Jamaica's economic policies and performance through the end of 1985, the year when the last of the three SALs to Jamaica was disbursed. In particular, we concur with the OED findings that, in 1985, the Jamaican economy was still characterized by several structural distortions, including large subsidies, high effective protection alongside discriminatory concessions, the crowding out of the private sector in the domestic credit market, and proliferation of public enterprises and statutory bodies. Economic performance too had remained poor in 1980-85 as a result of the slow pace of policy reforms and the virtual collapse of bauxite and alumina earnings. 47 ANNEX B In view of the unsatisfactory situation in 1985, the Region began emphasizing the need for more rapid and fundamental reforms, as attested by the Tripartite (IBRD-IMF-USAID) Report acknowledged in the OED Overview. Since the completion of the three SALs audited by OED, our adjustment operations in Jamaica have focused on: (a) phasing out generalized food subsidies and improving targeted nutrition aid (Social Sector Development Loan); (b) lowering import tariffs and eliminating various discretionary exemptions (First Trade and Financial Sector Loan, the Agricultural Sector Adjustment Loan, and the proposed Second Trade and Financial Sector Loan); (c) enhancing private sector access to credit by, inter alia, containment of the public sector borrowing requirements and promotion of open market operations in government securities (First Trade and Financial Sector Loan, and the proposed Second Trade and Financial Sector Loan); and (d) initiating basic reforms in the public enterprises sector, including promotion of cost recovery and divestment of government holdings (First Public Enterprises Adjustment Loan and a possible Second Public Enterprises Adjustment Loan). Our latest economic work and policy dialogue in Jamaica has also paid special attention to the lessons underscored in the OED Overview, notably the need to deploy a suitable policy-oriented analytical framework and close coordination of Fund and Bank analysis. In the last three years, two-medium-term policy framework papers have been prepared with Bank and Fund staff and their Jamaican counterparts working closely to devise macroeconomic scenarios and their policy implications. Beginning with the Agricultural Sector Adjustment Loan, we now also incorporate, in the Letter of Development Policy, formal government commitment to maintain a viable macroeconomic situation. To disseminate the findings of the OED Overview within the Region, we devoted a full session in our LAC Economic Seminar Series to a discussion of those findings, with the guest speaker from OED and with participants from the Region's CD III and other departments. We have also responded candidly to questions regarding the OED findings raised in other fora; for example, during Board discussion of the Jamaica Agricultural Sector Adjustment Loan and during the Jamaica Sub-group meeting of the Caribbean Group for Cooperation in Economic Development held in April this year. It is worth noting also that the OED Overview has been publicized and read extensively among Government officials in Jamaica. As a result, the officials are now quite aware of the assessment and lessons recorded in the Overview report. 48 ANNEX 9 Procurement and Construction of Highway Civil Works in Small Countries; OED Report No. 7911, June 29, 1989 The OED Report on "Procurement and Construction of Highway Civil Works in Small Countries" provides an important, precise and favorable assessment of the different procurement solutions applied to road construction during the last fifteen years in the Yemen Arab Republic and the People's Democratic Republic of Yemen (unified as the Republic of Yemen since May 22, 1990). The report provides an opportunity to compare the underlying assumptions of the Bank's procurement Guidelines with actual practices through case studies and confirms that due to the exiguity of the YAR and PDRY construction markets and the changeable economic situation in the Gulf area, the basic procurement method of International Competitive Bidding (ICB) among private contractors is not always feasible. Public-owned foreign contractors were a viable mode of construction, as was force account in the most difficult cases. The lessons learnt from the report on the experiences in both North and South Yemen were directly applicable to the procurement decisions taken under the Sixth Highway Project, Credit 1726-YAR, (August 1986). Under this project, a combination of ICB for procurement of construction materials, and force account with partial subcontracting to small domestic contractors for the execution of works led to a satisfactory completion of difficult and isolated road works on time, cost and quality in June 1990. The participating domestic contractors received important management training from the Highway Authority during the execution of works. Highway works procurement in the unified Republic of Yemen will continue to be implemented in the immediate future using the procedures described in the report, which can also be applied to developing countries with similar characteristics to YAR and PDRY. World Bank Experience with Irrigation Development - Socio- Economic, Institutional and Technical Impact and Lessons; Overview (Volume I), Mexico (Volume II), Morocco (Volume III); OED Report No. 7876, June 15, 1989 Mexico I have made a quick review of the OED study: "World Bank Experience with Irrigation Development", Overview, Mexico and Morocco volumes. In the past we have furnished substantive inputs 49 ANNEX 8 and project specific comments to OED concerning the Rio Sinaloa and Panuco Irrigation Projects in Mexico. The above OED study has now sharpened its analysis of the issues identified in project completion and audit reports for the above projects and has compared the Mexican experience with 'the Doukkala I and II Irrigation projects in Morocco. I do not intend to comment on project specific issues covered by the OED study. Rather, I will add a few pervasive issues which I think will help to explain the limited achievement of irrigation projects in Mexico and suggest new strategies that may help to increase. effectiveness of the Bank's assistance to the subsector. Several of the irrigation loans in Mexico, including the loans for the above projects, were made at a period of highly protected economy when there was major distortions in internal terms of trade favoring industry. The resulting negative protection in agriculture provided very weak incentives for the farmers to produce (or produce more). In the extreme case of rice, for example, the producers received only half of the international price. This led to reduced benefits and low returns for irrigation investments. It was only recently that, with the help of the Trade Policy Loans, industrial protection began to be dismantled. Further reduction of discrimination against agriculture were achieved subsequently with the Agricultural Sector Loan (1988) which required, inter alia, to bring the producer prices within a band of 90 - 120% of the international prices. Decision making for irrigation investment in Mexico is highly centralized. The investments are constructed, financed, and managed by the Federal Government. Partly to compensate the rural sector for the negative protection referred to above and partly because of the inherent inefficiency of large government approach, nearly all irrigation investments were made (by the Federal Government) at gratis and financed with fiscal resources. There has been no capital cost recovery and only limited O&M recovery (currently around 20%) in public sector irrigation. While, the Government occasionally considered cost recovery and legislated some recovery provisions from time to time, it was never serious about it. In fact, no serious analysis of the cost recovery issues, institutional arrangements and definition of the instruments of recovery were made. Moreover, the Government never knew the real cost of this policy. In this context, the Bank's reliance only on legal covenants for cost recovery under various irrigation projects was unreal.Vatic. 50 ANNEX E While the distortions in the pricing system prevented rational resource allocation, the administrative procedures for investment decision making were also ineffective. You will recall during preparation of the proposed-Irrigation and Drainage Sector Project we examined the National Water Commission's investment program for 1991-1993 to see whether the proposed investments meet the criteria and methodology which we had developed together with the Commission. Our examination showed that the investment proposals (including several ongoing investments) only satisfied engineering criteria and thus there was a strong engineering bias in investment selection. The investment proposals failed to satisfy the economic, agronomic and environmental criteria -- principally because the investments were not tested against such criteria. Evidently, this engineering bias has been there for decades. We have now required the Commission to carry out a rigorous screening of the entire investment program in accordance with the agreed criteria and methodology as a condition for project appraisal. Conclusion. Clearly, conventional project lending for irrigation carries major risks in that resource allocation objectives are ignored. The projects focus on the technical issues of specific investments without due regard to the sectoral context. A better lending strategy for irrigation would have a sectoral approach with a sharp focus on resource allocation issues such as pricing of water, beneficiary participation in funding of investment and operation, adequate institutional arrangements (e.g., moving from centrally micro-managed system to decentralized public utility type organizations managed by the beneficiaries) and improved analytical capacity and balanced skill mix where irrigation investment decisions are made. This can be achieved through sectoral time slice loans that early on defines investment selection criteria and methodology covering acceptable engineering, economic, agronomic and environmental standards and requiring that the entire investment program satisfies such criteria (except a very small portion of the investment program where the criteria may give a much higher weight to social considerations). Moreover, the institutional reform process for improved irrigation management and cost recovery must be substantially in place to ensure that the legal requirements for cost recovery are implementable. With this strategy the risks of transmitting national resources into a distorted environment will be minimized and measurement of investment priorities will be improved. 51 ANNEX E The World Bank and Senegal, 1960-87 OED Report No. 8041, August 31, 1989 AF5 staff had commented extensively on the report at various stages of its preparation, including during a review meeting which took place on December 19, 1989. The final version of the report reflects AF5's comments in all important respects except two. First, the report underestimates the critical importance of information constraints which led the designers of the initial SAL operations to seek, through a broad range of studies, to identify areas in thr economy which could respond to stimuli and then to tailor policy accordingly. Overall, AF5 staff are of the opinion that the real improvement in knowledge which has led to more sharply-defined and more effective adjustment operations could not have occurred so quickly if SAL I and SAL II had not been designed, inter alia, to generate this knowledge. Second, in suggesting that "climatic impediments" can be attacked through "development of irrigation systems and soil regeneration and conservation programs", the report fails to take into account the Bank's past efforts in these areas. Substantial sustained efforts have been and continue to be made by the Bank to develop economically viable irrigation and soil regeneration projects. However, so far success has been limited, and we believe that OED is mistaken in its implicit assumption that viable investment options are necessarily readily available. The above two disagreements are, however, relatively minor differences of view for a report as comprehensive and complex as the one on Senegal. Moreover, during the preparation of the report, there was a very fruitful exchange of views between the staff of the two departments on development policy and operational strategy. Consequently, many of the findings of the report have been incorporated in the recent Country Strategy Paper for Senegal and in the formulation of recent policy-based lending operations, namely SAL IV, the Financial Sector Adjustment Program, and a series of SECALs currently under preparation. In agriculture, for instance, the Bank's present strategy coincides in many respects with the prescriptions of the report. Top priority is being given to policy reform in order to improve the enabling environment, investment in agricultural research and extension services in order to perfect and disseminate technological packages, development of innovative new approaches to natural resource management and conservation, rehabilitation and improvement of existing irrigated agriculture, and institutional strengthening in these areas. Regarding the need for further macroeconomic reforms, our own evaluation also coincides in many respects with the findings of the 52 ANNEX E report. The fundamental direction in which the structural reforms are leading the economy, that is the increasing role of the private sector in a liberalized market environment and a reduction in the role of government with a refocussing of expenditure, appears appropriate for Senegal. Finally, the Audit report could be useful to Senegal in refining its development strategy. The report addresses critical elements of a "development agenda" that needs to be followed by Senegal if growth is to be sustained in the medium- to longer- term. The usefulness of the report could, however, have been further enhanced if it had been able to examine in more detail the rationale for Senegal's policy decisions and actions. As it is, the report overplays the Bank's responsibility for what happened in Senegal as both the government and other major donors were responsible for taking many critical actions, not always in consultation with the Bank. Conditionality in World Bank Lending: Its Relations to Agricultural Pricing Policies OED Report No. 7357, June 1988 This report analyzes project and non-project conditionality with respect to agricultural pricing. Through a review of 85 projects and 65 non-project lending operations, the report finds that agricultural pricing issues have been increasingly addressed in Bank operations particularly in the 1980s. It contrasts how pricing has been dealt with in project loans (strong emphasis on periodic review of producer incentives) and in agricultural non- project operations (emphasis on free markets, border pricing, and reduced input subsidies). The review emphasizes that pricing issues are complex, requiring consideration of equity, macroeconomic influences such as the exchange rate, and pricing in other sectors. It recommends that additional time be given to staff to study pricing issues, that workshops be held to review pricing policies, and that more funds be included in loans to improve in-country data collection and analytic capabilities. We agree with the general analysis and emphasis of the report on taking a broader view of pricing issues. Bank staff are a]ready addressing pricing issues through a broader analysis and scope in economic and sector work. Loan conditionality is increasingly focusing on making prices more flexible though reducing government controls and interventions and reducing macroeconomic biases against agriculture. Furthermore, equity 53 ANNEX B considerations are being addressed through special targeted programs of food distribution that effectively reduce the cost of basic staples to the poor. Bank staff are dealing with the indirect effect that industrial protection has on real agricultural incentives though agricultural inputs. In addition, exchange rate effects on agricultural prices are more routinely being taken into account in sector work through analysis of the actual incentives and trade protection offered the agricultural sector when exchange rate misalignment is present. This broader approach to pricing advocated in the report has also been substantiated by Bank-sponsored research. A recently completed Bank-sponsored research project (RPO 673-64) studying 18 developing countries reaffirms the validity of this focus on indirect factors that affect agricultural incentives. This research established that the effects of industrial sector and macroeconomic policies on agricultural prices overwhelm the direct incentives and trade protection offered producers. Bank staff have also addressed the recommendations with respect to improving the quality and efficiency of sector work on pricing issues. Workshops, seminars and trainings have been held to address pricing issues. In addition, a relatively simple multimarket approach has been developed that employs user friendly software for analyzing trade-offs in pricing policy. This approach, developed by the Agriculture and Rural Development Department, has been used in over a dozen countries to support sector work and policy-based loans. The main focus of the multimarket framework has been on assessing the tradeoffs between the real incomes of various groups (for example, earnings of farmers versus urban dwellers), the implications for foreign exchange earnings, and the savings in net government expenditures deriving from various pricing and taxation policies. The objective of the multimarket approach is to allow policymakers and country and sector economists to get an analytic assessment of policy options in a minimum of time even when data are incomplete. Additional actions taken by Bank management and staff to address some of the concerns raised in the OED report include: (a) a major review of covenants scheduled within the ARIS framework; (b) more attention to briefing and debriefing of appraisal and supervision missions, including a wider discussion of pricing issues to ensure suitable conditionality and enforcement of covenants; and (c) increased emphasis by Bank staff on improved data collection and more in-depth analysis. In addition, a clearer determination of the impact on equity of various pricing conditionality and policy reforms is being attempted through initiatives and research on food security and on the social dimensions of adjustment. 54 ANNEX E Renewable Resource Management in Agriculture ORD Report No. 7345, June 1988 This report concludes that (a) the Bank's agricultural and forestry Economic and Sector Work has increasingly (though often belatedly) recognized renewable resource managament issues; but (b) the Bank's agriculture and forestry lending, while showing "surprisingly little evidence of gross (environmental) negligence or damage" has not, significantly increased its relatively limited attention to environmental issues. It recommends (a) an assessment of "what the main issues in a country are", (b) Bank leadership in developing broader donor-country coordination on environmental issues, and (c) additional Bank resources for the environment. Environmental staff in Operations have reviewed the findings in this report. We agree with both the conclusions and recommendations of the study, and have taken measures to strengthen the Bank's environmental work, including (but not limited to) the agriculture and forestry sector. Specifically, (a) the staff and other resources for environmental work have been increased sharply with the creation of the Environment Department in PRE and the four Regional environmental divisions; (b) Environmental Issues Papers are being prepared for all active borrowers -- more than 70 have been completed already; (c) a new directive on Environmental Assessment has been issued which would lead to more thorough consideration of environmental issues during project preparation; and (d) in a number of countries (e.g. Madagascar and Mauritius), the Bank has organized a major conference between borrowing country and donor agencies, leading to participation of many donors in financing broad environmental action plans. An important area for future emphasis lies in the integration of forestry and agricultural work. There is a growing perception that forestry components (nurseries, windbreaks, fodder and fuelwood trees, community forestry, soil conservation, etc.) have a role to play in agricultural projects, especially in arid and semi-arid areas where integrated natural resource management projects are better suited to farmers' needs. Bank management will encourage sector work and lending efforts in this direction. 55 ANNEX 8 Agricultural Marketings World Bank Experience ORD Report No. 7353, June 1988 The overall conclusion of this review is that the Bank has paid less attention to agricultural marketing than the topic merited and, in particular, has neglected the potential role of the private sector. The report believes that Bank staff proceeded on the erroneous assumptions that existing marketing systems were adequate; that there was a case for state monopoly of strategic crops; and that private traders were inefficient or exploitative. To help redress both the neglect of the issue and the misconceptions about it, the report recommends recruiting more marketing specialists. Each Region has taken steps to disseminate the findings among operational staff. The report reflects experience i the 1970s and early 1980s but does not shed much light on why the Bank was often unable or perhaps unwilling to dissuade Governments from following monopolistic policies, or to change its attitude towards private traders. Currently, privatization is an area of emphasis for the Bank and the need to expand the role of the private sector in agricultural marketing is being addressed through structural and sectoral adjustment lending conditionalities, and through country implementation review dialogue with Governments. In addition, significant changes in the scope and mode of operation of marketing parastatals is incorporated in many projects, particularly in Africa, and technical assistance is being provided to help in the institutional development of the restructured parastatals. Further emphasis is required on the organizational aspects of marketing and on risk management techniques. Understanding the likely degree of government acceptance of privatization will be important for future project design. Regarding specific marketing issues, the report highlights the importance of energetic promotion of agricultural products in competitive world markets. The ability to discover market niches; develop new products in the agro-industrial sector; maintain quality at a high level; and transmit this information to farmers will be crucial to the success of agriculturally dependent 56 ANNEX E economies. These are, however, activities which require substantial entrepreneurial drive. The question is one of finding ways of developing skills and attitudes necessary to undertake these activities within the borrowing countries. Having this expertise in-house would be a good first step in the transfer to our borrowers. But it is not easy to specify the type of required skills. This issue is currently under consideration within PRE in consultation with the TDs. The recruitment of a marketing economist by both ASTAG and ENTAG and of people very knowledgeable of international commodity marketing in LATAG constitutes experiments which will be monitored closely. III. PROJECT EVALUATIONS The Bank's Regional Offices were requested to report on the follow-up to the findings of selected PCRs and OED audits. The regional responses are shown below. A. Africa Regional Office Malawi - Lilongwe (Water Supply) Engineering (Credit 1272) PCR No. 8642, May 7, 1990 We agree with the findings of the PCR for the above project that although the project was well-conceived, well executed (particularly on the water supply side), and achieved its specific objectives, there were a few areas where project design might have been improved and the six months implementation delay avoided. Project design might have been improved if: (i) LWB management team had been in place early enough to bring professional experience to bear on the preparation of the terms of reference for the Master Plan and Feasibility Study, and thus limit extensive review of the studies; and (ii) the low-cost sanitation studies had been under the responsibility of LWB to facilitate administration and management of the component. While the above problems did not unduly affect the implementation of the project, inclusion, under the same credit, 57 ANNEX R of components that would be outside the responsibility of the principal implementing agency would be carefully scrutinized in the design of future projects. simbabwe - Transport Rehabilitation Imports Program (Loan 1994) PCR No. 8068, September 21, 1989 We concur with the central finding of the PCR for the above program loan that a substantive increase in the railways' nominal capacity in the face of relatively stagnant traffic prospects weakened considerably the incentive for the railways to perform efficiently. In the course of supervising the follow-up project to the Transport Rehabilitation Program, we had become increasingly conscious of systemic weaknesses in operating practices and how these were masked--albeit progressively-- with less success--by the large size of the locomotive fleet and easy access to hired wagons. We are addressing these weaknesses in a proposed new project (Second Railways Project FY91) in a comprehensive manner through assisting the railways to strengthen maintenance and operating practices, management information systems and management development. Under the proposed Railways II project, we have reached an agreement with Government on supportive changes in the regulatory framework, and constraints on investments to a level consistent with an efficient degree of operating performance and a conservative projection of traffic prospects. In addition, we have reached agreement with Government and the railways on ceilings on the hire of locomotives and wagons and on initiation of procurement action for 30 percent of the assessed requirement for locomotives only with the approval of the Bank. This approach is consistent with the recommendations of the PPAR. The TD, which has links with the other SOD, is the main vehicle for disseminating the findings of the PPAR. Thus far, one meeting of railways specialists in the region has been held under the auspices of the TD at which the lessons of the PPAR and other OED audits of railways projects as incorporated in the aforementioned project were discussed. This meeting was attended by OED staff. 58 ANNEX E Cameroon - Feeder Roads (Loan 1494/Credit 749) Fourth Highway (Loan 1723/Credit 926) Fourth Railway (Loan 1734/Credit 936) PPAR No. 7828, June 12, 1989 Introduction. To assess the follow-up and feedback from audited projects to planned operations; one must first recognize two points: (i) several other projects have already been prepared and even completed when the audit reports for earlier projects are issued and their recommendations available; and (ii) during implementation of the audited projects the country has gone first through an economic bonanza, during which time their acceptance of sound economic advice was subjected to political decision, while currently the downturn of the country' economic requires the measures the Bank has been making. The audit report on the Feeder Roads, the Fourth Highways and Fourth Railways projects were combined. The audit, while endorsing the recommendations and conclusions of the various Project Completion Reports, went beyond the limited scope of these projects and reviewed IDA and Bank lending for transport over a period o 20 years. The specific recommendations from the audited projects may already have been tested or even implemented under completed and ongoing projects. Feeder Roads Project. This project's objective was to improve Government capacity to maintain feeder roads. This objective was too ambitious for a first project. Insufficient engineering and a lack of Government commitment and supervision resulted in a project, which despite a four year extension of the closing date only achieved 40% of its physical target and an average cost of constructed roads largely exceeding original cost estimates. The project had no sustainability and as soon as IDA financing ran out the project stopped and maintenance of feeder roads returned to its earlier status of neglect. These findings are incorporated in the design of the RURAL INFRASTRUCTURE PROJECT, the objectives of which are broader. They now support Government policy to bring economic development in rural areas. They not only deal with infrastructure but also with rural transport and private sector development in rural areas. Rural water supply to improve living conditions was also added to stimulate local Government commitment. Extensive studies were carried out and the selection of priority networks agreed with local authorities under the National Commission for the Coordination of Rural Roads. This Commission, established previously, only now gets to play its role in the preparation and implementation of the RURAL INFRASTRUCTURE PROJECT. 59 ANNEX B Processing of this project, however, was again delayed, because during pre-appraisal it became clear that a reorganization of the supervisory ministries was being contemplated, no reliable institutional arrangement could be firmed up. Fourth Highway Project. The objective of this project was to improve the Government's capacity to maintain its principal highways. Taking into account the problems the Feeder Roads Project, under implementation at that time, was having, institutional arrangements were thoroughly reviewed and clear definitions of respective responsibilities were agreed upon during negotiations. The project reached and even exceeded its objectives. Sustainability, however, fell victim to the economic crisis and once Bank financing ran out, despite an extension of one year to improve Government commitment to road maintenance, it became nevertheless a secondary concern and the results achieved under the project faded rapidly. Despite the fact that the Sixth Highway Project, appraised during the period when maintenance was being done well, was not designed specifically for maintenance, the project under implementation was redesigned to emphasize road maintenance. The maintenance backlog accumulated during the three years after the Fourth project was completed, while Government was adjusting the economy to new realities. How efficient the Fourth Highway Project ever was, changing circumstances affected the institutions, the project established. Fourth Railway Project. Its objectives were standard, namely to increase the capacity and efficiency of the railway's traffic handling facilities and improve management and operations. Physical components were substantially completed, while the financial targets of the project were not attained. The project was identified and negotiated directly with the railway authority. It failed to influence Government policy on pricing, the Government not considering itself bound by agraements negotiated directly with the railway. When the economic situation took a nose dive, the drain on Government resources by inefficient public enterprises forced the Government into action. A Public Enterprise Commission was established with assistance from the Bank and a TRANSPORT SECTOR PROJECT, on the books for a while already, suddenly got the Government's attention. Reformatory measures, as those recommended by the Audit, took on a new meaning. Financial restructuring, financial autonomy, and improved management as well as a clear 60 ANNEX B delineation of responsibilities became primary objectives. The solving of accumulated debts and arrears now hamper implementation of the measures, but Bank involvement and interaction between Government and the parastatals are now functioning at the level the Audit recommends. The TRANSPORT SECTOR PROJECT, now under preparation, is based n these new understanding and has been broadened to include other transport parastatals such as public bus transport, the national airline and shipping, while all to different degrees show the problems the Railway Project brought out. Conclusion. The sector objectives formulated 20 years ago, when the Bank first started lending for transport to Cameroon remain appropriate today and deserve to be preserved. The time needed to bring about major changes by far exceed the lifetime of a single project. Sector strategy conserved throughout a succession of projects, adjusted to varying economic circumstances, has the better chance to succeed. Nigeria - Bida and Ekiti/Akoko Agricultural Development (Loan 1667/Credit 1854) PPAR No. 8557, April 23, 1990 Validity/Implication of Audit/PCR Findings Findings. Following last year's OED's Annual Performance Audit Report, the Region has undertaken a complete review of the past and present agricultural portfolio in the Africa Region to make an overall assessment of what has caused only a 50% success rate in agricultural projects and whether improvements are likely with the present portfolio. You and OED have seen the drafts of the report entitled "The Performance of Agricultural Projects in Africa - A Review of Projects Ratings and Risk" which is in the final stages before being presented to the RMT with the ultimate objective of the possibility of it being presented to the Board at the time of the next OED Performance Review. The main causes of project failure are threefold: (a) Macro and sectoral policies. (b) Institutional and management; and (c) Suitable technology (which in turn is affected by the other two). 61 ANNEX 8 While project design has sometimes compounded these three problems by not sufficiently addressing them when possible, it is apparent that these three issues are the fundamental ones that have to be addressed if the portfolio is to improve. To the extent that these three issues remain a problem in Africa today, improvements in the agricultural portfolio will be directly affected by improvements in these areas. Macro and sectoral policies in borrower countries are receiving considerable attention. Where governments are not dealing with these issues sufficiently, the lending program to the countries in all sectors is being cut back. With regard to institutional and management issues the Region is working at three levels: (1) At the macro level it is trying to reduce the involvement of government and increase the role of the private sector through encouragement of farmers associations, privatizing some services where possible like the clinical veterinary services, and increasing the involvement of the commercial sector and NGOs in various aspects of the service sector. With prospects of a better balance between the government and private sector the Region is concentrating on developing the level of government services in a manner which are both financially sustainable and efficient. (ii) Within the government services the Region is initially concentrating on the transfer and creation of technology through improvements in the financing and efficiency of the extension and research services. It is foreseen that over time the private sector, NGOs and farmer associations may play an inireasing role in these activities. The Region has 20 staff in the field and several at HQ working full time on this effort. It is also undertaking two Regional Studies to examine and improve the effectiveness of their efforts. (iii) With the assistance of the government of Finland, Denmark and the U.S., the Region has established a small unit of three staff and consultant funds to help develop the farmer associations for savings, 62 ANNEX B credit, marketing and provision of inputs. They work both at the macro level to provide the conditions for their growth, and at the farmer level looking at the management problems of such associations. Many of these associations are linked to the extension effort and also involve women. Emphasis initially is being placed on savings and credit associations as a long- term solution to the problems of parastatal agricultural credit agencies. (iv) As part of the privatization issue, initial steps are being taken to use the IFC project preparation facility much more and to try to interest private companies, like Pioneer and Cargill to become more interested in seed companies (as in Cote d'Ivoire, Nigeria and Malawi), and agroprocessing activities. In the technology field the Region has been at the forefront in the Bank of looking at technology issues. AFTAG first examined the Region's experience with food crops in its study on some major ecological zones (WB Technical Paper No. 109 - Technology for Small Scale Farmers in SSA Africa by Stephen Carr), and the same author is being employed to look at some of the technology issues affecting the cash crops. Project staff are paying much more attention to available technology for extension and to the farmer/extension/research linkage. In research projects, apart from this essential linkage, emphasis is shifting from organization/reorganization to effective management, priorities for research and their funding and results. Agriculturalists and TMs in research and extension projects meet regularly to discuss and exchange experiences. The Region is now increasingly influencing the role of SPAAR in beginning to better mobilize African efforts in research by first associating them with SPAAR's deliberations and developing SPAAR as a forum for them to share experiences and coordinate their efforts. Efforts that Have Been Made to Disseminate the Findings The Study in Performance of Agricultural Projects in Africa has meant considerable discussion with all project staff on the measuring of past performance of the portfolio and their opinion 63 ANNEX 8 of likely future performance. These discussions have involved examination of individual projects as well as the three generic issues. While this Study has temporarily subsumed the role of the GT in disseminating lessons, the GT does raise lessons learnt relevant to the existing portfolio and discuss them with their staff (bearing in mind that the portfolio has changed its composition significantly). Apart from the need to continue to pay attention to the main issues, an important outcome has been the effort to reconcile the poor final outcome of projects with the generally more favorable ratings accorded in supervision reports during implementation. Projects consistently rated problems in the past usually were unsuccessful in the end, but there were still a considerable percentage rated favorably which subsequently were rated failures. Interestingly, on close examination of individual projects in the present portfolio, it was discovered that this bias was likely to continue unless supervisions more often examined the projection attainments of the project during implementation. As a result, a number of recommendations are being made to get better assessments. Nevertheless, risk will remain a major factor in Agriculture projects (see next para.). Impact on Onqoing and New Operations The three main issues are receiving much more emphasis than ever before in existing and the planning of new operations. However, it has to be faced that the macro/sectoral policy, institutional and technology problems in Africa remain, and as such inevitably must remain a major risk element. While the Region has to try and define these risks better in order to deal with them, the fact is that they are there. To improve from 50% to 75% will be a major achievement. There is also the worrying aspect that there seems to be no relationship between successful projects and a successful agriculture sector suggesting that our projects may not be aimed sufficiently at production opportunities so necessary for growth. The answer is not to cut the agriculture program to reduce the Bank risk (perhaps just the contrary). In the majority of countries unless agriculture moves forward the economies will not either. In fact the failures of agriculture raise some interesting questions for the other sectors. Agriculture's performance is measured by the world market place - if it cannot produce at world market prices it withers. Are these other sectors functioning at world competitive levels? 64 ANNEX B The fact that the major risks facing agricultural projects in Africa--macro -economic, managerial/institutional and technological--are large outside the scope of traditional agricultural projects, suggests that agricultural project performance depends critically on the performance of national economic and social system. That agricultural (and industrial) projects--the key productive sectors--perform badly in adverse African environments is not surprising as that other projects are said to perform so much better. It cannot be that agriculture and industry can be expected to transform the economic, governance and social environment in which they operate, but the other way around. The reason why the risks of agricultural lending are so high is precisely because of the lack of a functioning and supportive economic, social and political system. We clearly must look more carefully at how we can improve the performance of agricultural projects, but we have to look even more closely at whether everything else we are doing is improving the environment in which productive activities can survive and do well. The Region's Agriculture Lending Program is increasingly trying to address the sectoral economic, institutional and management issues with a major shift away from enclave rural development and ad hoc projects to projects directly addressing these issues (and also environmental forestry problems). Cote d'Ivoire - First Forestry (Loan 1735) PPAR No. 7952, June 30, 1989 The OED report's main findings and recommendations were that: (a) the project's industrial plantation approach did not permit to have a sectoral and country wide view of Cote d'Ivoire's forestry and environment problems and to deal with the complex issues of the agriculture/forestry interface; and (b) there was a major problem with the sustainability of government's efforts to protect the country's forest resources and manage efficiently its industrial plantation program. As recognized in the OED report itself, the recently approved Forestry Sector Loan addresses most of these issues: (i) its emphasis is on the protection and sustainable management of natural forests whose direct and indirect benefits are seen as 65 ANNeB higher than that of plantations; (ii) reforms in forestry taxes and logging policies are aimed at correcting distortions in the institutional environment and at promoting an efficient use of resources; (iii) industrial plantations are undertaken only on totally degraded soils and fallows with poor agricultural potential and in connection with natural forest management plans; (iv) the agriculture/forestry interface is tackled though the development of buffer zone programs including land-use planning and careful resettlement programs and though the involvement of local population in the preparation and implementation of forest management plans; and (v) the sustainability of government's forestry activities are being enhanced by the divestiture of some activities to the private sector, increased efficiently and reduced investment and operating costs, and improved marketing of SODEFOR's plantation products. Sudan - Second Agricultural Rehabilitation Program (Credit 1389) PPAR No. 7342, June 30, 1988 Regarding Sudan Agricultural Rehabilitation Program II, we have incorporated the lessons learned in the design of ARPIII, which was prepared in close consultation with OED staff. Documents on ARPIII were distributed to divisional staff and the experience was taken note of in other agricultural sector adjustment operations. B. Asia Regional Office Indonesia - Irrigation X, XIV, and XV (Loans 1578 and 1811, Credit 995) PPAR No. 7956, June 30, 1989 It is difficult to assess what lessons we have learnt from this audit report per Re. This is an audit for projects appraised in 1977-79. A decade has passed since these projects were appraised. During this period, we have absorbed lessons from several PCRs/audit reports. Reports from IIMI and IIRI, and through our own portfolio management. We have subsequently evolved a lending strategy which is consistent with the changing environment in, and needs of Indonesia. 66 ANNEX R Regarding the validity of the conclusions of the report, we had some disagreements, not because the conclusions were incorrect, but because they did not always present a complete picture. To cite a few exampless (a) the audit report stated that rice self- sufficiency was the motivating force in Bank lending, but did not add that rice self-sufficiency was a sound economic objective; (b) the audit report attempted to establish a linkage between irrigation and rice production - an impossible exercise, given the joint effects of various complementary inputs. Instead, the report could have emphasized the need to better monitor the rainfed and irrigated rice yields, and incremental project benefits. More recent issues such as competing uses of water, and inadequate planning for system operation and irrigation management practices, were transposed to the projects appraised in the late 1970s, although these were not the front burner issues then. Nevertheless, these and other conclusions of the report are relevant at present time, and the fact that they have been highlighted, is an important contribution of the audit report. During the last few years, we have been addressing a number of issues raised in the report. Our lending operations are focussing on O&M, cost recovery, privatization of small public schemes, increased involvement of beneficiaries, and full development of schemes. The audit report certainly provides a good check-list for our ongoing and proposed operations. The findings of this report were disseminated to various central and provincial agencies within GOI, other donor/international agencies such as ADB, FAO/CP, IIMI and IIRI. The report was also extensively discussed and commented upon by staff and management in Asia CD and TD, and RSI. Indonesia - First and Second Trade Policy Adjustment (Loans 2780 and 2937) PPAR No. 7947, June 30, 1989 Validity. Significance and Implications of Audit Findings The audit report gives a fair and balanced account of the performance and issues related to the First and Second Trade Policy 67 ANNEX B Adjustment Loans (TPALs I and II). Staff concerned with the preparation of the audit report did a thorough job of understanding the focus of the Government of Indonesia's (GOI's) adjustment program, carefully reviewed the Bank's strategy in responding to GOI's request for support, investigated the implementation of the policy measures through discussions with the Government and the business community, and presented a solid analysis of the impact of the reform programs. In erawing the lessons and implications for future adjustment operations, the audit report highlighted issues in three key areas: the policy framework; approach to adjustment lending; and GOI's policy formulation and monitoring capacity. * Policy framework. The audit report identified the following priorities for future reforms: (a) continued emphasis on prudent macroeconomic management; (b) emphasis on domestic resource mobilization; (c) further trade and investment deregulations; and (d) initiation of public enterprise reforms. * Approach to adjustment lending. The audit report endorsed the Bank's approach to adjustment lending in Indonesia, whereby loans are based on policy actions already taken, but within the framework of a medium-term adjustment program that had been clearly spelled out by the Government and agreed tL with the Bank. * GOI's policy formulation and monitoring capacity. The report emphasized the need to enhance the Government's in-house capacity to undertake trade policy analysis, monitor progress with policy implementation and assess their impact. These findings are similar to our own evaluation. First, the policy priorities indicated by the audit report are appropriate under Indonesia's present circumstances. Second, the Bank's approach to adjustment lending in Indonesia has been successful in securing a strong commitment from the Government, and avoiding 68 ANNER E difficulties on political sensitivities that are often raised by ex-ante conditionality. Finally, while GOI has already demonstrated a strong ability to formulate and implement difficult reforms within the framework of a medium-term program, there is scope for further enhancing its capacity for policy analysis and monitoring. Dissemination of Findings of the Audit Report Efforts have been made to dipseminate the findings of the audit report to three types of audiences: * Government. The audit report was circulated to the Government for comments. GOI reviewed the document carefully and sent comments to OED. * Other Donor Agencies. The document was given to representatives of OECF (Japan) and other donors in Jakarta who expressed their appreciation of the quality and objectivity of the analysis, and indicated that this would help in the assessment and design of their own operations. * Bank. The report was distributed to other departments in the Asia Region as well as to other regions in the Bank, based on an expressed interest by concerned staff in understanding and learning from the Indonesian experience with adjustment lending. The report was also drawn to the attention of PRE staff in the context of the Bank-wide review of adjustment lending. Impact on Onqoing and New Operations The findings and recommendations of the audit report have had a significant impact in designing the third adjustment loan- -First Private Sector Development Loan (PSDL I). The reform areas highlighted as priorities in the audit report-continued sound macroeconomic management, domestic resource mobilization, and further trade and investment deregulations--were all supported under PSDL 1. We now plan to undertake a fourth adjustment loan (PSDL II) which, in addition to consolidating and extending the reform program on the above areas, will also support public enterprise reforms--another area identified as priority in the OED audit report. 69 ANNEX E The audit report emphasized the importance of enhancing GOI's policy analysis and monitoring capacity. GOI has continues to develop this capacity, both on trade policy as well as in other areas including macroeconomic management, financial sector and public enterprises. Bank has been supporting this initiative actively through technical support from the Resident Mission (especially concerning trade policy) as well as through special missions from the headquarters (in areas of fiscal policy, public enterprises and national accounts). Indonesia - Second Water Supply (Loan 1709) PPAR Ito. 8622, April 30, 1990 (a) The following PPAR's findings are valid and significant: (1) Actual sales were lower than those forecast, partly because forecasters did not take adequate account of competition arising from "perceived" quality, reliability and availability of alternative sources (e.g., water from shallow wells). (2) The expected improvements in Unaccounted-for-Water did not materialize. (3) Cumbersome procurement procedures from GOI contributed to implementation delays. (4) Getting the civil works into over 100 contracts and evaluation of prequalification criteria for contractors contributed to project implementation delays. (5) The Internal Cash Generation Covenant would have been better than the Rate of Return Covenant agreed under the project. The creation of semi-autonomous water entities was the right step in principle but, because of staffing difficulties and the small size of the water systems, some of the entities have difficulties in being institutionally self-sustaining and financially viable. 70 ANNEX R (b) In the follow-on projects: (1) Sales Forecasts have taken into account the role of competition from other sources of water supply. (2) Specific measures to reduce UFW such as rehabilitation of the system and leakage control program have been introduced. (3) Dialogue between the Bank and GOI continues to further streamline GOI procurement procedures. (4) Civil works contracts are now optimally packaged and prequalification criteria have been applied more stringently resulting in reduced delays in project implementation. (5) The financial performance covenant has been modified. It is now fairly straightforward, readily understood and easy to apply, monitor and serves the purpose just as well. (6) Because some of the water systems are too small to be institutionally self-sustaining and financially viable, under the follow-on projects, the sector organization is under review. Some systems in a district are consolidated under one enterprise. (c) In addition, other important measures taken under the follow-on projects are: establishment of (i) project Management Unit which has the overall responsibility for project implementation; (ii) Project Implementation Unit which has the responsibility for day-to-day activities in project implementation; and (iii) Provincial Monitoring and Development Unit which has the responsibility to investigate and monitor the operational and financial performance of water enterprises. Substantial amount of technical assistance has also been included under the follow-on projects to provide training and to introduce adequate accounting/financial systems. 71 ANNEX E Philippines - Rainfed Agricultural Development (Loan 1815) PPAR No. 7949, June 30, 1989 The OED audit concluded that the project was successful and largely sustainable. The audit's principal findings and lessons learned were: (a) The rainfed cropping component could have been even more successful on a wider scale had various other components not been added to it; (b) the failure of the monitoring and evaluation system to produce timely analysis of field impact, although much data was collected, is a typical failing of such activity. Formal M & E arrangements should have been agreed at the outset; (c) the design of the project management system for such a complex multi-agency project was flawed. The project management unit and committee system proposed did not function as intended, partly because the committees were too large and unwieldy; and (d) although the rainfed cropping technology in Iloilo province is low cost and yields high benefits, the Bank had financed irrigation investments on areas contiguous with the rainfed crop areas. This indicated the need for careful evaluation of all production options when preparing and appraising irrigation projects. The Region largely agreed with the audit findings, but expressed some disagreement with regard to (a) and (d) above, in a memo on May 18, 1989 commenting on the draft PPAR. With regard to (a), the Region pointed out that at the time of appraisal, support and social services were deficient and some improvement was deemed necessary to complement production aspects. Despite severe counterpart funding constraints, several of these other components viz. the human nutrition and health, pilot village development, staff/farmer training, livestock, and revised marketing components did succeed (PPAR, Evaluation Summary, para. 6) and clearly benefited the rural poor. However, in general, the audit findings is relevant and the trend in this Department over the last few years has been to design focussed agricultural and rural development projects with relatively few components. 72 ANNEX B With regards to (b) above, the audit finding is valid and formal M&E arrangements are now regularly developed during project preparation and appraisal. Two recent examples are the establishment of a computerized management information system as an aid to effective performance monitoring as a component in FLECRA II Project (Loan 2917-MA) and RISDA Project (Loan 3139-MA). The audit finding at (c) above is also valid. There is now a clear trend to move away from complex multi-agency projects. Most of the recent agricultural projects in the Department have dealt with single agencies; in the others, there has been a conscious effort to minimize the number of agencies involved. The Region's disagreement with the audit finding at (d) is reflected in PPAR para. 52 and footnote 17. The irrigation investments on areas contiguous with the rainfed crop areas were under the Jalaur Irrigation Project (Loan 1367-PH) which was appraised in June 1976 when few farmers were using the rainfed technology in Iloilo province and the technology was still being verified. It is not clear whether the appraisal mission was aware of the trials and if so, whether the mission considered it premature to evaluate the technology as an "option". In later years when the rainfed technology (dry seeding) became well known, even irrigated farmers used it. This allowed the irrigated farmers a high certainty of two high yielding crops, whereas rainfed farmers suffered occasional losses due to short drought periods during the cropping season. However, the lessons regarding evaluation of options is valid and close attention to this aspect needs continued emphasis. It is being addressed in the planned Philippines Irrigated Agriculture Sector Study (FY91) which will, inter alia, review yield potential under irrigated and rainfed conditions by region and major agro-climatic zone and evaluate alternative strategies based on irrigated and rainfed farming. The Study will review existing reports and data, including OED reports. Korea - First Water Supply (Loan 2072) PPAR No. 8174, November 6, 1989 The project objectives were largely achieved and the implementation was reasonable within appraised parameters. The audit correctly identifies several problem areas which are valid for Korea and other countries. Of these the key issues are difficulties with assumptions behind projection of population and industrial growth, and high water losses. 73 ANNEX E This project was the first of five water supply projects so far completed or under implementation in Korea. In all of them the assumptions behind projections of population and industrial growth were the most discussed item. In Korea projections are provided for all cities and planning areas centrally by the government, are applied to all development projects in all sectors, and use of other data is generally not allowed without a very substantial reason. As a result of the experience with this project and the findings of PCR and PPAR staff have focussed in follow up projects on establishing realistic water demand projections without challenging underlying data. Extensive coverage in the audit report given to high water losses resulted in greater attention of the government and the cities to the subject. The cities established long-term programs for leakage control and reduction supported by permanent units with responsibility only for this activity. In addition, the government decided to implement from 1990 through 1995 a National Water Improvement Program which includes among others replacement of about 27,500 km of old corroded and leaking pipes in the cities in order to reduce water losses. The lessons provided through the first project were also used by the staff in the design of follow up projects with regard to reduction of water losses. In Korea the Water Supply II project included leak detection and control which was successfully started in fifteen cities, the Water Supply IV project included substantial rehabilitation of the distribution system and strengthening of the leak detection unit in Taegu (which, in 1990, achieved a reduction of losses to 28%, the lowest in Korea), and the Juam Water Supply Project (Water Supply V) ensures funding for a long-term leak detection program, including rehabilitation of old corroded pipes in Kwanju. C. EMENA Regional Office Yemen PDR - Greater Aden Water Supply (Credit 1038) PPAR No. 7553, December 27, 1988 The findings of the PPAR and PCR are valid. The three main findings were the importance of: (i) thorough investigation of groundwater resources during project preparation; (ii) institution- 74 ANNBX E building components; and (iii) the shortage of qualified and experienced staff. These findings and their recommendations have been discussed extensively with the Borrower and taken into account by both Bank and Borrower staff in the supervision of ongoing projects and in the design of new operations. Specifically, project preparation activities for all ongoing projects have ensued that adequate groundwater investigations were carried out in advance of appraisal. In addition, for the follow-up Greater Aden Second Water Supply Project (Credit 1739-YDR), a leak detection program and staff training has been included. Under the Al Mukalla Water Supply Project (Credit 1944-YDR), there are components to: (i) improve system analysis and control water installation of computerization of billing and accounting and staff training. A financial study will also review the appropriateness of the step tariff structure, assess the ability of consumers to pay and determine the price elasticity of demand. Under the proposed Tarim Water Supply project, further steps to improve the Public Water Corporation's operational and managerial efficiency include: (i) an intensive metering program to reduce unaccounted-for-water and promote energy conservation; (ii) devising a manpower development program and staff training, including assessing the feasibility of establishing a national training center for water supply; and (iii) implementing management information and inventory control systems. This project also includes a training component aimed at improving the accounting and auditing procedures and practices of the Central Organization for Control and Auditing, the government auditing agency. Turkey - TSKB XIII and XIV, Private Sector Textiles, Labor Intensive Industry, D.Y. Bankasi II and II (Loans 1749, 1754, 1755, 1952, 2093, 1379, 1998) PPAR No. 7883, June 29, 1989 and PPAR No. 8614, May 4, 1990 Validity, significance and implications of Audit/PCR findings: The lessons from the credit lines to private sector were significant in raising the Bank's awareness that (a) the Bank needs to diversify its channels and (b) the lack of branch network of these DFCs impedes a wide geographical distribution of subprojects. The above operations were the last in Turkey to include solely DFCs. Subsequent credit lines recruited several public and private commercial banks which account for 80% of the total banking sector. This has increased competition among financial institutions for Bank funds and improved the accessibility of sub-borrowers due to the vast branch network of these institutions. 75 ANNEX a Efforts to Disseminate Findings within Region% There are no organized or formal means to disseminate the findings of the PCRs. However, at every stage of project preparation and review, attention is drawn and references made to problems and issues both in completed and ongoing projects. As such, despite the lack of any formal review, the design of subsequent projects does taken into consideration the lessons learnt from past operations. Impact on On-going and New Operations: As mentioned above, the most important results of the above operations was that the Bank set out to increase competition among public and private commercial institutions in channeling medium and long-term (MLT) funds to industry. The increased competition for Bank funds has had institution building effects on the commercial banks for which MLT lending was a new area. As a result, they have established project appraisal expertise in line with Bank appraisal guidelines. In addition, the increased competition in an area which was the domain of the DFCs, has forced the DFCs to diversify their product lines and funding sources. TSKB, the larger of the two private DFCs, has been successful in becoming a sophisticated and diversified institutions; SYKB, the smaller one, has also introduced new services for its clients and attracted funding from other foreign sources, but to a lesser extent than TSKB. In general, this shift in focus in Bank lending has broadened the availability of financial services within the banking sector. D. LAC Regional Office Brazil - Export Development (Loan 2347) PPAR No. 8327, January 12. 1990 The 1983 project was intended to provide a quick-disbursing loan to encourage expansion in Brazil's manufactured exports by improving the conditions under which the duty drawbac. system (DBS) operated (particularly the automaticity with which DBS imports could be made). The Brazil Department produced a Project Completion Report on June 24, 1988, and OED a Project Performance Audit Report on January 12, 1990. The findings of the two reports are best synthesized and compared according to different specific types of lessons emerging from the two analyses (paras. 6.01 to 6.07 of the PCR and paras. 10 to 20 of the Evaluation Summary of the PPAR). 76 AM E i. The appropriate policy framework: a major conclusion of the PCR was that perfecting the DBS without a consistent policy framework emphasizing the key role of import liberalization in export development was inappropriate. The PPAR was more inclined to find an effective DBS system as an acceptable initial stage in a gradual trade reform process. (By the same token it was also generous in its judgements of the gradualistic trade-policy reforms achieved since 1987; the PCR is silent on these. ii. Consecutive Bank operations: the PPAR, in line with this greater belief in a gradual process, regretted the absence of a follow-up trade-reform project as had initially been envisaged. It can be inferred that such a criticism is absent from the PCR because of the PCR's belief that reform in the DBS was the wrong starting point. iii. Were the project's objectives obtained? notwithstanding the apparent differences in the two documents about the appropriate policy framework, both agree that the project did not achieve its own policy objectives, consisting of different steps to automate and enlarge DBS imports. iv. The extent of sanctions built into the project: both documents agree that the project did not have enough instruments to assure that agreed policy changes would be implemented, the PPAR specifically criticizing the use of a quick (18-month)-disbursing loan to accomplish a four-year implementation schedule. v. Clear criteria for supervision: both reports criticize the problems for supervision posed by vague (and arbitrary, according to the PPAR) criteria for policy changes. vi. "Politics" of lending: the PPAR feels that the Bank should have been more sensitive to the implementation problems of a loan to weak government in an economic crisis. The PCR has no comment in this area. 77 ANNEX E vii. Convincing the government of the merits of the proposed policy changest the PPAR criticizes the Bank for not having done a better job. While the PCR has no comment on this, it might be inferred from this report's criticisms of CACEX (the implementing agency) that the problem was not so much one of insufficient proselytizing, as one of the Bank being wrong in trying to take a reformist approach to an institution whose position was so fundamentally opposed to the direction of the reform. The Department began to prepare a trade policy reform SECAL in 1987, interrupted this in early 1989, but expects to resume preparation in a few months' time. From 1987, the Department has continued to emphasize the need to start reforms with thoroughgoing import-liberalization measures. It is unlikely that the Department would entertain a SECAL for reforms taken on the export side or for overly gradual import reforms. (It might entertain an investment project on the export side, but only as a second-best to an import- liberalizing SECAL). Thus, on the issue raised under (i) above - - the issue where any divergence between PPAR and PCR is at its most important -- the Department has been more inclined to consistency with the PCR, rather than with the PPAR. The Department is also aware of the need for effective provisions to attain the project's objectives (headings (iii), (iv) and (v) above). It would do this through a combination of: (a) making sure that the implementation of most major reforms was achieved by the time of loan effectiveness; (b) trenching for further reforms, if necessary; (c) clear (thus supervisable) criteria for gauging reform measures taken; and (d) consideration of a follow-up loan for reforms in the longer term. Finally, it is self-evident that any good project has to make the "political" judgements required under (vi) and (vii) above. Brazils Medium-Sized Cities (Loan 1720) PPAR No. 8302, December 29, 1989 The Medium-Sized Cities Project (MSC) was the first to introduce in Brazil modern urban management concepts and techniques, such as cost recovery, affordability, replicability, poverty targeting, the constitution of municipal funds, and project appraisal and supervision into the administration of medium-sized 78 ANNE cities in Brazil. Three priority investments categories -- employment and income creat .on, urban infrastructure and services, and municipal finance and administration -- were identified. Other major project objectives were to strengthen the technical, managerial and coordination capacities of urban development agencies at both national and local levels. The goals of the project were more clearly achieved in the urban and infrastructure components, where high visibility and popular appeal guaxanteed high government priority. The income creation and municipal administration components were not as successful, due in the first case to lack of detailed understanding, and in the second due to the effects of rapidly increasing inflation on tax revenues. In both bases clear lessons were learnt about project preparation for future Bank work in the sector. The institutional context in Brazil is now very different from that existing at the time the MSC project was appraised. A similar operation will no longer be replicable in its original form. However the lessons learnt have had a significant impact on the way integrated local development is approach in many parts of Brazil, and has influenced the Bank's strategy for the urban sector. Specifically, the Project demonstrated that (i) local governments had the capacity to select and implement high priority physical investments; and (ii) that greater emphasis was required to strengthen human resources. The Bank's urban development strategy in Brazil is presently geared to the development of core programs focused on Municipal Development Projects (MDPs) in various key states in Brazil, including the Northeast with the objective of inducing significant institutional and policy changes on a national scale in this sector. This is particularly important at this time, when the institutional disintegration of sectoral agencies at the federal level is especially pronounced. The experience of the project demonstrated the need to directly involve the traditional structures of local government for long term institutional strengthening, a lesson that was incorporated in subsequent Bank financed project in the sector such as the Parana Market Towns Improvement Project - PRAM (Ln 2343-BR), the Santa Catarina Small Towns Development Project (Ln 2623-BR) and the most recent operations in the states of Parana (Ln3100-BR) and Rio Grande do Sul (Ln 3149-BR). 79 ANNEX B Brazil - Supporting Rural Developing in the Northeast (Loans 1488, 1537, 1728) PPAR No. 7910, June 28, 1989 Report 7910 was a combined audit for the three "first generation" Northeast Rural Development Projects. Because of the timing of both the PCR and the PPAR (after implementation of the "second generation" PAPP program had begun), both reports were more "summaries after the fact" as opposed to trend-setting documents. However, a good part of the findings of both documents reconfirmed early, informal findings during preparation and appraisal of the follow-up PAPP program and thus have been reflected in both the initial design and subsequent supervision/modification of that program. Nonetheless, it should be pointed out that the Region made extensive comments disagreeing or qualifying many of the statements in the initial drafts of the PPAR, some of which were incorporated as footnotes to the OED audit document. This division still continues to be of the view that the PPAR gave greater emphasis to some general messages (e.g., importance of the general policy environment on investment project performance; the inadequacy of official rural credit policy in Brazil; the generic problems of integrated rural development) than a specific analysis of the performance of the three projects in question. We do not question the ge.aeal validity of many of the issues raised in the PPAR, but the PPAR seemed to us to be far too negative in tone when compared with the apecific findings and performance of the projects in questions. Mexico - Industrial Equipment Fund III and IV (Loans 1560 and 1712) PPAR No. 7859, June 13, 1989 The main criticisms of the OED audit was that: (i) the project design was based on insufficient understanding of financial sector issues and did not adequately reflect the effect that macro policy changes could have on the project outcome, and (ii) project supervision was minimal and focussed on short term project performance rather than the deteriorating macro policy environment. 80 ANNEX B The findings that financial sector problems were either ignored or not addressed sufficiently needs to be put into the proper perspective. Successive Mexican governments had been unwilling to commence a dialogue with the Bank on overall financial sector policy, and the Bank had taken a high level decision to nevertheless proceed with lending through financial intermediaries. Also, excessive segmentation of financial markets and proliferation of directed credit facilities were not considered fives years ago to be as detrimental to the development of efficient financial markets and they presently are. In addition, the Bank did not overlook interest rate subsidies, as is indicated in the audit. The GIRA system was established by the Bank and the Government to link all Bank financing into a market based indexing system, with the explicit objective of reducing and eventually phasing out existing interest rate subsidies. When the present Government took office, it indicated that it was prepared to tackle the whole spectrum of financial sector issues, and a dialogue was begun which culminated in the FY89 Financial Sector Adjustment Loan, which substantially resolved most outstanding financial sector issues. The apparent lack of adequate supervision, especially of relevant macro-economic developments and their impact on project execution, appears to have been due to low priority given to this activity. Coupled with a lack of country reporting and weak coordination between program and project divisions, this resulted in a failure to take into account the deteriorating macro policy environment. Th! situation began to improve rapidly after the Bank reorganization took place (i.e. about one year after the Fourth PONEI Loan was closed). Not only was the programs-projects controversy resolved, but much more emphasis has been placed on proper supervision. With the recent absorption of FONEI by NAFIN, reflecting the Government's policy of consolidating directed credit facilities, the supervision of FONEI is no longer an issue. OED also pointed out that the performance of FONEI's portfolio did not improve, in spite of better project selection criteria and improved project appraisal and supervision methodologies. It seems to us that this was mostly a reflection of the effects of the economic recession of the mid-1980s (which affected the performance of almost all FONEI's borrowers) rather than of poor performance by FONEI. 81 ANNEX E Bolivia - Emergency Social Pund (Credit 1829) PPAR No. 8449, March 28, 1990 The findings of the Project Performance Audit Report (PPAR on the above project were valid and relevant to the preparation of follow-up operations in Bolivia. The report findings included, among others, the under-representation of the very poor (lowest ten percent) among the benefitted population, and the challenge for ESF's successor institution to reach these marginal groups' the ability of the institution to work with NGOs, an outcome which was made possible given the wide range of investments/interventions, thus increasing the likelihood of a broad response by potential project entities; the deliberately flexible program content which permitted en-route adjustments; the excellent and hard driving leadership of the program and the quality and dedication of the staff, and the need to put in place an appropriate system of maintaining and operating the physical assets created through the program. The new Social Investment Fund Project (SIP) put the first and last points on its agenda by developing a methodology to improve poverty targeting, and by assuring that operating costs of subprojects would be covered beyond the investment period, either by the corresponding Ministry (in which case the Ministry's prior approval of the subproject would be sought), or by the responsible entity by making the approval of subprojects contingent upon presentation of a satisfactory plan for continuation, and demonstration that the costs had been included in the budget of the responsible entity. In terms of the replicability of the program, the audit report provided important remarks which should prove to be useful to countries planning to create similar institutions. They include, among others, the need for strong and unflagging governmental support at the highest level, combined with deliberate restraint from political intervention in daily operations; the bottom-to-top approach in developing effective relief proposals, the establishment of a financial intermediary (rather than an implementation) agency outside a regular Government structure, a sufficient degree of flexibility in criteria and procedures for project preparation and implementation, and excellence in leadership and staffing. These remarks have proved to be valid and 82 ANNEX 8 significant for the preparation and initiation of the new created SIF. The need for deliberate restraint from political intervention and the excellence in staffing continue to be two very important points to be followed and to be closely supervised by Bank staff in order to achieve the results obtained with the first Social Emergency Fund Project. Dissemination of the above findings has been substantive. Given the interest from the LAC region and from other regions on the ESF experience, several seminars have been organized on the topic. They have focused on past and current ESF experiences, and on the transition to the Social Investment Fund. During such seminars, the Audit/PCR report has been recommended to staff working on the preparation of similar programs in other countries. With regard to impact of the above findings on new operations and as was mentioned above, the PPAR findings were taken into account in the preparation of the Social Investment Fund Project. Furthermore, staff working on the preparation of similar institutions (i.e., Guatemala) have been referred to the Audit/PCR report. The report findings and conclusions have been fully supported by Bank staff in the preparation of similar projects. Regional and Departmental Feedback Attached are some recent examples of lessons learned in PCRs successfully incorporated into the design of subsequent lending operations. These reviews have taken place at a Departmental level, particularly in the context of the annual ARIS exercise, where the Departmental report specifically summarized the major lessons learned to be incorporated into subsequent lending operations. We have also participated in regional reviews of PCRs by sector in order to maximize cross fertilization throughout the Region in addition to disseminating and discussing special reports prepared for a Bank-wide audience (e.g. Levy report on financial sector reform). 83 ANNEX E Lessons Learned from Selected PCRs LA4) A. Financial Sector Several PCRs were influential in the design of new projects in the financial sector. In particular, five PCRs on credit line operations, prepared during the last eighteen months, have provided important lessons for new operations under preparation: a Second Industrial Ftnance Project in Ecuador and an Industrial Sector Loan to Paraguay, which has a credit line component. These PCRs are: - PARAGUAY: Industrial Credit and Regional Development Project (Loan 1866-PA) - ARGENTINAs Industrial Credit Project (Loan 1463-AR) - ARGENTINA: Second Industrial Credit Project (Loan 2063-AR) - PERUs Industrial Credit Project (Loan 1358-PE) - PERUs Second Industrial Credit Project (Loan 1968-PE) Some of the lessons learned from these PCRs are country specific; but, for the most part, they are common to any type of credit line operation. The PCRs show that these loans share some common characteristics. First, the borrowers or the main channel of funds were Government DFCst BANADE was the borrower of both loans to Argentina and COFIDE of both loans to Peru. Loan 1866- PA was made to the Republic of Paraguay, but most of the funds were lent through CPN. Two loans: the Second Industrial Credit Project to Peru and the loan to Paraguay, involved other financial institutions as loan retailers; the former using COFIDE as mechanism in the Central Bank. Second, most of the loans were justified on the grounds that they would support: (i) the restructuring or reorganization of the main lending institution 84 ANNEX 9 and/or (ii) the liberalization of financial or trade policies by a new government. Eventually, the expected institutional improvements did not materialize and the policy framework was either not implemented or reversed during the life of the loan. Third, the financial situation of the borrowers did not improve, and in several cases deteriorated, during the life of the loan. Fourth, the solution to some institutional and financial problems identified during supervision was left to a follow up operation, which was designed without the benefit of a full PCR. Some of the lessons from these PCRs that we are taking into account in the design of the new projects are the followings a) It was difficult for the Bank to convert a public entity into an effective financial intermediary with proper financial discipline and independence from political influence. Even carefully drafted statements of policies and procedures, and strong supervision efforts, failed to prevent political influences over the financial decisions of a government DFC, particularly with respect to loans they make with non-Bank resources (such as Central Bank rediscounts). In all five loans, but to a higher degree in those to Argentina and Peru, the borrowers were used to finance state owned enterprises and, so called, priority projects. COPIDE, BANADE, and to a lesser extent CNF are in serious financial difficulties as a result of excessive exposure to loans and guarantees to state enterprises and the low quality of their loan portfolio to the private sector. In future, we will avoid having public banks lend for public enterprises. b) It was difficult for the Bank to monitor the financial performance of the borrowers and to react to their institutional and financial deterioration. The use of maximum debt/equity ratios as the main, and often only financial covenant in those loans was ineffective to monitor the financial soundness of public banks, because governments were ready to 85 ANNEX E assume the debts of these institutions. Moreover, given different accountancy standards concerning accruals and provisions, the debt/equity ratio tended to be an unreliable indicator. When, as in the cases of COFIDE and BANADE, the government was eventually unable to assume the debts and the banks failed to comply with the debt/equity covenants, the Bank acted leniently and accepted to modify the covenant. c) It was difficult for the Bank to react to the reversal of the policy reform framework that justified the loans to Argentina and Peru, because credit lines, at the time, were seldom linked to policy conditionality. The Bank should link disbursements under a loan to the effective implementation or maintenance of the policy framework that had originally justified the loan. d) There was a serious conflict of interests when the same institution acted as both first and second-tier lender. This was the case of the second loan to Peru, where COFIDE had little incentive to lend through other financial intermediaries. We will avoid this in future lending operations. e) Under unstable macroeconomic conditions, with high inflation and periods of under and over valuation of the currency, it was difficult to pass on the exchange risk to the ultimate borrower. In Peru and Argentina, many of the dollar denominated loans went to non-exporting firms, which eventually found it impossible to repay the loans. We could infer that a number of these loans were stress borrowing or were taken with the intention of not repaying them. The latter is a plausible assumption, given the low collection efforts of most government DSFCs. f) Targeting different components of the loan to specific end-uses was ineffective: often the project designers miscalculated the demand for certain 86 ANNX H components and resources had to be reallocated several times. Two examples of failed targeting of loan components are the loan to Paraguay and the second loan to Argentina. B. Other Aruentina: The PCR of SEGBA IV and the lessons learned during supervision of Loan 1761-AR led to the innovative Power Sector Loan (2998-AR), which had a limited number of sector wide covenants regarding the least cost development plan and the financial recovery of the sector through appropriate tariff levels and cost reduction measures as opposed to the enterprise specific conditions of previous power loans. Chile: The PCR of Loan 2482-CH was used to prepare the Second Chile Housing Loan (Ln. 3030-CH). This second operation focuses more on mortgage cost recovery with proper incentives to improve loan collections by Regional SERVIUS by linking their participation to collection performance, using private firms as collector agencies, and eliminating routine reschedulings. Moreover, upfront direct subsidies were more clearly targeted to priority beneficiaries. Uruays The lessons from the PCR for the Fifth Power Project (Loan 1779-UR) were used for the Uruguay Power VI Project, including the design of an innovative institutional procedure as a condition to ensure that all government entities pay their bills to each other on time through the establishment of a special clearing account for all public agencies to net our overdues (which had been a long-standing problem for the public sector power utility --UTE--amongst others). 87 ANNEX P IFC INVESTMENT ASSESSMENT REPORTS FOR FY90 FILED IN THE OFFICI OF THE DIRECTOR-GENERAL, OPERATIONS EVALUATION Country Company 1 Fiji Sheraton Fiji 2 Chile Inverchile 3 Haiti Prinsa 4 Egypt 81 Dikheila Steel 5 Yemen Yemen Battery 6 Yugoslavia Tovarna (TGh) 7 Tunisia Comete-Engineering 8 Brazil Brasilpar 9 Liberia Liberian Timber 10 Malavi Nalavi Hotels 11 Argentina Terminal 6 12 Hungary UNICBANK 13 Kenya IPS-K 14 Uruguay Surinvest 15 Peru Ninera Regina 16 Barbados CFSC 17 Brazil Triunto 18 Brazil Alguim 19 Mexico Netalsa 20 Bangladesh Bata Shoe 21 Pakistan PPL 22 China China Bicycles 23 Philippines Davao Union Cement 24 Uganda Uganda Tea Corp 25 Brazil Quimica da Bahia 26 Indonesia P.T. Bali 27 Uganda SCOUL 28 Malawi Viphya Plywood 29 India Hero Honda 30 Mexico Celpap 31 Grenada Issa Nicholas 32 Chile Cocar 88 ANNEX G IPC SPECIAL EVALUATION STUDIES Subject Full Version Abridged Version 1/ Hotels IFC/SecM81-71, 6/30/81 - Cement IFC/SecM83-68, 8/12/83 IFC/SecN84-74, 08/84 Non-fuel Minerals JAC86-25, 10/15/86 IFC/SecN86-164, 12/19/86 Agricultural Processing & Storage JAC87-25, 11/16/87 IFC/SecK87-218, 12/31/87 Agricultural Production JAC89-6, 2/22/89 IFC/SecM89-47, 3/7/89 Textiles JAC89-42, 12/14/89 IFC/SecM89-157, 12/29/89 1/ Case studies and company names deleted to permit dissemination to member countries.