Report No. 4679 Annual Report on Operations Evaluation August 24, 1983 Operations Evaluation Department FOR OFFICIAL USE ONLY Document of the World Bank This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. л . А FOR OFFICIAL USE ONLY THE WORLD BANK Washington, D.C. 2G433 U.S.A. Office of Direct(w-General Operations Evaluation August 24, 1983 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Annual Report on Operations Evaluation Attached, for information, is this year's "Annual Report on Operations Evaluation." The report reviews the current status of the process by which the Bank, in its continuing search for ways to improve the effectiveness of its development assistance, evaluates the results of its operational activities and communicates them to its members and its staff. The report also contains a summary of the current status of the recommendations in OED major policy reviews and evaluation studies. Attachment This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. FOR OFFICIAL USE ONLY ANNUAL REPORT ON OPERATIONS EVALUATION Table of Contents Page No. I. Introduction 1 II. The Project Performance Audit Reporting System 1 Borrower's Role 4 III. Monitoring and Evaluation in Projects 5 IV. Post-evaluation in Member Governments 7 V. The Operations Evaluation Department 7 VI. Other Evaluation Activities 10 IFC 10 Research 11 Economic Development Institute 11 VII. Follow-Up 12 Annex A: Status Report of the Regions on 13 Borrower Governments' Involvement in the Post-Evaluation of Bank/IDA- Financed Project9 (All countries with five or more PCRs scheduled through CY84) Annex B: Status Report of the Regions on 19 Borrower Governments' Involvement in the Post-Evaluation of Bank/IDA- Financed Projects (Selected countries with less than five PCRs scheduled through CY84) Annex C: The Current Status of Recommendations 23 of OED Policy Reviews and Evaluation Studies Annex D: OED Reports Issued During FY83 44 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. ANNUAL REPORT ON OPERATIONS EVALUATION Abbreviations used JAC Joint Audit Committee OED Operations Evaluation Department PPAR Project Performance Audit Report PCR Project Completion Report OPS Operations Policy Staff IFC International Finance Corporation EDI Economic Development Institute M&E Monitoring and evaluation OMS Operational Manual Statement ANNUAL REPORT ON OPERATIONS EVALUATION I. Introduction 1. The purpose of this report is to review recent develop- ments in the Bank's operations evaluation activities and the status of still outstanding recommendations of past reports of the Operations Evaluation Department (OED). Details of the earlier evolution of operations evaluation in the Bank are available in past Annual Reports: R76-245 of October 12, 1976 to R82-269 of September 10, 1982. II. The Project Performance Audit Reporting System 2. The 1983 fiscal year was the first in which the modified audit system was in operation. The new system has developed thus far without difficulty. Of the 169 completed projects on which re- ports were issued, OED staff audited 91 projects (54%) and scrutinized and passed-through without audit another 78 project completion reports. Some 60% of the performance audits were based on country visits by OED staff. Borrower comments continued to be sought by OED on all draft project performance audit reports (PPARs) and pass-through project completion reports (PCRs). The 1983 Annual Review of Project Performance Audit Results also remains comprehensive in its coverage. As intended, the modified system (a full description of which was contained as Annex A to last year's Annual Report) has enabled OED's limited staff to focus more sharply on significant project experiences and to begin to group projects for audit by country and sector for commentary on larger issues of sector policy and institutional environment. One result of these changes has been an increased flow of borrowers' and their agencies' comments on draft reports. Another result has been to extend the elapsed time for completing the performance audit cycle, especially where borrowers' comments are received with some delay. 3. Scheduling of audits and the proportion carried over from one year to the next is influenced by the timing of the receipt of PCRs which, as noted in earlier reports, have reflected the rhythm of other operational activities in the Regions. Almost one-half of the 180 PCRs received by OED in FY83 arrived in the final quarter of the year, and 30% in the month of June. The table below shows the status of PCRs received from FY80 through FY83 and the esti- mate for FY84. It may be noted that the upward trend of PCRs re- ceived as well as of PPARs/PCRs issued continues. However, in line with the highest number of operations undertaken in any one year in the past, the number of projects to be annually reviewed beginning in FY85 and beyond is expected to level off at around 250. - 2- Status of PCRs Received Actual and Estimated FY80 to FY84 A c t u a 1 Estimated_/ FY80 FY81 FY82 FY83 FY84 PCRs brought forwardb/ 106 87 136 173 184 PCRs received 90 143 159 180 241 Total 196 230 295 353 425 PPARs issued 109 94 122 169.c/ 241SJ PCRs carried forward: 87 136 173 184 184 as percentage of Total above 44% 59% 59% 52% 43% a/ The FY84 estimate of PCRs received is based on the numbers of loans and credits with closing dates on or before the end of Calendar Year 1983 and which were more than 85% disbursed, adjusted as a result of discussions between OED and Projects staff. The resulting figure of 241 is greater than the earlier estimate of 193 PCRs contained in the Bank's FY84 Administrative Budget. The situation will be reviewed at the time of the mid-term budget review. b/ At various stages in the audit process. c/ The FY83 total comprises the issue of 91 PPARs and 78 pass-through PCRs scrutinized but distributed without audit. For FY84, it is envisaged that 119 PPARs will be issued and 122 PCRs distributed without audit. 4. New guidelines are being developed by OPS for the preparation of PCRs, and are expected to be circulated shortly. In this connection, OED staff have, on the basis of recommendations contained in specific audits, suggested that the new guidelines require PCRs to include a sensitivity/risk analysis on re-estimated rates of return at project completion, and that the assumptions and other data on which the re-estimated rates of return are based be set forth in an annex to the PCR. The sustainability of projects and their benefits after completion will also be a matter for more systematic review and comment in future PPARs. - 3 - 5. In FY83 the cost to operating units of preparing each PCR was 10 staffweeks on average and, to OED for preparing audits in respect of those projects selected for audit, about 6 staffweeks. The time spent by OED staff on PCRs not audited is estimated at half a staffweek for each PCR, resulting in an overall FY83 average of 3.5 OED staffweeks per PCR. In FY84 the average time spent on PCRs is estimated to remain at 10 staffweeks while a slight decline to 3.2 staffweeks has been budgetted in respect of average OED staff time, reflecting a larger proportion of PCRs to be reviewed but not audited--51% in FY84 as against 46% in FY83. The overall cost to the Bank of the audit process for FY84 is thus currently estimated at about 58 staffyears, comprising 39 staffyears for PCR preparation and the balance of 19 for the OED input. OED's current estimates of PPAR costs may have to be revised upwards somewhat, however, as experience accumulates with the cost of managing the interactions with staff and borrowers associated with its much larger pipeline of work-in-process. 6. Since 1977, Sub-Committees of the Joint Audit Committee have reviewed a number of PPARs each year to assess the adequacy of the PPAR system. The Sub-Committee for FY83 reviewed 21 PPARs, of which seven covering 11 projects were studied in depth and dis- cussed with OED and the concerned operating staff. The Sub- Committee concluded that the PPARs analysed provided an objective and thorough assessment of project implementation and performance. Among its several recommendations were that PPARs should place greater stress on policy aspects of projects, and that further attention be given to audits of non-project operations, especially structural adjustment loans. The Sub-Committee's recommendations were discussed by the Joint Audit Committee at its meeting on July 25, 1983. 7. One of the general recommendations made by last year's Sub-Committee and endorsed by the JAC was that the findings of PPARs should receive wider dissemination in the Bank, and that the procedure adopted for this purpose by some operational policy de- partments should be adopted by others; a commitment for a further commentary on this subject was contained in last year's Annual Re- port. In the course of FY83, OED staff conducted an informal sur- vey of operational staff at different levels with a view to as- certaining the extent to which OED reports were made available to and used by them in the course of their ongoing work. The findings varied as between sectors and Departments, but one general con- clusion stands out. The objective of using these reports to dis- seminate Bank operational experience across internal Bank functional or geographic boundaries and to borrowers across Regions appears not yet to be widely shared in the Bank. While there appears to exist an interest in OED reports, other commitments re- portedly prevent more than a cursory and selective reading of them bS+ staff, which is generally limited to the work related to the countries and sectors to which staff are assigned. There seems to -4- be an even lesser readership of other OED reports. It also appears that readership is less at levels below Division Chief because of time constraints, lack of awareness of relevant OED reports, and the feeling of some that little is to be learned from these re- ports. However, there are many who would welcome the systematic distribution of the "Highlights" of PPARs or periodic summaries by OPS of the principal lessons learned from the audit process. The "lessons learned" from PPARs on agricultural projects which are distributed periodically by the Agriculture and Rural Development Department. (AGR) of OPS to all sector operations staff are generally widely read and appreciated; a similar procedure is now to be initiated by the Water and Urban Development Department of OPS. To enhance dissemination further, OED staff have been joining AGR and PMD trainers in staff seminars and project design courses using PPARs as case material; and OED staff, in their individual capacities, are participating in a number of working groups and steering groups together with staff from other departments. 8. The Office of Environmental Affairs of the OPS Project Advisory Staff (OEA) continues to examine the environmental aspects of projects where relevant. Projects due for audit are reviewed by OEA to identify those likely to have environmental implications, and periodically OEA send a list of such projects to OED. Thirty- two PPARs distributed to the Board in FY83 had been reviewed by OEA. A sample of comments from PPARs on the environmental aspects of the project experience appears in the agriculture section of this year's Annual Review of Project Performance Audit Results. Borrower's Role 9. As noted in paragraph 19 below, an increasing number of borrowers are now contributing substantially to the preparation of PCRs and more borrower PCRs may thus be expected in the coming years. Comments on project experience continue to be made to OED by most borrowers, either to OED missions or by correspondence (82% of the PPARs issued in FY83 reflected such comments; some comments were received too late for inclusion). Country visits by OED staff were made to 58% of the projects for which PPARs were issued in FY83; this compared with 43% for the PPARs issued in FY82 under the then existing audit system. Borrowers' written comments, when received, continue to be reproduced in full in PPARs. 1o. An undertaking was given at the meeting of the Executive Directors which considered the Report on Operations Evaluation for FY81 to present annually thenceforth a report on the status of bor- rower involvement in the completion reporting process in respect of each borrower with a substantial number of completed Bank/IDA fi- nanced projects coming up for review. Status reports were included in the Reports for the last two years and the third is presented in - 5 - Annexes A and B of this Report. These reports document the in- creasing participation of some borrowers in project completion re- porting. They also reveal that arrangements for central review and feedback of project experience are still uncommon in member govern- ments and in the Bank's operational dialogue with members. III. Monitoring and Evaluation in Projects 11. Monitoring and evaluation (M&E) elements continue to be included in Bank projects in certain sectors. There has been a significant increase in the number of projects with built-in M&E in the agricultural and rural development sector, the percentage of projects including an M&E component having risen from 60% of those approved in FY74 to all projects save one approved in FY83. Similarly, the percentage of projects including a costed M&E com- ponent has risen from 35% in FY81 to 84% in FY83. 12. An internal survey of Bank experience with M&E in this sector from FY78 to FY80 concluded that there are significant differences among sub-sectors in the effectiveness of M&E. It was found to be more effective, for example, in irrigation and forestry than in credit projects. The trend, however, was found to be positive. The percentage of M&E systems which provide timely and useful information for management decision rose from 67% in FY75 to 81% in FY80. 13. Meanwhile, the Bank has continued its efforts to improve the effectiveness of M&E in this sector. Thus the Handbook for Monitoring and Evaluation of Agriculture and Rural Development, referred to in last year's report, has been published (with French and Spanish editions) by Johns Hopkins Press and the Bank has pub- lished a Philippine Case Study on the subject. Other studies have been prepared by OPS with a view to the further dissemination of information on M&E in this sector. In addition, a Regional Work- shop on Monitoring and Evaluation in Agriculture and Rural Development Projects for francophone countries in West Africa was held in March 1983 in Douala, Cameroon. The Workshop was attended by twenty-six participants from nine countries. They included M&E officers, project managers and central government agricultural statistics personnel. Observers from multilateral and bilateral agencies as well as Bank staff were also present. The Workshop report and the case studies presented by the participants have been distributed within the Bank by AGR. Similar workshops were held for East Africa and East Asia in 1979. 14. Staff of the Bank's Agriculture and Rural Development De- partment (AGR) have participated actively in field appraisal and supervision missions to help design and monitor built-in M&E. They contributed notably to the innovative design of M&E systems for the - 6 - Kenyan National Extension Project and the Thai National Rural Development Project, for which documentation is now available for Bank staff. This is in line with the increased attention being given to M&E at the national level, as against M&E which is project specific. 15. Among the other initiatives being undertaken within AGR's Monitoring and Evaluation Unit are the compilation of Country Pro- files which seek to identify methodological and institutional issues requiring attention, a further Case Study along the lines of that on the Philippines referred to above, and a survey on the application of computers to analyze M&E data. 16. In the urban sector, responsibility for the monitoring and evaluation of urban projects belongs to the Urban Development Department (URB). The joint effort undertaken by URB and the International Development Research Center (IDRC) in Canada, which was referred to in earlier Annual Reports, was completed this past year with the publication by the Bank of three Staff Working Papers on the principal findings of the evaluations and reports on the projects in El Salvador and Zambia,l/ and the publication by IDRC of an annotated bibliography of all research reports produced as part of the four-country study (El Salvador, Zambia, Senegal, and the Philippines).2! Assistance was provided by URB to ongoing evaluations in Kenya, Bolivia and Colombia, and the Department is participating in the design of new evaluations in a number of other countries. Work is under way on a set of evaluation manuals which will be published during FY84. 17. In addition, a ten-year URB review of urban operations, Learning by Doing: World Bank Lending for Urban Development, 1972-82, was published by the Bank in March 1983. 18. In the education sector, OPS evaluation work is directed toward (i) improving the quality and usefulness of evaluation com- ponents in education projects; (ii) increasing the capacity of Bor- rowers to gather and interpret information for improving project 1/ D. Keare and S. Parris "Evaluation of Shelter Programs for the Urban Poor: Principal Findings." Bank Staff Working Paper No. 547. M. Bamberger, B. Sanyal and N. Valverde "Evaluation of Sites and Services Projects: the Experience from Lusaka, Zambia." Bank Staff Working Paper No. 548. M. Bamberger, E. Gonzales-Polio and U. Sae-Hau "Evaluation of Sites and Services Projects: the Evidence from El Salvador." Bank Staff Working Paper No. 549. 2/ International Development Research Centre "Low-Income Urban Shelter Projects: An Annotated Bibliography of Research Funded by IDRC-IBRD." 1982 Ottawa. - 7 - implementation and design; and (iii) assessing the effectiveness of programs and institutions financed by education projects. The per- centage of projects including evaluation components remains at about 90%. Present OPS efforts are focused on improving the design of these components, by consulting with regional staff and con- ducting workshops. OPS staff are also assisting in the design and supervision of project components that strengthen Borrower monitoring and evaluation capacity. A series of case studies of the design and implementation of evaluations in Bank-financed edu- cation projects are in preparation. The first study examines a large-scale evaluation in the Pakistan Primary Education Project. Several evaluation activities are being conducted under the auspices of the Education Department's Research Unit. These in- clude the DiSCus Study, which is assessing the impact and effectiveness of diversification of secondary education in Bank- financed projects in Colombia and Tanzania. The study will be completed in FY84. A new study of the impact of lending for technical and vocational schools is in the planning stage. IV. Post-evaluation in Member Governments 19. The extent of borrower participation in the performance audit process has progressed in certain countries, as evidenced by the status reports on Borrowers' involvement in post-evaluation in Annexes A and B. In the Western Africa region, for example, seven countries are expected to play some part in the preparation of PCRs in the coming year, as compared with only three in FY81, the first year in which the Annual Report contained Annexes with a status re- port on this subject. The degree of participation varies consider- ably from country to country and ranges from the preparation of a PCR to merely providing some input thereto. In some countries, of course, there is very substantial participation, as in Korea where all five PCRs due this year will be prepared by the project authorities as compared with four out of the eight due in FY81; and in the Philippines, eight of the nine PCRs scheduled will be similarly prepared. In others, the main burden remains with the Bank. Most borrowers still appear not to be concerned with esta- blishing some systematic process to inform themselves of the effectiveness and the lessons of their major projects. V. The Operations Evaluation Department 20. Last year's Annual Report on Operations Evaluation (R82-269) was issued on September 10, 1982, and the Eighth Annual Review of Project Performance Audit Results (R82-270) on September 13, 1982. The Annual Review, with only the names of countries and projects removed, was issued for the general public - 8 - in December 1982 following the precedent set in respect of the Re- views issued in 1977 and subsequent years. This year's Annual Review will be issued shortly. 21. The findings of the project performance audits on which reports were issued by OED through June 30, 1983 have been classi- fied and entered into OED's computerized memory and retrieval sys- tem, which has now been in place for more than four years. The first computerized Concordance to Project Performance Audit Reports was issued on August 31, 1978 (SecM78-699). Updated versions of the Concordance covering project audits and other reports of OED have been issued periodically, most recently on October 7, 1982 (SecM82-848) and containing the findings of 780 project audits. A further revised and updated version of the Concordance is expected to be published shortly. A work plan has been developed in collaboration with the Administrative Services and Information Re- sources Management Departments for the implementation of a scheme to expand and improve the computerized memory and retrieval system, for which funds have been made available in the current fiscal year. The objective is to make the large and ever increasing num- ber of findings and other data from PPARs more readily accessible to Bank operating and research staff. Part of the system's im- provement entails restructuring the current classification system of the Concordance to better reflect the project cycle and special sectoral aspects. In response to a recent OED solicitation, ex- tensive comments have been received from operating staff for re- vision of the classification scheme. The new system will be open-ended, i.e., it will be easily expanded, contracted or restructured in line with emerging needs. 22. During FY83, OED issued seven special studies, among which were four Impact Evaluation Reports in the "second look" series: one on the First Education Project in the Philippines (SecM83-499) and three in the agriculture sector, comprising two irrigation projects--Colombia Atlantico and Indonesia Rentang (SecM82-662 and SecM83-709 respectively) and the second rural credit project in the Philippines (SecM83-708). In addition, the following studies were published: a Review of Bank Technical Assistance to Bangladesh (SecM83-9/1); Compliance with Loan Covenants and the Supplement: Selected Case Studies (SecM83-782); and a Review of the Water Supply and Waste Disposal Program in Tunisia (SecM82-881). All the foregoing studies were reviewed by the JAC and that on Compliance with Loan Covenants by the full Board. 23. Reports on Agricultural Research and Extension: the World Bank Experience; Experience of Institutional Development in Sub-Saharan Africa; and the Fourth Review of Built-in-Project Monitoring and Evaluation are at an advanced stage of preparation and are expected to be distributed shortly. In addition, work is - 9 - under way on the third of a series of Country Sector Operations re- views in agriculture (Malawi/Upper Volta), on reviews of Small- holder Livestock Development, and the Bank experience with fisheries projects. It is also planned to initiate studies of ex- perience with Bank lending for higher technical education, with sector policy and institutional implications of lending for trans- portation in two African countries, and to complete seven further Impact Evaluation Studies in the current fiscal year. An evalu- ation of Bank-supported operations and the surrounding policy dialogue with a major borrowing country is also planned. 24. The OED work program and budget for FY84 contains (in Annex C) an indicative compilation of topics on which choices about themes and subjects for future studies are to be based. This three-year rolling program of special studies has been drawn up in consultation with other departments in the Bank with a view to integrating better OED's work planning with the Bank's related operational policy and research planning. One outcome of this new inter-departmental dialogue is the plan for the Development Re- search Department's study of comparative research on agriculture and related policies in six African countries to draw on OED's work in that area. OED's rolling program of studies will continue to be subject to JAC review and approval at the time of considering OED's work program and budget for each fiscal year. 25. The OED budget for FY84 was formulated on the basis of an increase of two professional staff members. (In addition, pro- vision has been made in the capital budget for the improvement of the computerized memory and retrieval system referred to in paragraph 21 above.) Accordingly, OED's authorized staff now consists of 30 professionals, 6 research assistants and 19 support staff. Previous Annual Reports have noted the difficulties in the rotation of staff in and out of OED. In the past year, the situ- ation has improved somewhat. In the 13 months through mid-August 1983, six operational staff have transferred to OED and three OED staff have been reabsorbed in operational units. But three va- cancies currently remain to be filled. To protect the Bank's interest in OED being staffed to carry out its agreed work, new administrative arrangements for the reassignment of qualified OED staff and the filling of OED vacancies were recently put in place. They should sustain more effectively the policy of staffing OED preferentially through internal limited-term reassignment. This policy was first approved by the Board in 1976, and was most recently re-endorsed by the JAC at its meeting on July 6, 1983. The new arrangements are expected to facilitate transfers of quality staff in and out of the Department in the future. 26. The reorganization of OED on a divisional basis in December 1982 (one of the recommendations of a study made by the Organization Planning Department [OPD]) has contributed to greater efficiency and improved distribution of the workload. The addition of a programming and budgeting officer to OED's staff, which was - 10 - also recommended by OPD, has been deferred because of overall Bank budgetary constraints. 27. Staff of member governments continue to visit OED. In FY83, officials from five countries visited OED for periods generally ranging from one to two weeks, although in certain cases for a considerably longer time, on programs which exposed them to the project cycle as experienced in the Regions, OPS and OED. In addition, liaison with other development agencies continues as re- gards the evaluation process, particularly with bilateral aid agencies, the European Economic Community, the OPEC and Kuwait Funds, and the Regional Banks. At the request of the President of the West African Development Bank, OED is conducting a seminar on evaluation for the Bank's staff in Togo in September 1983. A similar seminar was conducted for the staff of Yugoslavia's Social Accounting Agency in Yugoslavia in September last year. VI. Other Evaluation Activities 28. As has been the practice in previous Annual Reports, note is made below of the evaluation activities of IFC, EDI and the Re- search complex, which OED keeps informed about but does not evaluate. IFC 29. During the year IFC prepared, and sent to OED, completion reports on 18 projects covering textiles, sugar, cement, fertilizer and other industries. These projects were located in 14 countries, six of them in Africa and five in Central and South America. The completion reports are not reviewed by OED, but are filed in the DGO's office for reference by interested Executive Directors. The IFC Management considers that these completion reports have enabled them not only to have a better understanding of implementation problems, but also have provided an opportunity to re-assess the prospects of the projects and to use the experience as feedback in improving appraisal of new investment proposals ("IFC's Five-Year Program FY84-88", IFC/R82-96, October 29, 1982, p. 59). 30. IFC completed during the year and sent to OED an evalu- ation report (in two volumes) on its cement operations. This report is based on 9 completion reports of 10 projects, prepared earlier by IFC, and represents the first time IFC has used its completion reports as a basis in evaluating its portfolio experi- ence. As regards the completed projects, the report provides an update on the projects since completion, calculations showing, - 11 - inter alia, the economic and financial returns of the projects, and derives the lessons on project design which flow from this experi- ence. In addition, the report examines in depth three issues: factors affecting capital cost of cement projects, impact of state intervention such as price controls, and IFC's developmental role. The report puts the IFC role in the perspective of worldwide trends in the industry and brings out the particular adverse impact of in- appropriate pricing policies relating to energy and cement on the growth of the industry. It finds that, despite the difficult economic environment of the recent years, overall the projects show a good economic, technical and financial performance. The evalu- ation report provides an example of how a number of completion reports can be analyzed as a group both to evaluate performance in a sector and to draw general lessons on economic policy and development. IFC is now discussing with OED its further evaluation activities and its arrangements for disseminating evaluation findings to the Corporation's operational staff. Research 31. The Executive Directors discussed the most recent report on the World Bank research program (Report No. 4328, Parts I and II, dated March 1983) at the Board meeting on April 7, 1983. In addition to describing each research project and research prepa- ration in the Bankwide portfolio, along with its published output, the report summarizes the evaluations of nine research projects that were evaluated in FY83, bringing to 129 the total number of projects evaluated to date. OED continues to be represented on the internal evaluation panels which evaluate completed projects. Evaluations of completed research projects are also filed in the DGO's office for reference by interested Executive Directors. Economic Development Institute 32. EDI has continued its practice of evaluating each of its training activities at its conclusion, at which time participants are asked their opinion in writing and orally about the training activity contents, relevance to their work, the quality of the teaching and the teaching materials distributed, together with suggestions for improving future training activities. EDI has started work on a system of sample tracer studies to follow up its training efforts and evaluate the extent to which participants in its training activities have been able to apply what they have learned. 33. Under the new system each participant prepares an action plan at the end of the seminar, listing what changes he plans to make back on the job as a result of the EDI training. After six months the EDI follows up by a mail questionnaire to see what changes have been implemented on the job. The system has been tested on five EDI courses with encouraging results. The system - 12 - also has attracted the attention of EDI lecturers as a method of encouraging the transfer of training. 34. During 1982, a questionnaire was mailed to all partici- pants of EDI rural credit courses, providing useful feedback on how to improve future courses. Arrangements were also made to obtain feedback from all participants in FY82 Washington courses by a mail questionnaire. 35. The evaluation report on EDI's Philippines training ac- tivities, which was mentioned in last year's report, has been completed and will be disseminated shortly. VII. Follow-Up 36. Reference has been made in paragraph 7 to the in- formal survey by OED to ascertain the extent to which OED reports are read by operational staff. In addition, PPARs (all of which include PCRs) and PCRs which are read but the projects covered thereby are not audited by OED are reviewed with operational staff in some sectors at meetings arranged by OPS and attended by repre- sentatives of the Regions, OPS and OED. 37. Annex C of this Report contains--as have similar Annexes since the 1978 Annual Report--the main recommendations of OED special studies distributed in the course of the fiscal year and management's reactions to them. In addition, the Annex keeps track of recommendations contained in studies published in earlier years where further reactions to them are expected from management. - 13 - ANNEX A Page 1 of 6 STATUS REPORT OF THE REGIONS ON BORROWER GOVERNMENTS' INVOLVEMENT IN THE POST-EVALUATION OF BANK/IDA-FINANCED PROJECTS (All countries with five or more PCRs scheduled through CY84) A. Western Africa (1) Cameroon - Six PCRs in two sectors (five in agriculture and one in technical assistance) are planned through CY84. Project agencies will prepare draft PCRs for the agriculture projects. In case of the technical assistance operation, Government has demonstrated considerable inter- est in completion reporting but this has not translated into any real contribution because of shortage of staff and budget resources for this work. (2) Ghana - Five PCRs in three sectors (two each in agricul- ture and power and one in highways) are expected to be issued through December 1984. The project implementation agencies for the agriculture and power projects will pre- pare preliminary completion reports. The highway agency is expected to furnish basic cost and implementation data for the PCR for Highways III. (3) Ivory Coast - Five PCRs in four sectors (two in agricul- ture and one each in transport, water and urban) are expected to be issued in CY84. The respective agencies (SOGB and SAPH) for the two rubber projects are expected to prepare completion reports. The project units for the urban and water projects will prepare basic data and back- ground information for the PCR while, in case of the feeder roads and highway maintenance project, we do not expect significant borrower involvement in PCR prepara- tion. Although there is no clearly defined process of reviewing the experience of major Bank projects within the Government, sectoral ministries as well as ministries of planning and finance normally review and forward comments on Bank-prepared PCRs and PPARs. (4) Nigeria - Six PCRs in three sectors (three in agriculture, and one each in power and IDF sectors) are expected to be prepared in CY84. While we do not expect significant bor- rower involvement in the PCR for an oil palm project, the completion report for a livestock project will be prepared by the project implementation unit with assistance from - 14 - ANNEX A Page 2 of 6 Bank supervision missions. In case of two rural develop- ment projects, the implementation agencies, assisted by the project evaluation unit under the ongoing Agricultural Technical Assistance Project, will prepare the project completion reports. The PCR for the Power IV project will be prepared by consultants financed under the Sixth Power Project. The PCR for the IDF project will be prepared by Bank staff on the basis of information provided by the Nigerian Industrial Development Bank. At present there is no central point in the Government for processing PCRs. (5) Senegal - Eight PCRs (three in transport, two in agricul- ture, one in education, and two for sectoral adjustment and technical assistance operations) are scheduled to be prepared in CY84. Preliminary completion reports will be prepared by the project management units for the two agri- culture projects and respective project entities will pre- pare basic completion data for the transport and education projects. We expect little direct Government contribution to the PCR for the technical assistance project, although ongoing technical assistance projects will help to estab- lish project monitoring and evaluation capabilities within the Ministry of Planning. The PCR for the structural adjustment loan will be prepared on the basis of progress reports prepared by ministries of finance, planning and rural development. B. East Asia and Pacific (1) Indonesia - Preparation of eight PCRs in four sectors (agriculture, education, transport and nutrition) is sche- duled through FY84. Project agencies are expected to pre- pare six of these eight PCRs. The degree of involvement by the borrower in the pre- paration of PCRs has varied considerably among projects depending primarily on the capacity of the implementing agency. In general, the burden of preparing PCRs in draft (rather than only assembling data) has shifted increas- ingly to the borrower as the capacity of these agencies has improved. For example, while the contribution will continue to vary from agency to agency the initial exper- ience with some of them in the agriculture sector has been encouraging. Such involvement has been common in the power sector for some time. There has not been, however, any substantial involvement by the borrower in the review of PCRs. There is no established procedure for review by a central agency such as the National Planning Agency, and the feasibility or desirability of establishing such a procedure has not yet been discussed with the Government. - 15 - ANNEX A Page 3 of 6 (2) Korea - Five PCRs (three in the transport sector and one each in the education and industry sectors) are scheduled for completion during FY84. Project agencies will be res- ponsible for preparation of all of the PCRs. The Government of Korea is developing a system for post-evaluation of major projects. During a Government reorganization in the fall of 1981, this function was assigned to a newly created Bureau of Post Evaluation and Coordination in the Economic Planning Board. This new Bureau has developed an initial work program which focuses on the evaluation of on-going projects. In March 1983, the Bureau completed a manual for post-evaluation proce- dures which has been distributed with "guidelines for 1983" to the Ministries and Agencies of the Government who implement projects. The manual is expected to enable them to prepare their own manuals of sectoral procedures. The Bureau has also been arranging training in post-evaluation theory and techniques for its staff including visits to the Bank's Operations Evaluation Department. (3) Malaysia - Seven PCRs in five sector (agriculture, trans- port, energy, urban and population) are expected to be completed during FY84. Four of these seven PCRs will be prepared by project agencies. The Government of Malaysia had discussed a few years ago the possibility of broadening the responsibilities of the Implementation Coordination Unit (ICU) in the Prime Minister's office to include the carrying out of post-completion evaluations of development projects in the country. However, this has not been feasible so far as the greatly expanded development program has strained ICU's capacity to perform even its normal tasks of moni- toring project performance. The prospects of the Govern- ment assigning post completion evaluation of projects to one of its other agencies, such as the Economic Planning Unit, in the foreseeable future are not particularly encouraging because of the pervasive staff shortages besetting these agencies. The Bank is therefore continuing to concentrate its efforts in strengthening the capabilities of the imple- menting agencies to prepare PCRs of high quality. (4) Philippines - Preparation of nine PCRs in six sectors (agriculture, education, energy, urban, water supply and population) is expected in FY84. Project agencies are expected to prepare eight of these nine PCRs. - 16 - ANNEX A Page 4 of 6 The Philippine Government has directed that all PCRs prepared by the Executing Agencies should be sent to the National Economic and Development Authority (NEDA) which reports to a Board that is a subcommittee of the Cabinet. NEDA will review the PCRs and draw therefrom lessons that are of wider relevance. Review procedures and standards have been established and the Government intends to seek OED's assistance in strengthening the post project perfor- mance review system. We expect that each forthcoming PCR would include NEDA's comments on the report. (5) Thailand - There are five PCRs in three sectors (agricul- ture, education and energy) scheduled for completion in FY84. Two of these five PCRs are expected to be prepared by project agencies. The Government of Thailand continues to play an increasing role in the preparation of PCRs and the imple- menting agencies now provide much of the input for those prepared by Headquarters staff. The Government's ability to assess the impact of development programs is being strengthened under its structural adjustment program whereby performance auditing as well as improved mechan- isms for monitoring and evaluating development projects are being introduced. C. South Asia (1) Bangladesh - Seven PCRs in four sectors (agriculture, irrigation, education, and industry) are scheduled for completion in 1984. Data needed for the preparation of PCR is collected by various agencies. Coordination within member government is generally the responsibility of the Project Authority, e.g., Bangladesh Agricultural Develop- ment Corporation (BADC), and the Ministry of Agriculture. However, in two cases, the External Resources Division (ERD) will provide the coordination. Heavy assistance from the Bank is required to complete draft. Three of the PCRs are being fully prepared by the Bank/IDA, and one by FAO/CP. (2) India - Fourteen PCRs in six sectors (agriculture, irriga- tion, education, industry, power, and urban) are scheduled for completion in 1984. Initial data collection is done by various agencies such as the Industrial Development Bank of India (IDBI), the State Department concerned, and the Indian Institute of Agricultural Research (IIAR). The Project Authority is responsible for coordination of PCRs - 17 - ANNEX A Page 5 of 6 for five projects, the Department of Economic Affairs (DA) for six projects, the Forest Department, the Indian Insti- tute of Agricultural Research, Ministry of Finance, and the Madras Metropolitan Development Authority for one pro- ject each. Two of PCRs are expected to be completely pre- pared by the Bank, four more projects are taking heavy Bank staff inputs. However, eight are indicated to require nominal or reasonable degree of Bank assistance. Quality of data collection for a majority of projects is good; only four projects are rated poor. (3) Pakistan - Six PCRs in three sectors (agriculture, power and transportation) are scheduled for completion in 1984. Data collection is generally done by the Project Authority like the Water and Power Development Authority (WAPDA), Pakistan Industrial Credit and Investment Corporation (PICIC), and the Ministry of Food, Agriculture and Cooper- atives (MFAC). In one case the initial data collection was done by Bank staff. Coordination of the PCRs is not done by a central agency; rather the preparing agency is responsible for all coordination. Heavy degree of Bank assistance is being required to complete three PCR drafts. For three others the needed Bank assistance is considered reasonable. D. Europe, Middle East and North Africa (1) Yugoslavia - Seven PCRs are scheduled for completion in four sectors (highways, railways, water supply and multi- purpose). The borrower prepared data and was helpful in collecting other required information for PCRs in high- ways, railways and water and air pollution projects. In the multipurpose project the borrower -- an enterprise -- is expected to prepare the PCR after the physical comple- tion of the project in June 1984. In general, in the agriculture sector, the borrower/project entity has pre- pared the PCR which usually has been acceptable. E. Latin America and the Caribbean (1) Brazil - Eight PCRs are forecast for 1983-84. In all cases major contributions are expected to be made by the agencies in the sectors concerned (Agricultural Research, Irrigation, Agro-Industry, Water Supply, Power, Education, Railways and Highways.) In some instances the PCRs pre- pared by the agencies are likely to be so comprehensive that only relatively limited complementary work by Bank staff will be required. - 18 - ANNEX A Page 6 ot 6 Regional staff has continued to discuss with the Federal Government's Planning Institute the desirability of establishing a project evaluation function within the Institute but no action has been taken thus far. (2) Jamaica - Five PCRs are forecast for 1983-84. Only limit- ed assistance by the agencies (primarily in the form of data collection) is likely. (3) Mexico - Seven PCRs are forecast for 1983-84. The execut- ing agencies' role in helping in the preparation of the PCRs by Bank staff is expected to take the form of draft PCRs which will furnish a significant basis for the Bank's work. The evaluation units which have been organized in the major external borrowing intermediary institutions con- tinue to assist the executing agencies in ex post evalua- tion of projects and are strengthening their in-house capacity to prepare PCRs. - 19 - ANNEX B Page 1 of 4 STATUS REPORT OF THE REGIONS ON BORROWER GOVERNMENTS' INVOLVEMENT IN THE POST-EVALUATION OF BANK/IDA-FINANCED PROJECTS (Selected countries with less than five PCRs scheduled through CY84) A. East Africa (1) Comoros - One PCR in the transport sector is nearly com- pleted. A Government consultant prepared a report which was used as background information in the Bank's PCR. (2) Madagascar - Four PCRs in four sectors (energy, agricul- ture, transport and education) are planned through CY84. While the borrower is expected to prepare a PCR in accor- dance with the legal agreement, in actuality, the PCR is prepared by Bank staff on the basis of data sent to the Bank by project authorities. The low rate of borrower participation is due mainly to unavailability of qualified staff and lack of familiarity with PCR procedures. (3) Mauritius - Two PCRs in two sectors (transport and indus- try) are forecast through CY84. The Bank does not expect significant borrower involvement in preparation of PCRs. B. West Africa (1) Liberia - Four PCRs in four sectors (agriculture, educa- tion, highways and water) are planned through CY84. The Planning and Programming Bureau of the Ministry of Public Works (MPW), will prepare a draft PCR for the Fourth High- way Project, similar to the PCR prepared for the Third Highway, which was of good quality. A central Planning, Monitoring and Evaluation Unit is being established under Lofa ADPII and would handle preparation of the draft PCR for the Bong ADP as well as PCRs for future agricultural projects. Government's participation in the PCR for the Third Education Project will be limited to the preparation of basic data and some background papers on key project elements. The Liberia Water Supply Corporation is expect- ed to prepare a preliminary completion report for the Water project. While Ministry of Planning has an overall coordination role for Bank-financed projects, there is no central point for the processing of project completion reports. (2) Upper Volta - Four PCRs in three sectors (two in agricul- ture and one each in transport and urban sectors) are planned for CY84. The project implementation units for - 20 - ANNEX B Page 2 of 4 the agriculture projects will prepare draft PCRs. Govern- ment involvement in the completion process for railway and urban projects will be confined to provision of completion data. C. Europe, Middle East and North Africa (1) Egypt - Four PCRs in two sectors (railway and industries) are scheduled for calendar year 1984. In railway projects the borrower was not committed by the Loan Agreement to assist in the preparation of PCR but did provide a considerable amount of information requested. For indus- trial projects the Industrial Development Bank of Egypt prepared a detailed review of its own financial status, as well as an ex-post situation report on the subprojects financed under the projects. The contribution was well prepared, accurate and on schedule. (2) Algeria - Two PCRs in the power sector are scheduled. Borrower willingly prepared draft PCRs, but these were of limited use in that they dealt only with the contracts let under the loans. They did not address institutional matters, the financial performance of the borrower, or borrower-Bank relationships. (3) Tunisia - Three PCRs in three sectors (urban, power and sewerage) are scheduled. An Urban Planning and Public Transport, Staff of Tunis Municipality's Public Works Department, the Tunis District and the Soci6t& R6gionale de Transports (SNT) helped Bank staff to collect informa- tion and to assess project performance. SNT also deliver- ed to the Bank a consultant's report assessing the impact of the project on SNT and evaluating the company's ser- vice, technical and financial performance. In the sewer- age project, the borrower prepared detailed data to facilitate preparation of the PCR. In the power project, the borrower, Soci6tg Tunisienne de 1'Electricit6 et du Gaz (STEG) prepared the PCR, which was very good, and was most cooperative in its total production. Bank staff have had discussions with the Director General of Projects of the Ministry of Planning and Finance with a view to developing further cooperation in the supervision and evaluation of projects. (4) Israel - Two PCRs in two sectors (DFC and Highway) sche- duled. In DFC Training Component, the borrower provided substantial input which was timely. For highway PCR the borrower provided data and information together with analysis of the effects of currency devaluation and infla- tion on project costs. (5) Romania - Two PCRs in the agriculture sector are schedul- eTAn acceptable PCR was prepared by the borrower (BAFI) in accordance with guidelines provided by the Bank. The - 21 - ANNEX B Page 3 of 4 next PCR is being prepared by BAFI in conjunction with the implementation organization (DGEIFA) and is expected to be completed in September 1983. This agency has already pre- pared a similar report for a previous project, which was acceptable. BAFI is the borrower for all agricultural projects in Romania, and we expect that it will continue to prepare PCRs for ongoing projects. (6) Turkey - Two PCRs in two sectors (railway and urban) are scheduled. In railway project, the borrower in spite of not being committed, provided considerable amount of information for PCR. In urban sector although the PCR was prepared by the Bank staff, during the mission government officials from the Ministry of Reconstruction and Reset- tlement, the State Planning Organization and the Istanbul Master Planning Bureau cooperated fully with Bank staff in collecting information and assessing project implementa- tion, performance and results. (7) Ireland - One PCR in the education sector was produced by the project unit on schedule and required only minor inputs from the Bank. (8) Spain - One PCR in port sector is scheduled. Although the borrower was not committed to prepare a PCR by Loan Agree- ment, they made a very large effort both in assisting in the preparation of the PCR and in providing information. (9) Morocco - One PCR in water supply sector scheduled. The borrower prepared detailed project data to facilitate pre- paration of the PCR. (10) Jordan - For a PCR in education sector, a representative of the borrower participated in the completion mission and the Project Unit in Amman produced the requisite document- ation. (11) Oman - For a transport component of a technical assistance PCR the borrower provided considerable data and other necessary information. D. Latin America and the Caribbean (1) Bolivia - Three PCRs are forecast for 1983-84. The executing agencies in Aviation, Railroads and Mining are expected to make significant contributions in the form of draft PCRs. (2) Colombia - Four PCRs are forecast for 1983-84. Except for Cerro Matoso, where the borrower will prepare the report, these would be prepared by the Bank, with input from the executing agencies. - 22 - ANNEX B Page 4 of 4 Recently the National Planning Department (DNP) has initiated a project review scheme with the Bank's Resident Mission and will attempt to monitor projects on a regular basis. This will be followed by periodic portfolio reviews with the Bank. DNP's objective is to establish a project monitoring and evaluation system within DNP which will serve as a planning device and will aid the Govern- ment in solving problems during project implementation. The unit is hoped to be operational within 6 months to one year; it is expected that OED's advisory assistance will be requested. In time, such a unit could serve as a basis for project audit/completion reviews. (3) Ecuador - Four PCRs are forecast for 1933-84. The role of the executing agencies is expected to range from the pre- paration of relatively comprehensive reports in the case of agriculture to more limited participation in the case of water supply and (contrary to our expectation of a year ago) in Vocational Training. (4) Paraguay - Three PCRs are forecast for 1983-84. Except for the Livestock Project where the borrower is expected to provide a draft PCR, the participation of the executing agencies in the preparation of the PCRs will be rather limited. - 23 - ANNEX C THE CURRENT STATUS OF RECOMMENDATIONS OF OED POLICY REVIEWS AND EVALUATION STUDIES Introduction Annex C-I attached hereto provides information on the status of the recommendations contained in the Interim Report on Procurement Issues in Bank-Financed Projects dated July 15, 1981, which updates Annex D of the 1982 Report on Operations Evaluation. Annex C-II to C-IV contain the main observations of Reviews of Bank Technical Assistance to Bangladesh dated December 23, 1982; the Water Supply and Waste Disposal Program in Tunisia dated October 20, 1982; and Compliance with Loan Covenants dated September 1, 1982. The sequence of the recommendations set forth in the attach- ments is not necessarily as in the Reports themselves. Annex D to last year's Annual Report included an outline of certain points which had emerged as of recurring concern in the Annual Reviews of Project Performance Audit Results. This year they are repro- duced in Volume II, Annex III of the Ninth Annual Review which will be distributed shortly. These points continue to be tracked through the audit process. Conclusions drawn from the PPARs (which include PCRs) and from PCRs read but not audited which were distributed in CY82 will also be contained in the Ninth Annual Review. - 24 - ANNEX C INDEX Part Report Page No. I. An Interim Report on Procurement Issues in Bank-Financed Projects 1 II. Review of Bank Technical Assistance to Bangladesh 5 III. The Water Supply and Waste Disposal Program in Tunisia 10 IV. Compliance with Loan Covenants 13 - 25 - ANNEX C Page 1 of 19 I. AN INTERIM REPORT ON PROCUREMENT ISSUES IN BANK-FINANCED PROJECTS (SecM81-636, September 2, 1981) The OED Interim Report on Procurement Issues in Bank-Financed Projects (Report #3557) dated July 15, 1981 was circulated on September 2, 1981 (SecM81-636) and was considered by the Joint Audit Committee on September 9, 1981 (JAC/M81-9). Subsequently, the OED Report, together with the Staff Comments thereon (SecM81-1037 dated December 31, 1981) was the subject of two seminars for Executive Directors, on January 22 and January 29, 1982. Annex D to the last Annual Report set forth a summary of the principal recommendations contained in the Report together with the Management's response. The following notes those recommendations where action remained to be taken by Management. A. Advanced Planning and Local Procedures Many procurement problems can be traced to insuffi- cient preparation in the early stages of the project cycle and therefore: (i) a procurement checklist should be provided to assist Bank staff in appraisal; (ii) special country reviews should be made of local procurement procedures jointly with Government and Bank staff; and (iii) increased training should be given to borrower staff. The Bank agrees with all the above and specifically considers a checklist a useful tool, particularly for inexperienced project officers. Some divisions have prepared checklists and the Central Procurement Unit in Projects Advisory Staff (PAS) has prepared a checklist of procurement matters to be considered during appraisal. - 26 - ANNEX C Page 2 of 19 In addition, the Bank agrees that special country reviews should be made. With Government's support, the potential benefits from country reviews are high and Bank staff are prepared to assist in such cases. Two such country reviews were conducted in FY83. Training seminars help to reduce problems and disputes and to expedite project procurement and implementation. About a dozen country seminars have been undertaken so far. Present staff resources permit three or four additional borrower seminars a year; four were held in FY83. Consid- eration has been given to holding regional seminars, with a sub-regional seminar having been held for three countries in Eastern Africa (Tanzania, Uganda and Zimbabwe) in FY83. B. Organization and Training (i) The Bank should continue its efforts to train projects officers in procure- ment; (ii) The Bank should provide more guidance on procurement to the project staff such as by appointing more specialists in procurement; and - 27 - ANNEX C Page 3 of 19 (iii) The location of additional procurement spe- cialists requires consideration. While placing them in a central procurement unit working across regional boundaries either in an advi- sory or line function has the advantage of economy, an alternative would be to have, in addition to a central procurement unit, one or more procurement specialists assigned to the office of each Regional Projects Director to be more readily available and bring to the work the knowledge of local conditions and procedures of the countries within the Region. The Bank agrees that training is an important aspect of the work program of the Central Procurement Unit in PAS. Bank procurement seminars comprising one week of half-day sessions were started in 1975 and 40 seminars have been conducted for some 20 to 25 participants per seminar. An advanced seminar has been begun recently as well. During FY83 six basic seminars were held for about 135 partici- pants and in addition two advanced seminars for about 50 staff. The Bank also agrees that procurement specialists should be available in Projects Departments, and that PAS should complement their work. An additional position of procurement advisor has been authorized for the Central Procurement Unit in FY84. - 28 - ANNEX C Page 4 of 19 C. Forms and Procedures (i) The Bank should develop a model or standard set of tender documents to provide guidance to borrowers and Bank staff. The Bank agrees, but notes that the differences in legal frameworks that exist among borrowers and the variety of goods and works to be procured make it very difficult to provide a single standard set of tender documents. However, the absence of a specimen or sample document to guide the borrower staff, as well as inexpe- rienced Bank staff, has been increasingly noted. Parti- cularly where borrowers do not already have standard documents that have been used for international bidding, a good sample could reduce the time and costs involved for borrowers in the employment of consultants to prepare documents or review documents, the considerable work in Bank review of documents, and the number of unsatisfac- tory bid documents. Accordingly, the Bank has prepared such a sample document for the purchase of goods and equipment, and expects to have a similar one ready for construction contracts by the end of CY83. - 29 - ANNEX C Page 5 of 19 II. OPERATIONAL POLICY REVIEW: REVIEW OF BANK TECHNICAL ASSISTANCE TO BANGLADESH (SecM83-9, January 4, 1983) The OED Review of Bank Technical Assistance to Bangladesh (Report #4231) dated December 23, 1982 was circulated to the Executive Directors on January 4, 1983 (SecM83-9) and was considered by the Joint Audit Committee on March 2, 1983 (JAC/M83-3). This Review presents the first attempt by OED to assess the efficiency, effectiveness and impact of Bank Technical Assistance (TA) to Bangladesh and identifies some of the more important constraints to its effective use. The Review suggests necessary conditions for effective technical assistance, and certain recommendations of which many may have more general relevance to Bank- financed technical assistance. The following is a summary of the Report's findings, together with Management's reaction thereto: A. Necessary Conditions for Effective Technical Assistance: 1. Clear agreement between the Bank and the Borrower on the objectives of the task to be undertaken; 2. An accurate and realistic definition of the task to be performed, its time frame, and intended output; 3. Recognition by Government and the recipient agency of the need for, and benefits to be derived from, the use of TA; 4. An appropriate structure within Government to design the TA component, oversee its implementa- tion and utilize its output; 5. Clear definition of the responsibilities of the TA personnel, recipient and implementation agencies; 6. Careful selection of TA personnel for technical qualifications, relevant experience and personal suitability; - 30 - ANNEX C Page 6 of 19 7. An adequate level of support from the recipient agency and, where necessary, technical back-up from the consultants' headquarters; and 8. Continuous, close supervision by the Government. B. Summary of Recommendations 1. Definition of objectives a. TA's specific objectives should be clearly defined and be consistent with the borrower's policies and development strategy; and b. Bank staff should avoid undue reliance on TA, particularly when other measures to strengthen the borrower's own capacity are more appropriate. 2. Borrower participation a. As a general guideline, the Bank should finance TA only when the borrower accepts the need for it and is committed to use its output effectively; and b. The inclusion of TA for ensuring successful project implementation, when the borrower is not fully convinced of its need, should be clearly explained in the appraisal report with an analysis of: (i) the justification for including TA in the project; (ii) the objections or reservations of the borrower; (iii) the risks associated with the TA, including the risks of the borrower not making full use of TA; and (iv) the possibility that the Bank may need to devote an unusually large amount of time for supervision and follow-up. - 31 - ANNEX C Page 7 of 19 3. Design of technical assistance components a. More attention should be given by the Bank and the borrower to the design of TA components, with special reference to: (i) realistic scope, cost and time frame of the assignment; (ii) the necessary support services and local funds to be provided by the recipient agencies; (iii) a precise definition of the responsibility of the consultants, the responsibility within the agency for supervising the work of the consultants and utilizing their out- put, and the role of Bank staff vis-a-vis both the Government and the consultants; and (iv) a precise definition of the counterpart arrangements agreed upon. b. When one or more of the conditions for the effec- tive use of TA given in (A) above cannot be assured, the feasibility of proceeding with a TA component should be carefully reviewed and alternative means for achieving the agreed objectives considered. c. Although it is the Bank policy to promote the development of local capabilities, the Review suggests that the Bank staff tend to rely heavily on foreign consultants for the successful imple- mentation of Bank-financed projects. The following are the suggested measures to maximize the tech- nical and managerial role of local agencies: (i) there is a need for a more systematic procedure for breaking down the various skills and activities involved in the technical task to be undertaken in order to allocate them to implementing agency, local consultants, and foreign consultants; - 32 - ANNEX C Page 8 of 19 (ii) when possible, the overall management of a project or TA assignment should be the direct responsibility of the appropriate agency, rather than an overseas consulting firm, and a local project manager or team leader to be appointed from the beginning; (iii) the concept of a local project team or task force under local management, supplemented by local or foreign consultants, should be considered as an alternative to using a large consulting team for the whole task; and (iv) greater use of part-time consultants or visiting agencies might reduce the costs of TA and foster greater responsibility on the part of recipient agencies. 4. Management of technical assistance a. In order to build up a good TA management, it is necessary to strengthen Government's capacity to determine its own TA needs, design TA components and ensure the effectiveness of TA. To this end, Government might consider strengthening its project proforma evaluation procedures in the following ways: (i) proposals for TA emanating from line agencies should be reviewed within the context of a broader strategy for building up their capacity. Line agencies and parent ministries should be required to outline a program for institutional devel- opment to reduce TA needs in the future; (ii) Economic Resources Division (ERD) might apply its broad experience in administering TA by playing a more active role in the design of TA components, and making maximum use of local personnel; and (iii) the ERD and other relevant agencies need to play a greater role in ensuring that recipient agencies have the capacity to absorb the proposed TA, including proper supervision of consultants. - 33 - ANNEX C Page 9 of 19 b. While the present efforts of Government to stan- dardize the terms of service of TA personnel will be useful in providing guidelines, a rigid application of standard terms may not be desirable; far greater cost savings can be obtained by ensuring that TA components are well designed, efficiently executed and effectively used than in trying to effect minor savings in the costs of hiring consultants. c. Separate guidelines, in the range of subjects covered in this report, should be prepared for Bank Operational staff to help in better prepa- ration, appraisal and supervision of TA projects and components, in addition to those available for training and free-standing TA credits. d. The Bank, when requested by the borrower, should be prepared to play a more direct role in helping to design and supervise TA contracts, particularly when the TA is required in a highly specialized field in which a recipient agency has little or no experience. e. Coordination of TA among donors is most appro- priately performed by Government. However, if donors wish to continue their own coordination and information activities, it is suggested that a single annual publication should be produced. This will require donors' agreement on a stan- dard method of presenting data on TA for aggre- gation and comparability. The Bank Management has indicated that the findings of the Review have general relevance to the Bank's tech- nical assistance work, which was also a subject of exten- sive discussion in the Project Implementation Review held in the spring of 1983. The Bank has under preparation a revision of the OMS guidelines to staff regarding tech- nical assistance. - 34 - ANNEX C Page 10 of 19 III. SECTOR OPERATIONS REVIEW: THE WATER SUPPLY AND WASTE DISPOSAL PROGRAM IN TUNISIA (SecM82-881, October 27, 1982) This report (#4146 dated October 20, 1982) presents the find- ings of the first study undertaken by OED to assess the program of operations of the World Bank Group in the water supply and waste disposal sector in a particular country. The study focuses on the Bank operations in this sector in Tunisia up to the end of FY81 and was considered by the Joint Audit Committee on January 26, 1983 (JAC/M83-2). The water supply and waste disposal sector in Tunisia was selected for review for several reasons: (i) the emphasis given by the Bank Group to the sector with direct and indirect lending to this sector exceeding 20% of total lending to Tunisia through FY81 compared with 5.5% of overall total Bank lending for the sector for the same period; (ii) Tunisia represents an example of how, through a series of lending operations, the Bank, despite a serious shortage of physical resources, achieved the goal of extending the benefits of this sector to a wider target group; (iii) Tunisia was one of the first countries in which Bank financed national water systems rather than individual systems, and (iv) the compact size of both the country and the sector combined with the number of audited projects facilitated the task of identifying the impact of Bank lending in the sector in one country. A. Conclusions The following principles which constitute the ingre- dients for a successful strategy in the water supply and waste disposal sector have been applied energetically and effectively in Tunisia: 1. The overall sector strategy has been well defined in the national plans and investment programs, the Government and sector agencies have committed themselves to the successful implementation of that strategy and close cooperation exists between the Government and these agencies; 2. New solutions in technology have been tested; 3. Efficient and independent institutions have been established, with capable staff; - 35 - ANNEX C Page 11 of 19 4. Tariffs have been carefully structured to cover operating and a fair proportion of capital costs with the burden on customers graduated as far as possible on ability to pay; and 5. Services have been provided to thousands of addi- tional people, including increasingly the poorest segments of society. B. Main Lessons The essential ingredients which provided the founda- tion for the Bank's general strategy in Tunisia and which were crucial for its eventual success have relevance to other countries. The fact that there have been few examples of the Bank's involvement in the water supply and waste disposal sector as successful as the Tunisia experience demonstrates that the ingredients of success are not always present. The principal lessons to be learned from the study are: 1. The essential ingredients which lay behind the Tunisia success should be disseminated widely to ensure their presence as far as possible in similar future lending operations. They are: a. The Government's policy objectives should be detailed and coherent; appropriate institutions to implement these objectives should be in place; technical, financial and economic implications of the policy objectives should be explicitly recognized; b. The Government commitment to these policy objectives should be expected to continue with a high degree of probability in the longer term; the degree of probability should be measured against the strength of the objective forces, such as the extent of scarcity of water resources and the impor- tance of the development of the sector for the development of the economy; and - 36 - ANNEX C Page 12 of 19 c. The Bank's program should be in harmony with the Government's sector policy objec- tives and loan covenants be accepted by the Government as necessary and achievable. 2. Where the essential ingredients for success exist, the Bank should be prepared to devote manpower to the sector even though the Bank's own financial contribution may be limited. 3. A full-scale sector study need not be an essen- tial precondition for lending in the sector in every country. A sector strategy can be developed successfully through a logical series of lending operations in cooperation with the Government. If it is appropriate in a particular country to proceed without full-scale formal sector studies, the Bank's Economic Reports should pay special attention to the main lines of a coherent sector strategy. - 37 - ANNEX C Page 13 of 19 IV. OPERATIONAL POLICY REVIEW: COMPLIANCE WITH LOAN COVENANTS (SecM82-782, September 3, 1982) The OED Report on Operational Policy Review: Compliance with Loan Covenants (Report #4090) dated September 1, 1982 was circulated to the Executive Directors on September 3, 1982 (SecM82-782). The Joint Audit Committee considered the OED Report on September 22, 1982 (JAC/ M82-8) and the Management Response thereto (R83-2 dated January 5, 1983) on January 26, 1983 (JAC/M83-2). The OED Report and the Management response were considered at the Board Meeting of July 14, 1983. The Bank Management is in broad agreement with the recommenda- tions of the OED Report as they relate to the procedures for strengthen- ing the process of preparing, negotiating and supervising legal covenants. It also considers the Bank's current system of legal documentation to be adequate in providing for a range of alternative instruments to register agreements and undertakings considered essential to attaining project objectives. The principal recommendations of the OED Report are summa- rized hereunder. The Management's comments are summarized under the recommendations. Specifically on OED's recommendation that a selective use be made of loan covenants, the Assistant General Counsel made a statement to the Board on July 14, 1983. This statement outlined the action proposed to be taken by the Bank in making a more discriminating and flexible use of legal instruments available for designing loan and credit agreements. A couple of important points from that statement appear at the end of this section. Summary of Principal Recommendations A. The need for greater flexibility in loan documents through more selective use of covenants specifically: 1. Covenants dealing with a string of "minor" actions be consolidated into a plan of action or implemen- tation schedule which may take the form of a schedule to the Agreement, a supplemental letter or a combination of both. The Bank Management agrees that such consolidation has merit in reducing the body of the loan agreement and putting related covenants together in one place. Such consolidation - 38 - ANNEX C Page 14 of 19 has been used on occasion in the past, and it is Manage- ment's intention to use it more frequently and consistent- ly in the future. 2. Project objectives or general undertakings which are targets or intentions rather than firm obli- gations might be registered in less formal and more flexible instruments such as letters of intent or memoranda of understanding, rather than in covenants. The Management believes that the range of alternative instruments to register agreement and understandings currently available is adequate. There are three major types of legal instrument currently in use and covered in' OMS 3.18, namely, covenants, supplemental letters directly related to a covenant and supplemental letters of intent. Of these, the first two (covenants and supple- mental letters directly related to covenants) are legally binding. The third, the supplemental letter of intent, is designed to cover "statements of intention and the like which are significant to the project but not clearly meant to constitute a covenant" [OMS 3.18, paras. 31(a) and 32]. Such letters, which can serve as a basis of a continuing dialogue between the Bank and the borrower, have been used in the past. Because they are not legally binding, they constitute less formal undertakings, as suggested by OED. The scope for enlarging the use of legally non-binding instruments is limited, as the Report indicates. -39 - ANNEX C Page 15 of 19 B. Improved procedures for reaching agreements To strengthen the process of preparing, negotiating and supervising legal covenants, and to ensure that their implications are fully understood by the borrower: 1. Appraisal missions should generally leave with the borrower an agreed Aide Memoire which would include recommendations regarding specific cove- nants that might be required. The Bank Management generally agrees with the intent of the recommendation, namely, that borrowers should be advised as soon as possible of the covenants that may be required. However, since in many cases a detailed deter- mination of probable covenants is not possible during field appraisal, it would be more helpful to advise bor- rowers of probable undertakings after the Decision Meet- ing at which the staff from all interested departments review the recommendations of the appraisal team and reach tentative decisions in major issues. In this way the mission's recommendations will receive some internal review before being conveyed to the borrower. It is now common practice to advise borrowers in writing of the details of the project and of major conditions at this time which, it is considered, is more helpful than the preliminary and unreviewed findings at the time of appraisal. - 40 - ANNEX C Page 16 of 19 2. As a rule, the legal documents and the draft Ap- praisal Report should be made available to the borrower four weeks before the start of negotia- tions. Alternatively, any time-efficient mecha- nism which would fully inform the borrowers in good time could be considered. The Management agrees that the borrower should be fully and promptly informed about the Bank's appraisal of the project and the related covenants. The Operational Manual (OMS 2.00, para. 44) permits the Regions to distribute the draft legal agreements and the Appraisal Report to the borrower prior to negotiations, and this is done in some cases. In other cases, the key covenants and their ratio- nale are submitted to the borrowers at the time of the invitation to negotiate, which can be anything between several days to two months before negotiations. Because a four-week requirement would be unnecessarily rigid and would lead to significant delays, a more flexible approach is preferable, as recognized in the OED Report. Current practice is to provide borrowers with the draft legal documents and Appraisal Report shortly before negotiations. Where borrowers request these documents for a longer period in advance, the Bank is prepared to do so within reasonable limits appropriate to the case, e.g., taking into account the number and complexity of the issues, prior experience, and other relevant factors. -41 - ANNEX C Page 17 of 19 3. Agreement reached on the interpretation of specific covenants should be recorded in Agreed Minutes of Negotiations, signed by both parties. The Management agrees. This is current practice. OHS 3.18 recognized that "the legislative history of an inter- national agreement may be a supplementary means of inter- pretation of the agreement. Therefore, it is at times useful to have a concise and brief agreed record of nego- tiations, especially concerning controversial matters." 4. Supervision missions should carefully monitor the borrower's compliance with all covenants. The Management agrees. The Operational Manual (OMS 3.50, Attachment B) requires a specific annex to all Supervision Reports summarizing the status of compliance with the most important conditions on which the loan was based. 5. Borrowers should be promptly informed in writing if they are deemed to be in violation of covenants and be asked what corrective action will be taken. Covenants which cannot be complied with should be re-negotiated. The Management agrees. An Aide Memoire is usually prepared by supervision missions and left with the borrower summarizing the mission's findings, including those relat- ing to the question of compliance with loan conditions. After each supervision mission, a letter is sent to the borrower indicating the main issues and problems identi- - 42 - ANNEX C Page 18 of 19 fied by the mission and outlining recommendations on actions to be taken. If, after following up these issues, a covenant is no longer appropriate, it is usually amend- ed or waived. C. The need for greater selectivity in the use of loan cove- nants. In his statement to the Board on July 14, 1983 the Assistant General Counsel summarized the approach proposed to implement this recommendation as follows. First, the objectives of the project should clearly distinguish between (i) the issues which are required to be agreed and covenant- ed within the context of the project requirements in order to achieve the project objectives, and (ii) such matters as would more appropriately serve as a starting point of a dialogue for future lending or as a follow-up to current project work or other work in the same sector. This latter type of dialogue has sometimes crept in in the form of rather wide covenants in loan covenants. These type of arrangements could be incorporated into letters of intent rather than covenants in the agreements. Second, selectivity can also be applied in another sense. Once the issues that are required to be agreed and covenanted for have been identified, these should be formu- lated in the loan document in a balanced manner, which would afford some indication of their relative weight. This is the issue of proportionality, of so-called major/minor - 43 - ANNEX C Page 19 of 19 covenants. This could be remedied by leaving in the main body of the agreement the important financial, managerial and other covenants and by taking the wide range of admi- nistrative arrangements, the administrative and organiza- tional covenants that are loosely described as minor cove- nants and putting them in plans of action in narrative form. These plans of action are subject to modification by agreement between the parties as the project progresses. There are other ways in which the legal documents can be streamlined and simplified. The Legal Department intends in the near future to present proposals in that regard. Conclusion Management considers the adequacy of loan covenants to be an important and continuing concern since they form an essential part of the Bank's dialogue with the borrowing member countries. The OED Report is a helpful reminder that: (i) project objectives and requirements must be reasonable and realistic; (ii) that the dialogue with the Bank's clients must be conducted so as to assure full mutual understanding; and (iii) that the Bank must be flexible in situations which are inherently complex and changing. The Management also proposes to introduce greater selecti- vity in the use of loan covenants; proposals to streamline and simplify legal documents are under way. - 44 - ANNEX D Page 1 of 8 OED REPORTS ISSUED DURING FY83 Project Performance Audit Reports Public Utilities Malaysia Second Kuala Lumpur Water Supply (Loan 908-MA) SecM83-13 12/30/82 Thailand First and Second Telecommunications SecM83-36 12/30/82 (Loans 864/1253-TH) Senegal First Telecommunications (Loan 866-SE) SecM83-564 05/31/83 Mali First Telecommunications (Credit 321-MLI) SecM83-645 06/06/83 Brazil Minas Gerais Water Supply & Sewerage (Loan 1009-BR) SecM.83-710 06/30/83 Egypt Telecommunications I (Credit 548-EGT) SecM83-711 06/30/83 Liberia Fourth Power (Loan 1600-LBR) SecM83-737 06/30/83 Mexico - Mexico City Water Supply (Loan 900-ME) SecM83-765 06/30/83 Guyana First Power (Loan 875-GUA) SecM83-766 06/30/83 Transportation, Tourism and Urban Development Algeria Port of Jijel (Loan 1427-AL) SecM82-787 09/06/82 Indonesia Fertilizer Distribution (Loan 1139-IND) SecM83-34 12/30/82 Brazil First Railway (Loan 786-BR) SecM83-59 12/30/82 Indonesia First Shipping (Credit 318-IND) SecM83-299 03/25/83 Kenya Fourth and Fifth Highways (Credit 276/Loan 932-KE) SecM83-601 06/03/83 East African Community Third Railway (Loan 674-EAC) SecM83-624 06/03/83 Trinidad and Tobago Second Highway (Loan 1235-TR) SecM83-736 06/30/83 Panama Fishing Port (Loan 1114-PAN) SecM83-738 06/30/83 Mexico Zihuatanejo Tourism and Tourism Development SecM83-755 06/30/83 (Loans 793/1524/1524-5-ME) Indonesia Jakarta Urban Development (Loan 1040-IND) SecM83-756 06/30/83 DFC, Industry and Non-Project Lending Turkey First Program Loan (Loan 1627-TU) SecM82-774 08/27/82 Bangladesh Fourth, Fifth, Sixth and Seventh SecM82-798 09/10/82 Import Program Credits (Credits 591/676/676-1/752 and 866-BD) India First Bombay High Offshore Development SecM82-855 10/11/82 (Loan 1473-IN) - 45 - ANNEX D Page 2 of 8 Project Performance Audit Reports DFC, Industry and Non-Project Lending - continued Senegal SOFISEDIT I (Credit 987-SE) SecM83-632 05/12/83 Egypt First and Second Development Industrial Bank SecM83-662 06/07/83 (Credits 412/576-UAR) Korea - The Small Scale and Medium Industry Bank SecM83-667 06/23/83 (Loan 1175-KO) Agriculture Sudan Second Mechanized Farming (Credit 311-SU) SecM82-820 09/28/82 Benin Technical Assistance (Credit 716) SecM82-955 11/24/82 Indonesia Sugar (Credit 405-IND) SecM82-956 11/24/82 Madagascar First Mangoro Forestry (Credit 525/Loan 1065) SecM82-984 11/24/82 Thailand Northeast Irrigation Improvement SecM82-979 11/29/82 (Credit 401-TH) Malaysia Johore Land Settlement (Loan 967-MA) SecM82-1009 12/13/82 Philippines First and Second Fisheries Credits SecM83-17 12/13/82 (Loans 891/1270-PH) Brazil First Agro-Industries Credit (Loan 924-BR) SecM82-1038 12/22/82 Morocco First Doukkala Irrigation (Loan 1201-MOR) SecM82-1039 12/22/82 Uganda Smallholder Tobacco (Credit 212-UG) SecM83-15 12/23/82 Tanzania Second Livestock Development (Credit 382-TA) SecM83-16 12/27/82 Brazil Alto Turi Land Settlement (Loan 853-BR) SecM83-19 12/29/82 Ecuador Milagro Irrigation (Credit 425-EC) SecM83-32 12/30/82 Romania Sadova Corabia Agricultural Credit SecM83-33 12/30/82 (Loan 1083-RO) Sudan Savannah Development (Credit 718-SU) SecM83-37 12/30/82 Turkey Fruit and Vegetable Export SecM83-38 12/30/82 (Loan 762/Credit 257-TU) Nepal Birganj Irrigation (Credit 373-NEP) SecM83-50 12/30/82 Korea Second Integrated Dairy Development (Loan 1193-KO) SecM83-51 12/30/82 Guyana Tapakuma Irrigation (Loan 1016-GUA) SecM83-57 12/30/82 Mali Office du Niger Technical Assistance SecM83-281 03/25/83 (Credit 854-MLI) Paraguay Small Farmer Credit and Rural Development SecM83-294 03/30/83 (Credit 509-PA) Pagei 3 o rCC1ect»Pefo-.rance Audit Reports 2=Zze ;rans juba Livestock (Credit 462-S0) S:ecM83-5ý21 05 04 8 svia Agricultural industries (Lean 894-YU) SecM83-659 J5 /i3 83 u3Av Sao Hil! Forestry (Lcan 1307-TA) SecM83-616 06/3/8 LUoca Usg3uriba Agriculturvi Development Sec}83-608 06/ 5' Yoga %i r moa-Coffee Deve2opment (Credit 503-TO) SecM83-629 06/1/8 x±eRice Processing (Loan 720-PH) SeeM83-631 c 15/8 Upe a Scnd Rural Development Fund (Credit 640-LV) eM83-633 06/1s/83 is Aurora Venarand, irrigation SecM33-615 I616/83 (Loar 984/Credit 472-AH) taacon 22yastock Development (Loan 983-cm) SEcN3-626 06/1/8 AcA Rural Development (Credit 644-GM) SecM83-628 06/ vovy Mast Conon Areas Rural Development SecM33-649 06 20 83 SqUiement (Crediz 505NEP) SeM83-652 G06/20/83 Chbal Command Area Development (Rajasthan and SecM83-666 0620 KaAraesh) (Loan 1011/Credit 562-IN) s orestry (Credit B00-NIR) SecM83-660 06/2 J/3 -= n Second integrated Agricultural Development SecM83-689 06/24/E a aMnd Fisberies (Loan 1398-PAN) SecM83-7- O3 /39/83 a inagrated Rural Development (PIDER I) SecM83-731 , /30/3 amaia Smcond Education (koan 727-jM) SecM82-899 11/01/ 2 Philippimes Second Eduvation (Credit 349-PH) SeYM82-898 11 I/C7 MauriMius First Education (Loan 1033/Credit 501-MAS) Se-M83-l 12/ 3/82 Ecuador First Education (Credit 124-EC) SeeM83-618 05 / 16/8 Zari7a Virst and Second Educacion (Loans 592/645-ZA) SaeMS3-617 Q5/23/83 Terea Second Education (Loan 906/Credit 394-KO) SecM83-520 03/24/'"3 Morocco Second Educacion (Credit 266-MOR) SecM83-599 06/14/3 - 47 - ANNEX D Page 4 of 8 Project Performance Audit Reports Education - continued Singapore Second Education (Loan 964-SI) SecM83-600 06/14/83 Korea Third Education (Loan 1096-KO) SecM83-625 06/14/83 Thailand Second Education (Loan 822-TH) SecM83-627 06/17/83 Pakistan Second Education (Credit 206-PAK) SecM83-665 06/17/83 Thailand Third Education (Credit 369-TH) SecM83-672 06/27/83 Haiti First Education (Credit 618-HA) SecM83-712 06/30/83 Ivory Coast Second Education (Loan 1125-IVC) SecM83-715 06/30/83 Pakistan First Education (Credit 50-PAK) SecM83-717 06/30/83 Trinidad and Tobago Second Education (Loan 862-TR) SecM83-730 06/30/83 Paraguay Second Education (Loan 1252-T-PA) SecM83-767 06/30/83 Population Egypt First Population (Credit 437-UAR) SecM82-869 10/15/82 - 48 - ANNEX D Page 5 of 8 Pass-Through Project Completion Reports Public Utilities Liberia Third Power (Loan 1140-LBR) SecM82-1018 12/06/82 Oman Engineering for Power and Urban Water Supply SecM83-10 12/17/82 (Loan 1442-OM) Turkey Istanbul Power Distribution (Loan 892-TU) SecM83-49 12/29/82 Panama Third Power (Loan 948-PAN) SecM83-56 12/30/82 Portugal Sixth Power (Loan 1301-PO) SecM83-83 01/04/83 People's Democratic Republic of Yemen - Water Supply SecM83-134 01/24/83 Engineering and Technical Assistance (Credit S-21-YDR) Venezuela Second Telecommunications (Loan 770-VE) SecM83-223 03/10/83 Philippines Fifth Power (Credit 296/Loan 809-PH) SecM83-240 03/16/83 Costa Rica Fourth Telecommunications (Loan 1006-CR) SecM83-300 03/30/83 Fiji Second Telecommunications (Loan 1140-FIJ) SecM83-342 03/17/83 Tunisia Second Power (Loan 1355-TUN) SecM83-394 04/22/83 Upper Volta Second Telecommunications (Credit 431-UV) SecM83-621 05/17/83 Sierra Leone Third Power (Credit 734-SL) SecM83-592 05/31/83 Colombia Palmira Water Supply and Sewerage (Loan 738-CO) SecM83-714 06/28/83 Ghana Accra Tema Water Supply (Stage II) (Credit 499-GH) SecM83-716 06/28/83 Tanzania Kidatu Hydroelectric (Stage II) (Loan 1306-T-TA) SecM83-762 06/30/83 Transportation, Tourism and Urban Development Sudan First Highway (Credit 331-SU) SecM83-30 12/21/82 Peru Sixth Highway (Loan 1025-PE) SecM83-14 12/30/82 Paraguay Fourth Highway (Loan 1059-PA) SecM83-22 12/30/82 Honduras Second Port (Loan 767-HO) SecM83-31 12/30/82 Honduras Sixth Highway (Loan 894-HO) SecM83-58 12/30/82 Colombia Sixth Highway (Loan 680-CO) SecM83-107 02/02/83 Botswana Francistown Urban Development SecM83-180 02/25/83 (Credit 471-BT) Argentina Third Highway (Loan 734-AR) SecM83-179 03/03/83 Guinea First Highway (Credit 596-GUI) SecM83-232 03/15/83 Argentina First Railway (Loan 733-AR) SecM83-260 03/21/83 - 49 - ANNEX D Page 6 of 8 Pass-Through Project Completion Reports Transportation, Tourism and Urban Development - continued Israel Second Highway (Loan 781-IS) SecM83-579 05/02/83 Ecuador Fourth Highway (Loan 1231-EC) SecM83-519 05/09/83 Thailand Fifth Highway (Loan 870-TH) SecM83-522 05/09/83 Upper Volta Third Highway (Credit 653-UV) SecM83-580 05/16/83 -EEC Special Action (Credit 38-UV) Mauritania Third Highway (Credit 519-MAU) SecM83-581 05/16/83 Brazil Santos Port (Loan 756-BR) SecM83-614 05/16/83 Yugoslavia Naftovod Oil Pipeline (Loan 1173-YU) SecM83-591 05/31/83 Ivory Coast Tourism Development (Loan 1124-IVC) SecM83-661 06/24/83 Tunisia Hotel Training (Loan 1029-TUN) SecM83-671 06/24/83 Tunisia Tourism Infrastructure SecM83-682 06/24/83 (Loan 859/Credit 329-TUN) Ivory Coast Fifth Highway (Loan 1161-IVC) SecM83-757 06/24/83 Burundi First Highway (Credit 467-BU) SecM83-673 06/27/83 Mali Second and Third Highways (Credit 383/599-MLI) SecM83-758 06/29/83 Sudan Domestic Aviation (Loan 1175-SU) SecM83-686 06/30/83 Sudan Third Railway (Credit 457-SU) SecM83-690 06/30/83 Algeria Al Bethioua Port (Loan 995-AL) SecM83-691 06/30/83 Zambia Lusaka Squatter Upgrading and Site and Services SecM83-779 06/30/83 (Loan 1057-ZA) DFC, Industry and Non-Project Lending Colombia Pre-Investment Studies (Loan 971-CO) SecM82-961 11/30/82 Peru Program Loan (Loan 1693-PE) SecM82-1000 12/06/82 Dominican Republic Urgent Import Requirement SecM82-991 12/16/82 (Loan 1782-DO) Morocco CIOR Cement (Loan 1383-MOR) SecM82-1040 12/28/82 Senegal Dakarmarine Engineering (Loan S-3-SE) SecM82-1041 12/28/82 Botswana Development Corporation (BDC) (Loan 1019-BT) SecM83-228 03/16/83 Israel Industrial Development (Loan 1116-IS) SecM83-380 04/14/83 Cyprus - Second Loan to the Cyprus Development Bank SecM83-481 04/21/83 (Loan 1278-CY) - 50 - ANNEX D Page 7 of 8 Pass-Through Project Completion Reports DFC, Industry and Non-Project Lending - continued Egypt Second Talkha Fertilizer (Credit 484-EGT) SecM83-393 04/22/83 Colombia Small Scale Industry (Loan 1451-CO) SecM83-464 04/27/83 Pakistan National Development Finance Corporation SecM83-419 04/28/83 (Credit 540-PAK) Zambia - The Development Bank of Zambia (DBZ) SecM83-563 05/27/83 (Loan 1210-ZA) - Bangladesh First and Second Technical Assistance SecM83-609 06/15/83 (Credits 409/622-BD) India IFFCO Fertilizer Expansion (Loan 1079-IN) SecM83-639 06/15/83 Philippines - Philippines Development Corporation SecM83-620 06/17/83 of the Philippines (PDCP) (Loan 1052-PH) Agriculture Indonesia Fourth Agricultural Estate (Credit 319-IND) SecM82-938 11/08/82 Mexico Fifth and Sixth Agricultural Livestock Credits SecM83-13 12/28/82 (Loans 1217/1569-ME) Honduras Second Livestock Development (Credit 434-HO) SecM83-18 12/30/82 Uruguay Fifth Livestock Development (Loan 1166-UR) SecM83-52 12/30/82 Cameroon Plaine des M'bo Rural Development (Credit 672-CM) SecM83-280 03/09/83 Chile Livestock, Fruits, Vineyards and Agroindustry SecM83-407 04/25/83 Credit (Loan 1350-CH) Indonesia Fisheries Credit (Credit 480-IND) SecM83-565 05/03/83 Liberia Lofa County Agricultural Development SecM83-593 06/14/83 (Credit 577-LBR) Colombia Second Atlantico Development (Loan 849-CO) SecM83-668 06/23/83 Burundi Second Coffee Improvement (Credit 593-BU) SecM83-683 06/23/83 Education Jordan Second Education (Credit 534-JO) SecM83-35 11/30/82 Ireland Second Education (Loan 959-IRE) SecM83-147 01/31/83 Malaysia Second Education (Loan 810-MA) SecM83-623 03/30/83 Indonesia Third Education (Credit 387-IND) SecM83-622 06/17/83 Malawi Second Education (Credit 590-MLI) SecM83-670 06/24/83 Ethiopia Third Education (Credit 417-ET) SecM83-650 06/27/83 Somalia Second Education (Credit 511-SO) SecM83-681 06/27/83 - 51 - ANNEX D Page 8 of 8 Other OED Reports Operational Policy Review: Compliance with SecM82-782 & 09/01/82 Loan Covenants 782/1 Annual Report on Operations Evaluation R82-269 09/03/82 Eighth Annual Review of Project Performance R82-270 09/09/82 Audit Results Concordance to Project Performance Audit SecM82-848 09/20/82 Reports Issued by the Operations Evaluation Department March 1972 to June 30, 1982 Sector Operations Review: The Water Supply SecM82-899 11/01/83 & Waste Disposal Program in Tunisia Operational Policy Review: Review of Bank SecM83-9 & 9/1 12/23/82 Technical Assistance to Bangladesh Impact Evaluation Report - Philippines: SecM83-499 05/06/83 First Education (Loan 393-PH) Impact Evaluation Report - Philippines: SecM83-708 06/17/83 Second Rural Credit (Loan 607-PH) Impact Evaluation Report - Indonesia: SecM83-709 06/22/83 Irrigation Rehabilitation (Credit 127-IND)