Report No. 5247 Annual Report on Operations Evaluation August 30, 1984 Operations Evaluation Departme it FOR OFFICIAL USE ONLY U Document of the World Bank This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. FOR OFFICIAL USE ONLY THE WORLD BANK Washington. D.C. 20433 U.S.A. Ofce of Director-Ceeral Operatons Evakation August 30, 1984 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Annual Report on Operations Evaluation Attached, for information, is this year's "Annual Report on Operations Evaluation." The report provides an overview and updates the status of the operations evaluation function in the Bank Group. Mention is also made of the progress being made in the Bank's borrowing member countries in establishing evaluative mechanisms, and of the links with other multilateral and bilateral aid agencies in the common effort to improve the efficiency and effectiveness of the development process through a properly functioning evaluation system. The report contains a summary of the present status of the outstanding recommendations of the Operations Evaluation Department's major operational policy reviews and evaluation studies. Attachment This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. FOR OFFICIAL USE ONLY ANNUAL REPORT ON OPERATIONS EVALUATION Table of Contents Page No. I. Introduction ............................................. 1 II. The Operations Evaluation System ......................... 1 III. Operations Evaluation Department ........................ 7 IV. OPS: Monitoring and Evaluation in Projects .............. 10 V. Other Evaluation Activities .............................. 13 VI. Post-Evaluation in Member Governments .................... 16 VII. The Wider Network ........................................ 19 VIII. Follow-Up ................................................ 20 Annex A: Status Report of the Regions on Borrower Governments' Involvement in the Post- Evaluation of Bank/IDA-Financed Projects (All countries with five or more PCRs scheduled through CY85) ................... 21 Annex B: Status Report of the Regions on Borrower Governments' Involvement in the Post- Evaluation of Bank/IDA-Financed Projects (Selected countries with less than five PCRs scheduled through CY85) ................. 29 Annex C: The Current Status of Recommendations of OED Policy Reviews and Evaluation Studies ..... 35 Annex D: OED Reports Issued During FY84 ............... 55 This document has a restricted distribution and way be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. ANNUAL REPORT ON OPERATIONS EVALUATION Abbreviations used JAC - Joint Audit Committee OED - Operations Evaluation Department PPAR - Project Performance Audit Report PCR - Project Completion Report OPS - Operations Policy Staff IFC - International Finance Corporation EDI - Economic Development Institute M&E - Monitoring and Evaluation OMS - Operational Manual Statement OECD/DAC - Organisation for Economic Co-operation and Development/Development Assistance Committee JIU - Joint Inspection Unit, United Nations - 1 - ANNUAL REPORT ON OPERATIONS EVALUATION I. Introduction 1. This report presents an overview and updates the status of the various constituent units of the operations evaluation system in the Bankl/. It provides some informa- tion in respect of ongoing efforts being made in the borrow- ing countries to set up central evaluation systems or to participate in the evaluations made by Bank staff by way of preparing project completion reports and/or by contributing to project performance audit reports and project completion reports before these are finalized. The report also updates the status of still outstanding recommendations of past reports of the Operations Evaluation Department (OED), and provides some information on new initiatives being taken in the operations evaluation field in the Bank and the Interna- tional Finance Corporation (IFC). II. The Operations Evaluation System 2. A selective project performance auditing system2/ for the Bank's operations was introduced with the approval of the Executive Directors in mid-1982 and has continued to function smoothly since then. As the number of completed projects has grown, it has become increasingly clear that the introduction of the selective auditing has been very timely and that the cost of a fully comprehensive system would have been prohibitive and the incremental value ques- tionable. The number of completed projects reviewed jumped from 122 in FY82 to 195 in FY84 and could well reach 250 in the current fiscal year. Of 195 projects reviewed in FY84, 111 (57%) were directly audited by OED staff, 70% of them following country visits: a significantly higher proportion than in the past. For the remaining 84 projects, completion reports prepared by operating staff were scrutinized in OED and released to the Executive Directors and the President. 1/ For earlier reports in this series please see R76-245 of October 13, 1976; R77-256 of October 3, 1977; and reports No. 2190 of August 28, 1978 to 4679 of August 24, 1983. 2/ For details see Annex A of report No. 4104 of Septem- ber 3, 1982. - 2 - Borrower comments continued to be sought on all draft project performance audit reports (PPARs) and pass-through completion reports (PCRs). The annual reviews of project performance audit results remain comprehensive in their coverage and include the findings of both PPARs and PCRs. 3. The table below shows the number of projects covered in the last four years, the number of project completion reports received from the operating departments, and the carry-over from one year to another together with the estimates for FY85. It will be noticed that the total Status of PCRs Received Actual and Estimated FY81 to FY85 Actual Estimated/a FY81 FY82 FY83 FY84 FY85 PCRs brought forward lb 87 136 173 184 167 PCRs received 143 159 180 178 250 Total (A) 230 295 353 362 417 Projects reviewed: PPARs 94 122 91 111 125 Pass-through PCRs - - 78 84 125 Total (B) 94 122 169 195 250 PCRs carried forward; 136 173 184 167 167 as percentage of Total (A) above 59% 59% 52% 46% 40% /a The FY85 estimate of PCRs received is based on the numbers of loans and credits with closing dates on or before the end of Calendar Year 1984 and which were more than 85% disbursed, adjusted as a result of discussions between OED and Projects staff. The resulting figure of 250 is greater than the earlier estimate of 198 PCRs contained in the Bank's FY85 Administrative Budget. The situation will be reviewed at the time of the mid-term budget review. /b At various stages in the audit process. - 3 - number of PCRs brought forward and received from year to year has grown by 57% between FY81 and FY84 and the carry- over by about 23%. Larger numbers of project completions and a heavier inflow of PCRs is likely to lead to larger stocks in hand in OED at the end of the fiscal year, the more so because of the pattern of PCR receipts. Almost 51% of the 178 PCRs received by OED in FY84 arrived in the final quarter of the year, and 31% in the month of June alone. Despite this bunching, the ratio of PCRs carried forward to those in hand at the beginning of the fiscal year is being brought down but is likely to reach a plateau in about a year's time. This plateau will be determined not only by the inflow pattern of PCRs but also by the size of PCR stock needed in OED for cost-effective advance planning of work. 4. The linchpin for the satisfactory operation of the modified project performance auditing system is the quality of project completion reports. This quality is now gen- erally of a high standard. On the basis of the findings of project performance audit reports, OED staff have suggested that the PCR guidelines require the inclusion in the PCRs of a sensitivity/risk analysis of ree3timated rates of return at project completion and that the assumptions and other data on which these reestimates are based be set forth in an annex. The intention is to heighten the sensitivity of staff to the longer term viability of projects and their benefits and to permit more systematic review and comment on project sustainability in future OED work. A special section on the sustainability issue appears in the Tenth Annual Review of Project Performance Audit Results. 5. According to Operational Manual Statement No. 3.58, Guidelines for Preparing Project Completion Reports, operating staff are required to provide in an annex of the project completion report the kind of data and sensitivity analysis required at appraisal. However, this requirement is frequently not observed, in part because of deficiencies in the earlier project documentation. More recently, in September 1983 instructions were issued by OPS requiring that the rationale for Bank support to the project and the key assumptions and judgments bearing on the project justification be more fully covered in the staff appraisal reports. These instructions require that the documents indicate the assumptions underlying the allocation of costs and benefits among different components of a multi-component project, and that the justification include an analysis of the sensitivity of results to the main assumptions, together with a judgment on the likely variations around the assumed central values. What is needed now is to place these various instructions in the sustainability context above, and to require that the project completion reports also regularly provide the needed background data and sensitivity analysis. OPS now plan to issue instructions to reiterate these requirements, and OED staff will check on their observance in individual project completion reports as they are received and reviewed. 6. The cost effectiveness of the modified evaluation system is being kept under review. In FY84 the cost to operating units of preparing each PCR was ten staff weeks on average and to OED for preparing audits in respect of selected projects, about six staff weeks. Including the limited time spent on making a judgment whether a project should be subjected to full OED audit or not and in scruti- nizing PCRs selected for pass through, the overall evalua- tion cost to OED per project reviewed in FY84 was 3.6 staff weeks. This unit cost is expected to decline further as the proportion of PCRs to be passed through increases to the planned 50% in FY85 as against 43% in FY84. The overall cost to the Bank of the evaluation process for FY85 is currently estimated at about 59 staff years, comprising 39 staff years for PCR preparation and 20 for the OED input. 7. In allocating OED staff time to different activi- ties, some trade-offs are involved. Since the evaluations of individual projects serve as building blocks for group reviews and other special studies, it is essential to maintain a high level of quality and objectivity in these evaluations. It is considered that direct evaluation by OED staff of about half of the completed projects, based on approved criteria, would be about right for this purpose. Cost effectiveness would be further enhanced by increasing recourse to grouping of projects for evaluation, exemplified by the combined performance auditing of six agricultural projects in Tanzania in FY84. Such sets of reviews are also expected to provide more substantial lessons and recommenda- tions based upon broader patterns of experience. New direc- tions of OED work program - reviews of structural adjustment loans and other policy-based program operations, and the ongoing country program and policy reviews - are also likely to lead to changes in the allocation of resources within the operations evaluation function. 8. In discharge of its oversight of the operations evaluation function, the Joint Audit Committee of the Board has since 1977 appointed a subcommittee each year to review a number of project performance audit reports to assess the adequacy of the system. The subcommittee for FY84 had the additional responsibility to review the workings of the modified system introduced in 1982. The subcommittee - 5 - reviewed five PPARs and two PCRs which were passed through without audit, concentrating on projects in the rural development and education sectors in addition to the audit of the Kenya SAL I and a pass through PCR on a development bank project. 9. The subcommittee concluded that the PPARs reviewed generally covered the basic issues that had to be addressed and provided an objective and thorough appraisal of the projects examined. Both project performance audit memoranda and project completion reports were observed to be of high quality, and OED was seen to be performing its function adequately and efficiently. Among its several recommenda- tions were that the first projects or programs in a series should be audited, that the PPAR Highlights provide reasons for changes in the estimated rates of return and more fully discuss time and cost overruns, and that the Bank should consider providing technical assistance to borrowers for the preparation of PCRs, where necessary. The subcommittee reiterated an earlier recommendation that summaries of the lessons to be learned from a project experience be issued - probably by OPS - in a format primarily oriented to the needs of operational staff. The subcommittee's report and recommendations were discussed by the Joint Audit Committee at its meeting on July 12, 1984. 10. The issue of dissemination of the findings of evaluation has been engaging the attention of the Bank for some time. Special attention was drawn to it by the Joint Audit Committee and its subcommittee for review of the operations evaluation function in 1982, and last year's annual report addressed the matter in some detail. The dissemination of PPAR findings remains a matter of priority concern and further comments on some recent developments are provided be".ow. 11. As stated last year, OED reports are distributed in large numbers throughout the Bank but are apparently either not always easily accessible to staff at the working level or, in the pressure of other work, do not receive the attention that they deserve. To some extent the problem has been aggravated by an increase in the number of completed projects and in the number of evaluations being put out annually. The matter is therefore being addressed at different levels. First, in response to complaints of insufficient dissemination from working level staff in one Region, OED looked into the matter jointly with the regional management. It was the intention, if found necessary, to extend an existing informal partial distribution system which has been operated by OED so as to assure distribution - 6 - of all OED material to every division in at least one sector. However, the investigation revealed that sufficient copies of the reports were already being received in the Region and that internal distribution needed to be stream- lined. Regional management has now acted to ensure this, and the attention of other managers has been drawn to these arrangements for possible replication in the other Regions. 12. Second, in OED's work program special emphasis is being given to overviews of project experience in selected subsectors or related to specific topics. Examples include an overview of Bank experience with fisheries development which was recently completed, and ongoing reviews of cost recovery in irrigation projects and the experience of settlement projects. Such digebts of experience, focussing on recurring designs and patterns, not only provide broader and more firm conclusions but should also help to ease the pressure of numbers. As already stated (para. 7) grouping of projects for performance audits also has other benefits and will be increasingly emphasized in OED's work programming. 13. Third, the possibility of OPS sector departments issuing memoranda on "lessons learnt" from PPARs, on the lines of the excellent procedure already being followed by the Agriculture and Rural Development Department (AGR), will be pursued further (see also para. 50 below). The need for this was reiterated in the report of the JAC subcommittee mentioned above. OED is also planning to issue a series of booklets, one for each sector, carrying the lessons of experience that have emerged from project completion and project performance audit reports. Work on the first such booklet relating to the agricultural sector has already been jointly initiated with AGR and is expected to be completed in the coming months. 14. Fourth, increasing use is being made of OED staff participation in seminars and project design courses to directly disseminate the findings of PPARs. OED staff made a major contribution to the Bank's First Agricultural Symposium in January 1984 by presenting a review of experi- ence with agricultural projects in Sub-Saharan Africa audited over the last six years. As a result, it was decided that arrangements would be made for joint action between OED, OPS and Regional staff to develop improved designs for agricultural projects in Sub-Saharan Africa. Earlier, in September, 1983 OED organized a seminar in Lome, Togo, for staff of the West African Development Bank (BOAD). The seminar resulted from initiatives taken by the President of BOAD and regional management and was intended -7- to review OED procedures and experience to determine how these could be adapted to suit the needs of BOAD. In addition to providing exposure to the evaluation function to BOAD staff, concrete proposals emerged from the seminar for the establishment of an evaluation function in the office of the President of BOAD. Also earlier in 1983, OED and the Agriculture and Rural Development Department staff jointly sponsored a seminar to review how socio-cultural variables affected agriculture and rural development projects. The underlying study was based on a review of 68 PPARs and impact evaluation reports. 15. Fifth, a recent instruction to operational staff from the office of the Senior Vice President, Operations, requires that the presentation of new loans to the Board should, whenever appropriate, make reference to related lessons emerging from the PCR/PPAR process, thus reinforcing feedback. This instruction also reinforces a standing requirement that the President's report for a loan proposal should reflect the findings of performance evaluations of earlier projects in the same series. Sixth, changes are being made in the Concordance to project performance audit reports, and OED's memory and retrieval system for the findings of the audit process is being refined and enlarged with a view to making such findings more accessible to operating staff and to the research complex (see also paras. 19-21 below). III. Operations Evaluation Department 16. Three instruments are being used to consolidate the findings of OED's work, making them accessible, and develop cross-cutting and comparative analyses over time and within and across sectors, regions and countries. The annual review of project performance audit results is one such document. The Ninth Annual Review together with its companion Annual Report on Operations Evaluation were discussed first in the JAC and subsequently by the Executive Directors in November, 1983. The annual review, with the names of countries and projects removed, was also released to the general public following the precedent established in 1977 and subsequent years. The Tenth Annual Review will be issued shortly. 17. In terms of the second approach to differentiated analysis within the review of certain groups of projects and - 8 - functions, OED issued 12 special studies in FY84.3/ These included eight impact evaluation reports and four special studies, ranging from a review of Bank experience with agricultural research and extension and with fisheries development, to institutional development in Sub-Saharan Africa. A special review was undertaken of built-in monitoring and evaluation in rural development projects in Northeast Brazil: a case study which led to some broad conclusions (also see para. 26). These impact evaluations and special studies were reviewed by the Joint Audit Committee. Staff from OED and from operating departments joined the discussions which sought to relate the findings of the studies to the Bank's current approach and practice in the areas under discussion and, occasionally, to planned changes. 18. Two studies (Review uf Agricultural SecLor Opera- tions, Malawi and Burkina Faso, and Factors Affecting Project Sustainability) are nearing completion and are expected to be distributed shortly. Work is under way on seven other studies4/; and a special study of experience 3/ Thailand First Education (SecM84-458); Malawi Lower Shire Valley Agricultural Development Projects (SecM84-16); The Gambia Agricultural Development Project (SecM84-656); Benin-Hinvi Agricultural Project (SecM84-302); Dominican Republic Livestock Development Project (SecM83-941); Kenya First and Second Forestry Projects (SecM84-144); Ivory Coast Oil Palm and Coconut Development Projects (SecM84-462) and Senegal Settle- ment Projects in the Terres Neuves Region (SecM84-653); Strengthening Agricultural Research and Extension - The World Bank Experience (SecM83-879); Harvesting the Waters - A Review of Bank Experience with Fisheries Development (SecM84-240); Built-In Project Monitoring and Evaluation: Rural Development in Northeast Brazil (SecM84-466); Institutional Development In Africa - A Review of Bank Project Experience (SecM84-457). 4/ Experience with Land Settlement Projects; Review of Experience with Smallholder Livestock; Water Charges and Cost Recovery in Irrigation Projects; Achieving Adjustments in Agriculture Input/Output Pricing Policies; Built-in Monitoring and Evaluation: An Overview; Review of the Bangladesh Imports Program Credits; Impact of World Bank Lending for Education (Korea); and Develcpment of the Trapsport Sector in Kenya and Tanzania: World Bank Role. - 9 - with rainfed agricultural development and five impact evaluations are planned in FY85. 19. The findings of the project performance audits on which reports were issued by OED through June 30, 1984 have been c3assified and entered into the computerized memory and retrieval system which has now been in place for more than five years. This system comprises the third instrument for the consolidation and analysis of the findings of project performance audits. The first computerized concordance to project performance audit reports was issued in August 1978 and updated versions have been subsequently issued periodi- cally. The latest version, incorporating the findings of 949 project performance audit report was issued tn October 1983, and a revised and updated version will be published shortly. 20. Substantial progress has been made in the develop- ment of an improved computerized information storage and retrieval system. The first phase of a two-phase program is expected to be completed by the end of this year, at which time the operation of the system will be reviewed and a further program developed for phase two. The first phase will permit the storage, management, and analysis of textual and numeric information which is contained ia the PPARs and PCRs. It will also lead to the design of a new concordance classification system. The second phase will include more extensive project results, essentially numeric, including essential information on product output, beneficiaries, and income effects of projects. 21. The new memory and retrieval system will be integrated with the bibliographic data system developed by Administrative Services, and with the larger Management Information System now in process, so that project results can be readily linked with information on project design and objectives at the time of appraisal and with salient points of the project experience during implementation. In addition to the production of the concordance, major benefits to be derived from the system at completion will include an extremely flexible search and retrieval of targeted information. It will be possible to undertake searches by country, region, sector or subsector, to support analyses for various purposes. 22. The OED work program and budget for FY85 reflects qualitative changes in parallel with the recent shift towards policy-based lending in the Bank's operations. Provision has been made for the timely review of structural adjustment loans and special purpose program loans which - 10 - tend to disburse quickly and whose performance evaluation raises special issues. OED has also initiated a series of overall country program and policy reviews to examine the results of interaction with selected borrowers over time at the macro economic, sector policy and operational levels. OED's staffing pattern and the use of consultant resources have been adapted to take care of these new directions of work. As in the previous year, the budget document also carries a three-year rolling program of special studies. This program will be subject to JAC review and will be modified from year to year in line with operational needs and the concerns of Executive Directors and the Bank's operational management. 23. OED's staffing difficulties, which have been the source of considerable concern in the past, have substan- tially eased. The department presently has only two higher level vacant positions, one of them newly authorized. In the 13 months through mid-August 1984, seven operational staff had transferred to OED and an equal number of OED staff have been reassigned to operating departments. With continued assistance from senior operating managers and the Personnel Management Department, it is expected in the future to smoothly operate the policy of limited-term assignment to OED of high caliber operating staff and their reassignment back to the operating complex on completion of that term. 24. OED continues to be visited by staff from member governments interested in learning about OED structure and way of working or for exchange of views on evaluation. Visits were received not only from staff from developing countries but also from other multilateral and bilateral aid agencies. IV. OPS: Monitoring and Evaluation in Projects 25. Monitoring and evaluation (M&E) elements continue to be included in Bank projects in certain sectors. All agriculture and rural development projects approved in FY84 included these elements to serve as management tools and to provide for assessment of results. A significant proportion of the built-in monitoring and evaluation components have followed the Design Guidelines and the "Handbook for Monitoring and Evaluation of Agriculture and Rural Develop- ment" published for the Bank by the Johns Hopkins Press. Demand for the English, French and Spanish versions of the handbook continued to increase throughout the year. - 11 - 26. In order to underpin specific technical aspects of monitoring and evaluation, AGR is preparing technical notes on "Estimating Project Crop Production: Methods and Limita- tions" and on "Statistical Design for Monitoring and Evalua- tion Surveys". A case study on monitoring and management of the Thailand Agricultural Extension Project is now in the final stages of preparation, and work has been initiated on a document addressing the monitoring and evaluation of the sociological aspects of agriculture and rural development projects. In the meantime OED has done three special studies of the monitoring and evaluation experience built into agricultural projects and one for education projects. OED is now working on an overview which will cover the experience of about 20 projects already reviewed specially for their monitoring and evaluation components, and will also draw upon a selection of PPARs from about 300 agricul- tural projects which have been reviewed and in which built-in monitoring and evaluation was a feature. The publication of all this material, and the distillation of conclusions and lessons which will follow, should further enhance the quality of monitoring and evaluation activities of project specific systems. 27. In the last year, AGR monitoring and evaluation staff have participated in the design, appraisal and super- vision of monitoring and evaluation activities in borrowing countries at the regional and national level. The country profiles, mentioned in last year's report, will help to identify methodological and institutional issues requiring attention and will provide an important input to a series of country seminars being planned in conjunction with the concerned projects/programs divisions. AGR staff have also taken the initiative to foster closer coordination among co-financing international and bilateral agencies with similar concern in developing institutional capacity within borrower agencies. The first stage of a pilot activity in monitoring and evaluation for a major irrigation project in Thailand, jointly executed with the Asian Development Bank, has just been completed. Further opportunities for similar action in other countries are being explored. Internally, AGR staff have shifted their review and support services in monitoring and evaluation to earlier stages of the project cycle and begun to examine monitoring and evaluation issues at the broader sector and country levels. Meanwhile, as evaluation results from earlier efforts become available, the lesson learning and experience dissemination roles have correspondingly increased. 28. In urban development projects, effective monitor- ing and evaluation systems established early in the project - 12 - cycle are becoming standard requirements. Systematic efforts have been made in the past year to build M&E activi- ties into new projects at the pre-implementation stage and to incorporate lessons from evaluation activities conducted in earlier projects. Where necessary, operating staff have consulted with evaluation specialists in the Urban Develop- ment Department in designing such systems at headquarters and in providing technical assistance in implementing them in the field. All 15 urban development projects approved in FY84 included the provision of monitoring and evaluation systems. 29. Demand continues to grow for assistance from the Urban Development Department in replicating the partic- ipant/observer qualitative evaluation method which was successfully employed on a pilot basis in urban projects in Ecuador and Bolivia in FY82-83. This approach goes beyond the traditional impact evaluation, its special strength being its ability to vicariously monitor the process whereby change occurs. Managers are thereby provided ongoing feedback from the beneficiaries' point of view on a regular basis with which they can make adjustments in the project and constantly improve performance. Results of the first two years of participant observation have been disseminated through a department discussion paper followed by a symposium in May 1984. They are also planned for publica- tion in book form in FY85. This evaluation approach is also being replicated in project settings in Brazil and Thailand (urban) and in Bolivia (rural). 30. The OPS Education Department continues to direct evaluation work to three objectives: (i) to improve the quality and usefulness of evaluation components in education projects; (ii) to increase the capacity of borrower agencies to gather and interpret information to assist policy formulation and to improve project design and implementa*- tion; and (iii) to assess the effectiveness of programs in institutions financed by education projects. A group of case studies of monitoring and evaluation in education projects is nearing completion. They will examine the evaluation components of the Pakistan Primary Education project and the Philippines Educational Radio project, and the monitoring and evaluation system of the Haiti Educa- tional Reform which is being assisted by several projects. A general operational review of evaluation undertaken in education projects is also uqder way and will be completed in FY85. In cooperation with several bilateral aid agencies, a manual for the Evaluation of Vocational Training was developed and field tested and is now being published. - 13 - The DiSCus Study, which assessed the impact and effective- ness of diversification of secondary education in Bank financed projects in Colombia and Tanzania, conducted under the auspices of the research unit, has been completed. V. Other Evaluation Activities 31. Note is made below of the evaluation activities of the Research complex, the Economic Development Institute and the International Finance Corporation. OED's relationship with each has been changing over time; this note reflects the present status of that relationship and indicates some proposals for the future. Research 32. In June 1984, the Research Policy Council (RPC) approved two sets of procedures for the evaluation of completed research projects by the Research Projects Approval Committee (REPAC). The first applies to the backlog of completed projects not yet evaluated; the second, to all ongoing and future research projects. The backlog comprises projects completed before July 31, 1984. They will be evaluated by nine separate subject panels, each under the chairmanship of a REPAC member and comprising both internal and external reviewers as well as a representative from OED. All ongoing and future research projects will be subject to formal evaluations by ad hoc panels set up by REPAC' which would consist of both internal and external reviewers and an OED representative. Each panel will be chaired by a REPAC member. 33. Fairly tight procedures have been developed to ensure timely evaluation and for the dissemination of its results. Evaluation will follow the filing of a completed report on the project by the research sponsor. Failure to file a completion report by the closing date of a project will preclude the researcher(s) from submitting new proposals to REPAC until such completion report has been filed. In order to ensure the promised technical quality of the research work, REPAC will take into account the technical evaluation of his/her previous research projects, when a researcher submits a new proposal. Broad distribution of the evaluation reports throughout thz Bank will be undertaken. 34. The "Updating Report on the World Bank Research Program" (R83-356) was discussed by the Executive Directors - 14 - on January 12, 1984. The evaluation of completed research projects was held off in the past year due to the reorgani- zation of the research management. In line with past practice, evaluation of completed research projects under the revised procedures will be filed in the office of the DGO for reference by interested Executive Directors. Economic Development Institute 35. EDI has continued the evaluation activities described in last year's Annual Report on Operations Evalua- tion. These evaluations are based on questionnaires given to participants at the end of the course, and seek the participants' evaluation of the relevance of the course objectives and EDI's teaching methods to their needs and their success in achieving their objectives. Course director reports then draw lessons from participant ques- tionnaires and also review the course experience generally and make recommendations for the future. OED has reviewed some 20 of such recent reports and noted their candor and good analytical quality as well as responsiveness to partic- ipants' comments. EDI has also increasingly undertaken reviews of joint courses in which the comments and sugges- tions of counterpart national institutions are invited. An excellent recent example, which brought together various evaluation mechanisms, was a post-evaluation/follow-up survey mission to China which reviewed the experience with four courses,5/ all organized in the country. A question- naire was sent in advance to the participants who were then interviewed by the EDI mission. Meetings were also held with the Institutes of Finance and Economics of the Ministry of Finance which had undertaken the selection of partici- pants. The evaluation exercise helped EDI and the Chinese authorities in assessing the contribution of the courses in reZponding to China's training needs, and in drawing lessons to improve future EDI training activities in the country. 36. EDI's future plans include the expansion of its evaluation activities with the help of OED and Bank opera- tional staff. Evaluation surveys, similar to the one under- taken in China, will be made with other training institu- tions in other countries to help EDI evaluate its impact on training and institution building. An effort will be made 5/ The General Planning Project Course (GPP) of 1981; the GPP Course for Trainers of 1982; and the National Economic Management Courses of 1982 and 1983. - 15 - to assess the effectiveness of its training courses/seminars and institutional development activities in strengthening the analytical and managerial capacities of member coun- tries' development agencies. In time, regular economic and sector work may also help to assess the contribution EDI programs are making to strengthen institutional capacities in borrowing countries. 37. In response to a request from the JAC, OED has prepared an Options Paper setting out the various alterna- tives and their cost and other implications for OED involve- ment in the evaluation of EDI training and its impact. This Options Paper (JAC84-18 of July 25, 1984), which seeks to build on the existing evaluation mechanisms in EDI and proposes a four-stage OED involvement, will be considered at a forthcoming meeting of the Committee. In the meantime, EDI and OED managements have agreed to collaborate in the mid-term evaluation of its five-year program in FY87. This evaluation will also assess the effectiveness of EDI's links with the rest of the Bank. International Finance Corporation 38. In line with past practice, during the year IFC prepared and sent to OED completion reports on 11 projects, located in ten countries, covering cement, mining, pulp and paper, and other industries. The completion reports are filed in the office of the DGO for reference by interested Executive Directors. These completion reports continue to provide IFC's management with a better understanding of implementation problems as well as affording an opportunity to reassess the prospects of the projects and to use the experience as feedback in improving project preparation, appraisal, negotiation and supervision. 39. On February 10, 1984 the Joint Audit Committee considered the report entitled "An Evaluation of IFC Opera- tions in the Cement Industry" (IFC/SecM83-68). The report was prepared by IFC's Development Department, which has prime responsibility for the evaluation function, in consul- tation with the Engineering Department and with OED. It was the first evaluation of IFC activity in a manufacturing industry and was based on a series of nine project comple- tion reports covering ten completed cement projects. Besides assessing IFC's past performance in the ceLLent industry, the report attempted to draw lessons for the Corporation with respect to its developmental role and to provide practical guidance for future investments in the industry. Two important issues that emerged from the study were the variations in capital costs among cement projects - 16 - and the effects of state intervention in the cement industry. The Committee members in general commended IFC for a comprehensive evaluation of an important sector in its investment portfolio. IFC has now prepared a condensed version of this report, without confidential information on private companies, for dissemination to its operating staff and to member countries. 40. Cement industry evaluation and the earlier review of the tourism sector have established the usefulness of evaluation for IFC. In view of this, and of the benefits expected from a systematic review of its experience, a gap in the IFC structure of evaluation has now been filled with the appointment of a senior staff member in the Development Department with full-time responsibility for the function. Work has been started on an evaluation of IFC's operations in the minerals sector which represents a significant part of the Corporation's total operations and is of major consequence to a number of member countries. IFC is also developing a three-year rolling program of work for presentation to the Joint Audit Committee. Close relations with OED are being maintained and OED continues to be consulted at every stage of the evaluation process and the IFC remains responsive to OED's criteria for independent evaluation. VI. Post-Evaluation in Member Governments 41. The development of an evaluation function in the borrowing countries continues to make slow and uneven progress. The difficulties are threefold. The primary difficulty is the lack of political will to have a fully functioning and independent central evaluation function. This difficulty is compounded by a general shortage of trained investment planners and project managers leading to a widespread and not unnatural feeling that in such a situation of shortage greater priority attaches to efficient planning and execution rather than to ex post evaluation. The normal divisions within the political organization and the territorial imperatives of political power make centralized evaluation even more difficult. 42. In the above circumstances, Bank effort to promote evaluation and its feedback in the borrowing countries has been aimed at three levels. The first is the level of the project itself where the introduction of systematic monitor- ing and evaluation is not only invaluable to successful - 17 - project implementation and its longer term viability but also provides continuing feedback at the level of project management thus helping to build local capabilities. Built-in project monitoring and evaluation systems have the further potential of developing a network of trained personnel in different agencies in the borrowing countries and thus building evaluation capability from the grassroots up, as it were, in addition to demonstrating its useful- ness. Finally, project monitoring and evaluation units develop necessary data on a concurrent basis to make more comprehensive ex post evaluation possible. 43. It is in realization of the above benefits that the Bank has made considerable effort in building monitoring and evaluation functions into the projects that it supports. Starting a few years ago, the practice is now universal in respect of all new projects being approved in the agriculture and rural development fields and in educa- tion: the two people-oriented sectors which are most exposed to change and adjustment during and after project implementation and where data are hard to come by. The Bank's efforts have included workships and seminars to encourage intra-regional or intra-country comparisons of experience and to provide training in monitoring and evalua- tion (also see paras. 27 and 30 above). 44. At a second level, ex post evaluation is being encouraged at the executing agency or sector ministry level, primarily by requiring that borrowers prepare project completion reports. The effort is painfully slow, and there have been failures either when operating staff have not provided systematic and timely guidance to borrowing agencies charged with the preparation of completion reports, or when the borrower obligation to prepare such completion reports has been narrowly interpreted and the borrowers encouraged to hire outside consultants to do the work for them. This potentially eliminates the learning opportuni- ties for the borrower, makes the internalization of the lessons of experience more difficult, and, because of the additional cost incurred in the preparation of completion reports by consultants, alienates the borrowers and makes them feel that this is a job which has to be performed as an obligation to the Bank and carries no benefits for them. 45. Nevertheless, there is progress, and efforts for further development continue. Around 10% of 178 project completion reports received in OED in FY84 were prepared in full by borrowers, in addition to substantial borrower - 18 - contributions to preparing the data and other material for other completion reports. Borrower participation in project performance audit reports, by comment and discussion with visiting DED staff members and by written comments made on draft PPARs, reached 89% of all PPARs and pass-through PCRs released to the Executive Directors in FY84. In September 1983, OPS filled an important need by issuing guidelines for the preparation of project completion reports, modified and presented in booklet form for wide distribution to, and guidance of, borrower agencies. In order to engage the borrowers further in the process, the guidelines for the moment have been issued in draft form and borrower comments for improvement or modification have been invited. Similar initiatives in respect of project evaluation are under way by OED, by the Joint Inspection Unit of the United Nations and by the OECD/DAC Expert Group on Aid Evaluation (see para. 49). In the meantime EDI courses continue to include some training in project evaluation, and OED is maintaining its program of inviting staff from borrowing countries and from other aid agencies to work with OED staff members for different periods as a form of associate training. 46. Annexes A and B provide a status report on the borrower involvement in the post-evaluation of Bank-financed projects, mostly in the preparation of project completion reports. It will be noted that borrower participation in several countries is substantial, being most advanced in the Philippines, where the National Economic and Development Authority continues to exercise a centralized function responsible to the Cabinet of Ministers; Korea where, in addition to the preparation of PCRs, a centralized evalua- tion function has been lodged in the Bureau of Post Evalua- tion and Coordination in the Economic Planning Board and a systematic approach is being adopted to the further develop- went of the function, including training in coordination with OED; Indonesia, where the progress is mainly in the preparation of PCRs; in India, where, in addition to the Central Evaluation Organization of the Planning commission, a network of evaluation units are operating at the state level and extensive training is being provided. Large numbers of project completion reports are also being prepared by the borrower agencies in India. 47. A centralized evaluation function exists in Pakistan; the External Resources Division of the Ministry of Finance in Bangladesh is playing an increasing role in coordinating the preparation of project completion reports; and the Directorate General of Projects in Tunisia takes an active part in project reporting and cooperates with the Bank in evaluation. The participation of borrower agencies - 19 - in the preparation of PCRs is increasing in Tanzania and the Central Evaluation Unit in the Prime Minister's Office is coordinating and reviewing this work. There are develop- ments which could lead to centralized coordination of the preparation of project completion reports in Colombia; a central evaluation organization already exists in Yugoslavia; and is being established in the Ministry of Contraloria in Mexico. Initial contacts are being made to introduce monitoring and evaluation in investment projects in China and there are furtier plans for systematic training and the eventual development of a central evaluation function in that country. VII. The Wider Network 48. In evaluation as in externally supported and financed operations there is an imperative need for coordination by the lending agencies to minimize confusiag and overstraining borrower agencies and maximizing effec- tiveness. In recognition of this, OED not only maintains continuing contacts with the existing and potential evalua- tion functions in borrowing countries but also maintains close liaison with other aid agencies. At the practical level, joint evaluations were completed last year with Caisse Centrale of 20 years of the World Bank and Caisse Centrale support to the oil palm and coconut sector in the Ivory Coast, and with the Ministry of Economic Cooperation of the Federal Republic of Germany of jointly financed phosphate development projects in Morocco. Such joint evaluations help to align the evaluation criteria among donor agencies for the previously mentioned purpose. 49. The same purposes are served by OED membership in the OECD/DAC Expert Group on Aid Evaluation and in the Inter-Agency Evaluation Group of the Joint Inspection Unit of the United Nations. Both these groups, among their other functions, are concerned with the development of evaluation capabilities in the borrowing countries. The JIU has prepared documents aimed at promoting common criteria of evaluation within the UN system, and the DAC Expert Group on Aid Evaluation is attempting a survey of the extent to which the member countries of DAC have assisted developing countries in setting up evaluation functions. The DAC Group is also putting together a handbook which would set out the different methodologies and techniques evolved by the various DAC countries. This textbook, the JIU material and the Bank's own guidelines for the preparation of project - 20 - completion reports and for project evaluation should, in some future date, be reviewed together so as to develop a unified basis of training and guidance for developing country staff. VIII. Follow-Up 50. For the last several years, the preparation of the annual review of project performance audit results has been used as a device to follow up on actions taken to remedy recurring failures and shortcomings which are identified in these reviews. Meetings of all sector managers are convened by the Directors of OPS sector departments to review portions of the annual review relating to their sectors, to update the material in line with current practice, and to indicate the actions taken or proposed on the findings of previous reviews. These meetings and discussions have been particularly effective in the Agriculture and Rural Develop- ment Department and in the Education Department, but less so in others. The Education Department has introduced a further procedure of periodic meetings of managers in the sector to discuss the findings of PPARs/PCRs on a continuing basis in order to strengthen the process of dissemination and feedback. 51. Since 1978, this annual report has also been used as a tracking mechanism for implementation of the main recommendations of OED's special studies. Annex C of the report thus continues to follow the recommendations con- tained in studies published in earlier years. It indicates the action taken on specific recommendations before dropping them from the annex while continuing to show in the annex other recommendations to which reactions from management are awaited. The main recommendations of new special studies distributed in the preceding year, and the management's reactions to them, are added from year to year as the earlier recommendations, already acted upon, are taken out. - 21 - ANNEX A Page 1 of 8 STATUS REPORT OF THE REGIONS ON BORROWER GOVERNMENTS' INVOLVEMENT IN THE POST-EVALUATION OF BANK/IDA-FINANCED PROJECTS (All countries with five or more PCR9 scheduled through CY85) A. Eastern Africa (1) Botswana - Five PCRs in four sectors (Transport, Agriculture, Water Supply and Urban Development) are being processed through CY85. Only one PCR (Water II Project - Ln. 1763-BT) will be prepared by the Borrower, the Water Utilities Corporation, and their consultants. Active Borrower participa- tion is expected in preparing the PCRs for the Second Livestock Project (Ln. 1497-BT) and the Second Urban Project (Ln. 1584-BT). No specific procedures have been established for reviewing the ?CRs at a central point at the national level. (2) Burundi - Five PCRs are planned in this period. The Second Highway Project (Cr. 773-BU) required the Borrower to prepare the PCR. In March 1984, the Association received a draft PCR from the Government. Since it was incomplete, additional information has been requested. Two technical assistance projects (Cr. 613- and 917-BT) will have PCRs prepared by consultant, since Borrower involvement remains low because of local staffing constraints. PCRs for forestry and highway proj- ects will be prepared by Bank staff. No special review or processing arrangements exist outside of the agency or technical ministry directly con- cerned. (3) Kenya - Sixteen PCRs are being processed or are schedulea for completion by end CY1985. Seven of these are in agriculture and rural development, two are in industrial finance, and one each for SAL, technical assistance, urban development, edu- cation, water supply, transport and wildlife/ tourism. Of the sixteen PCRs only one will be completed by the Borrower (the Narok Agricultural Project - Cr. 858-KE). The Borrower was required to prepare a PCR for the Export Promotion Techni- cal Assistance Project (Cr. 1075-KE), but because - 22 - ANNEX A Page 2 of 8 of the complex issues involved and since govern- ment has few qualified staff familiar with PCR procedures and requirements, it is expected that most of the work will be done by Bank staff. There is no central focal point for preparing and processing PCRs, but the Ministry of Finance and Planning, in close consultation with the Office of the President, has overall responsibility for coordinating Bank Group financed projects. (4) Malawi - Five PCRs are expected to be produced in this period. Two are in agriculture, and one each for water supply, industrial development finance and industrial forestry. The Ministry of Agricul- ture now prepares draft PCRs for all agricultural projects and these are finalized with assistance from the Bank's Nairobi Office. There is no central focal point at the national level for processing PCRs. (5) Mauritius - Five PCRs are being processed. Three of these are for loans to the Development Bank of Mauritius. The PCR for the Power Transmission project will be prepared by the Borrower with Bank staff assistance. The Financial Secretary and the Director General of Planning have been effective focal points iin dealing with PCRs. (6) Tanzania - Ten PCRs Rre scheduled for completion by the end of CY1985. Three are in industry, two in agriculture and rural development, and one each for transport, educatAon, tourism, technical assistance and development finance. For the first time in agricultural and rural development, a PCR was completed by the Borrower (Tabora Rural Devel- opment Project - Cr. 703-TA). Most of this work was done by staff in the implementing Region. It was reviewed by the Central Evaluation Unit in the Prime Minister's Office befcre submission to the Bank. The Tanzania Sugar Corporation is taking an active part in preparing the PCR for the Tanzania Sugar Project. The Ministry of Education, the Prime Minister's Office and Auditing provided con- siderable inputs into the PCR for the Fifth Educa- tion project. B. Western Africa (1) Cameroon - Five PCRs in three sectors (three in agricultiire and enie each in education and - 23 - ANNEX A Page 3 of 8 technical assistance) are planned through CY85. The project implementation units will prepare draft PCRs for the agriculture projects but, in the light of past experience, this effort will need to be supplemented by Bank completion mis- sions with final PCRs prepared by Bank staff. The arrangements for the Education PCR are pending but role of the project implementation unit is expected to be limited to provision of basic data. The PCR for the technical assistance proj- ect will be prepared by the Resident Mission staff with inputs from beneficiary agencies. While there is no formal review process, draft PCRs/ PPARs are reviewed by the project implementation agencies and, centrally by the Ministry of Plan- ning for overall project implementation and policy issues. C. East Asia and Pacific (1) Indonesia - Twelve PCRs are scheduled for FY85, five of them will be in agriculture, three in edu- cation, three in IDF and one in the urban sector. Government agencies are expected to prepare all five PCRs on the agriculture projects; the one for the Urban II project (Ln. 1336-IND) will be pre- pared by the Bank staff; and the remaining six PCRs will be a joint product of the Government agencies and Bank staff. The degree of involvement by the Borrower in the preparation of PCRs has varied considerably among projects depending primarily on the capacity of the implementing agency. In general, the burden of preparing PCRs in draft (rather thar only assembling data) has shifted increasingly to the Borrower as the capacity of these agencies has improved. For example, while the contribution will continue to vary from agency to agency the initial experience with some of them in the agri- culture sector has been encouraging. Such involvement has been common in the power sector for some time. There has not been, however, any substantial involvement by the Borrower in the review of PCRs. (2) Korea - Preparation of five PCRs (two in the transportation sector, one each in the energy and IDF sectors, and one on the First and Second - 24 - ANNEX A Page 4 of 8 Structural Adjustment Loans) is planned in Korea for FY85. The Government agencies will prepare the PCRs for the transportation, energy and IDF projects, and the PCR on the Structural Adjustment Loans will be prepared jointly by the Government and Bank staff. The Government of Korea is developing a system for post-evaluation of major projects. During a Government reorganization in the fall of 1981, this function was assigned to a newly created Bureau of Post Evaluation and Coordination in the Economic Planning Board. This new Bureau has developed an initial work program which focuses on the evaluation of on-going projects. In March 1983, the Bureau completed a manual for post-evaluation procedures which has been distrib- uted with -guidelines for 1983" to the Ministries and Agencies of the Government who implement proj- ects. The manual is expected to enable them to prepare their own manuals of sectoral procedures. The Bureau has also been arranging training in post-evaluation theory and techniques for its staff including visits to the Bank's Operations Evaluation Department. (3) Philippines - Seven PCRs are planned for comple- tion in the Philippines in FY85: Four in the agriculture sector, and one each in the education, transportation and urban sectors. The PCR on the Third Highways project (Ln. 1373-PH) will be pre- pared as a joInt report by Bank staff and the project agency, but all the remaining six PCRs will be the responsibility of the agencies concerned. The Philippine Government has directed that all PCRs prepared by the Executing Agencies should be sent to the National Economic and Development Authority (NEDA) which reports to a Board that is a subcommittee of the Cabinet. NEDA will review the PCRs and draw therefrom lessons that are of wider relevance. Review procedures and standards have been established and the Government intends to seek OED's assistance in strengthening the post project performance review system. We expect that each forthcoming PCR would include NEDA's comments on the report. (4) Thailand - Seven PCRs are scheduled for completion in FY85 (two each in the agriculture and - 25 - ANNEX A Page 5 of 8 transportation sectors, one each in the education and energy sectors, and one on the First and Second Structural Adjustment Loans). Preparation of the PCR on the energy project (Rural Electri- fication - Ln. 1527-TH) and one transportation project (Highways VI - Ln. 1519-TH) will be the responsibility of the power entity and the High- ways Authority respectively; Bank staff will prepare the remaining five PCRs. The Government of Thailand continues to play an increasing role in the preparation of PCRs and the implementing agencies now provide much of the input for those prepared by Headquarters staff. The Government's ability to assess the impact of development programs is being strengthened under its structural adjustment program whereby perform- ance auditing as well as improved mechanisms for monitoring and evaluating development projects are being introduced. D. South Asia (1) Bangladesh - Nine PCRs in four sectors (agricul- ture, irrigation, education, and industry) are in process. Data needed for the PCRs is being collected by the agencies directly (4), or with the help of consultants (3), educational insti- tutes (1) and FAO Cooperative Program (1). The role played by the External Resources Division in coordinating the preparation effort is increasing; it will be providing the coordination in six of the ongoing PCRs. The coordination is being provided by the project authority in other cases (4). Heavy Bank assistance will be needed to complete four drafts; Bank assistance needed for the other five is considered reasonable. The quality of the PCR drafts prepared by the Borrower has generally been poor. (2) India - Eighteen PCRs in 5 sectors (agriculture, water supply, industry, education and urban) are scheduled before end CY1985. In most cases, the initial data collection and coordination within GOI is the responsibility of the project author- ity. One project completion report is being pre- pared completely by Bank and two others have required heavy Bank input. In all other cases, the amount of Bank involvement is considered rea- sonable. For PCRs which have reached the writing - 26 - ANNEX A Page 6 of 8 stage, the quality of drafts prepared by project agencies has been rated good in nine cases, and poor in six cases, by staff members responsible for the respective PCRs. (3) Pakistan - Seven PCRs in five sectors (agricul- ture, industrial imports, irrigation, education and transportation) are being prepared. The proj- ect authorities prepare the initial draft and pro- vide coordination within the government. Heavy assistance from Bank has been necessary in the preparation of all PCRs except one (ADBP IV). The quality of the PCR drafts prepared by the borrow- ers (again with the exception of ADBP IV) has also been poor. E. Europe, Middle East and North Africa (1) Tunisia - Six PCRs are scheduled for completion in four sectors (agriculture, transport, water supply and IDF). Although actual PCR finalization will be done by Bank staff, the borrowers are actively involved in each instance with input ranging from data collection to assessment of project bene- fits. In addition, Bank staff and the Directorate General of Projects in the Ministry of Planning and Finance are continuing to cooperate in the area of project evaluation with Ministry staff taking an active part in completion reporting. (2) Yugoslavia - Seven PCRs are scheduled for comple- tion in three sectors (agriculture, transport and water supply). Generally, borrowers have prepared backgrounO data and have been most cooperative in providing information for PCR preparation. In agriculture sector, the borrower has normally pre- pared at least a draft PCR and it is expected that this trend will continue. (3) Romania - Five PCRs are scheduled in 1985 covering two sectors (agriculture and power). In the agri- culture sector, an experienced borrower (BAFI) will prepare :he PCR and it is expected that a good quality product will be received. In power, the borrower will prepare a report which will form the basis for the PCRs. - 27 - ANNEX A Page 7 of 8 F. Latin America and the Caribbean (1) Brazil - Fifteen PCRs are forecast during CY1984 and CY1985. In all cases, major contributions, usually in the form of a draft report, are expected to be made by the agencies in the sector concerned. Agriculture Division B proposes to combine the preparation of the PCRs for three rural development projects in the NE of Brazil (1195-BR, 1362-BR and 1537-BR) in order to enable a more in-depth review than if three separate PCRs were prepared. Regional staff have continued to discuss with the Federal Government's Planning Institute the desirability of establishing a proj- ect evaluation function. No action has been taken thus far. (2) Colombia - Six PCRs are forecast during CY1984 and CY1985. In all cases, major contributions, usually in the form of a draft report, are expected from the executing agencies involved. The National Planning Department (DNP) has ini- tiated a project review scheme with the Bank's Resident Mission to monitor the projects on a reg- ular basis. We also understand that a similar unit, to deal only with the agricultural sector, is being established in the Planning Office of the Ministry of Agriculture. It is envisaged that this unit would participate in review of PCRs. (3) Honduras - Six PCRs are forecast during CY1984 and CY1985. Except for the Education, Agricultural Credit and Sites and Services I Projects, limited Borrower involvement in the PCR preparation is expected. (4) Jamaica - Seven PCRs are forecast for CY1984 and CY1985. Major contributions (draft report) from the executing agencies involved are expected in six of these. (5) Mexico - Six PCRs are forecasted for CY1984 and CY1985. In all cases, significant contributions from the executing agencies are expecLed in the preparation of the PCRs. There has been a con- tinued strengthening in the performance of the evaluation units of the major borrowing inter- mediary institutions. - 28 - ANNEX A Page 8 of 8 (6) Paraguay - Six PCRs are forecast for CY1984 and CY1985. The executing agencies are expected to prepare draft reports or make major contributions to the preparation of all but one of these. - 29 - ANNEX B Page 1 of 6 STATUS REPORT OF THE REGIONS ON BORROWER GOVERNMENTS' INVOLVEMENT IN THE POST-EVALUATION OF BANK/IDA-FINANCED PROJECTS (Selected countries with less than five PCRs scheduled through CY85) A. Eastern Africa (1) Ethiopia - Draft PCRs and related background materials are prepared for most projects as a matter of routine by the concerned project authorities. Bank staff provide help to finalize these reports. A high level meeting is held on each PCR, usually chaired by the Minister of the sectoral ministry concerned. UNESCO prepared the PCR for the Fourth Education Project (Cr. 553-ET) with assistance from Government. (2) Madagascar - While there is no formal requirement to have Borrower write the PCR, the Fourth Highway Project (Cr. 641-MAG) will have government pre- paring a report on project implementation. On this basis, the PCR will be written by Bank staff and consultants. Active Borrover participation is also expected in preparing the PCR for the First Power Project (Cr. 817-MAG). SODEMO, the imple- menting agency for the Morondava Irrigation and Rural Development Project, produced a good retro- spective analysis on which Bank staff completed the PCR. There is no known focal point for PCR reviews at central level in government, but this issue will be raised with government in the near future. (3) Lesotho - Some Borrower involvement is expected in preparation of a PCR for the Urban Project (Ln. 1036-LSO), but staffing constraints indicate that substantial Bank staff work will be needed. (4) Rwanda - The implementing agency has prepared a draft PCR for the Cinchona project (Cr. 656-RWA). The Borrower is required to prepare the PCR for the Fourth Highway Project (Cr. 769-RWA) and has been requested to start this work. (5) Sudan - Government was actively involved in pre- paring the PCR for the Rahad Project. - 30 - ANNEX B Page 2 of 6 (6) Zambia - A draft PCR was completed by ZESCO, Borrower for the Kafue Hydro-electric Project (Ln. 919-ZA). It is now being updated by Bank staff following an extension of the closing date. (7) Zaire - Although the Fourth Highway Project (Cr. 916-ZAI) and the Ituri I and Cotton I projects require the Borrower to prepare the PCRs, it is expected that Bank staff will actually do them. B. Western Africa (1) Benin - A PCR for a feeder roads project will be prepared by a Bank consultant on the basis of detailed cost analysis and traffic data provided by the project implementation agency. There is no formal PCR review process. The PCR will be reviewed by the Chief of the "Pistes Rurales" sec- tion in the Ministry of Works and by technical advisers in the Ministry of Planning. (2) Central African Republic - The national coordi- nator of the project will prepare the draft PCR for a technical assistance project. He will be assisted in the task by several resident experts, whose contracts are expected to be extended under a second T.A. project, and by the project admin- istrator. Although there is no formal process of reviewing the experience of Bank projects within the Government, the following authorities will undoubtedly be asked for their inputs and com- ments: High Commissioner for Planning, Minister of Agriculture, Minister of Public Works, Minister of Finance, Directrix-General of the CAADE, High Commissioner of Mines, High Commissioner of Water and Forests, and Minister of the Civil Service. All of these authorities were directly affected by the project and will be asked to give their com- ments and opinions on its implementation, results and effectiveness. (3) Congo - PCRs for two projects (education and transportation) are likely to be prepared through CY85. We expect minimal participation by the Government in the preparation of the PCR for the Education project, which will be prepared with assistance from the UNESCO Cooperative Program. The PCR will be reviewed by the Ministry of - 31 - ANNEX B Page 3 of 6 Education, in particular the directors of the Project Unit, the Education Planning Directorate, the Technical Teacher Training Institute and the Pedagogical Applied Research Institute. A first draft of the PCR for the transportation project will be prepared by the project execution agency and reviewed by the Ministry of Transport and the railway agency. There is no formal PCR/PPAR review process within the Government. (4) Ghana - Two PCRs (agriculture and import credit) are expected to be issued in CY85. The PCR for the agriculture project will be prepared by the project urit with assistance from consultants financed urder the loan and reviewed by the Minis- try of Agriculture. There is no formal PCR/PPAR review process in the agriculture sector. The PCR for the import credit will be prepared mainly by Bank staff on the basis of data and information provided by the Project Unit and Crown Agents, the procurement agent of the Borrower. The PCR will be reviewed by the Project Unit within the Bank of Ghana, the project implementation agency, and the Ministry of Finance and Economic Planning. (5) Guinea - A PCR for an education project is planned through CY85. It is expected to be prepared with the assistance of UNESCO Cooperative Program based on a preliminary report already prepared by the project unit. The report will be reviewed by the Ministries of Higher Education and Vocational Training and the Directors of the Project Unit and the Technical Training Institute. (6) Mali - Two PCRs (education, agriculture) are expected to be done by end-1985. The Government's participation in the PCR for the education project will be limited to the preparation of basic data and some background papers on key project ele- ments. The PCR is expected to be reviewed by the Ministry of Education particularly by the project unit and the directorates responsible for various project components. The PCR for the livestock project will be prepared by the project implemen- tation unit and reviewed by the Ministry of Agriculture. - 32 - ANNEX B Page 4 of 6 (7) Sierra Leone - A PCR on a technical assistance project is due in CY85. It will be prepared by Bank staff on the basis of data tr be furnished by the Ministry of Development and Economic Planning (MDEP) and the concerned sectoral ministries (agriculture, transportation, education). The PCR/PPAR will be reviewed by MDEP and other con- cerned agencies. C. East Asia and Pacific (1) Malaysia - Four PCRs are expected to be prepared in Malaysia during FY85 in four sectors: one each in the education, energy, water supply/sewerage and agriculture sectors. All these PCRs will be prepared by project agencies. The Government of Malaysia had discussed a few years ago the possibility of broadening the responsibilities of the Implementation Coordina- tion Unit (ICU) in the Prime Minister's office to include the carrying out of post-completion evalu- ations of development projects in the country. However, this has not been feasible so far as the greatly expanded development program has stressed ICU's capacity to perform even its normal tasks of monitoring project performance. The prospects of the Government assigning post completion evalua- tion of projects to one of its other agencies, such as the Economic Planning Unit, in the fore- seeable future are not particularly encouraging either because of the pervasive staff shortages besetting these agencies. The Bank is therefore continuing concen- trating its efforts in strengthening the capabili- ties of the implementing agencies to prepare PCRs of high quality. D. Europe, Middle East and North Africa (1) Cyprus - Three PCRs are scheduled in two sectors. In transport, the borrower will participate fully in data collection although Bank staff will pre- pare the final report. In agriculture, a govern- ment-prepared PCR of acceptable quality is anticipated. - 33 - ANNEX B Page 5 of 6 (2) Greece - Three PCRs will be issued in two sectors. In the agriculture sector, the borrowers will prepare the draft PCRs. In education, borrower participation has yet to be finalized but is expected to be substantial. (3) Syria - Three PCRs in two sectors are scheduled. In agriculture, one has already been drafted by government, the second will be initiated by the borrower although considerable Bank staff involve- ment is expected. In the power sector, data collection will be handled by the borrower. (4) Yemen Arab Republic - Two PCRs are expected in 1985 in agriculture and transport. In both cases, the borrowers are likely to prepare the draft PCR. (5) Morocco - Two PCRs are scheduled in agriculture and urban. Borrowers are expected to assist in data collection though they are not obligated to do so. (6) Portugal - Three PCRs are expected in three sec- tors (agriculture, education and IDF). It is unclear at this time what borrower participation will ensue although cooperation in data collection is at the minimum anticipated. (7) Oman - A transportation PCR will be issued in 1985. The borrower will participate fully in the data collection phase. (8) Israel - One PCR in the water supply sector is scheduled. The borrower has submitted a compre- hensive report on project execution which will be the basis for the PCR. (9) Turkey - Two PCRs in ..ransport and water supply are scheduled. Both will be prepared by staff but the borrowers are expected to provide background data on project performance. (10) Algeria - Two PCRs will be issued in transport and education. The borrowers are expected to collect the data necessary to start report preparation. (11) Jordan - Government will assist in the preparation of a PCR in the education sector. - 34 - ANNEX B Page 6 of 6 (12) People's Democratic Republic of Yemen - Two PCRs in two sectors are scheduled (power and education). In both cases, the borrower will pro- vide data to assist in report preparation. (13) Ejgy - Three PCRs will be issued, two in IDF and one in urban. Data will be provided by the borrower although Bank staff will be responsible for PCR preparation. E. Latin America and the Caribbean (1) Ecuador - Four PCRs are forecast during CY1984 and CY1985. Significant contributions from the exe- cuting agencies are expected for two of these. Bank staff would prepare the other two with modest assistance (information gathering, etc.) from the Borrower. - 35 - ANNEX C THE CURRENT STATUS OF RECOMMENDATIONS OF OED POLICY REVIEWS AND EVALUATION STUDIES Introduction The conclusions and recommendations of three of the reports which appeared in Annex C of the last annual report on operations evaluation have been accepted by Bank management and necessary action taken. The recommendations of these studies therefore no longer appear in this annex. The studies were: (i) Interim Report on Procurement Issues; (ii) Review of Technical Assistance to Bangladesh; and (iii) The Water Supply and Waste Disposal Program in Tunisia. Annex C-I updates and provides additional informa- tion on the status of the recommendations contained in the Operational Policy Review: Compliance with Loan Covenants, of September 1, 1982. Annexes C-II to C-IV contain the observations and recommendations of three new studies: Strengthening Agricultural Research and Extension; The World Bank Experi- ence (September 1, 1983): Harvesting the Waters; A Review of Bank Experience with Fisheries Development (March 13, 1984); and Institutional Development in Africa; A Review of Bank Project Experience (May 17, 1984). The sequence of recow- mendations set forth in the attachments is not necessarily as in the reports themselves. Findings which have emerged as of recurring concern in the performance auditing process are tracked through the process itself. The more important ones, together with some indication of feedback, appear in the Tenth Annual Review, Volume One (Summary Findings, Lessons and Feedback) and Volume Three (Follow-Up). - 36 - ANNEX C INDEX Part Report Page No. I. Operational Policy Review: Compliance with Loan Covenants 1 II. Strengthening Agricultural Research and Extension - The World Bank Experience 4 III. Harvesting the Waters - A Review of Bank Experience with Fisheries Development 11 IV. Institutional Development in Africa: A Review of Bank Project Experience 14 - 37 - ANNEX C Page 1 of 18 I. OPERATIONAL POLICY REVIEW: COMPLIANCE WITH LOAN COVENANTS (SecM82-782, September 3, 1982) The OED Report on Operational Policy Review: Compliance with Loan Covenants (Report #4090) dated September 1, 1982 was circulated to the Executive Directors on September 3, 1982 (SecM82-782). The Joint Audit Committee considered the OED Report on September 22, 1982 (JAC/M82-8) and the Management Response thereto (R83-2 dated January 5, 1983) on January 26, 1983 (JAC/M83-2). The OED Report and the Management response were considered at the Board Meeting of July 14, 1983. The principal recommendations contained in the above report were set forth in last year's Annual Report (Annex C, Item IV) together with the Management Response thereto. Since then the Management of the Bank has given extensive consideration to this subject which culminated in the formation of a Legal Department Task Force on "Simplification of Loan Documents." The Report of the Task Force (with one minor exception) has been fully supported by Bank Management and has been considered by the Managing Committee of the Bank. The proposals will in due course be translated into instructions to staff and a timetable set for implementing the recommendations. Five major issues were addressed: 1. Procurement Provisions. It was agreed that the procurement schedule included in the loan documents should contain only specific provisions relating to the project concerned, and that all the standardized clauses containing general rules and procedures for procurement should be removed to the Guidelines for Procurement. The Bank will pursue its current efforts through procurement seminars and other means to instruct borrower officials concerned as to the specifics of the Guidelines and the importance of following the Bank's rules. 2. Other Standardized Provisions. Management decided to remove from the loan documents and into the General Conditions standard clauses governing such matters as negative pledge, - 38 - ANNEX C Page 2 of 18 insurance, use of goods and services, plans and schedules, records and reports, inspec- tion of facilities and documents, publication of awards of contracts, project completion reports, and other matters listed in the Task Force Report. Loan documents should, to the extent possible, contain clauses specific to the operation. The need to instruct officials of the borrowers and familiarize them with the provisions of the General Conditions was stressed. 3. Letters of Intent. The Task Force had suggested a more selective approach to the use of covenants by excluding from the loan documents general statements of future policy which would more appropriately serve as the subject of a dialogue in the context of ongoing relationships between the Bank and the borrower. A statement of intention by the borrower should replace covenants that are directional in nature and that do not contain definite undertakings which can be monitored and enforced. Very often the undertaking is framed in such a way as to militate against proper monitoring of covenant observance and to render enforcement impossible. In such cases it would be preferable to obtain a letter of intent outside the legal documents framework which describe the overall policy objectives and the directions towards their realization. Specific actions that are definable and monitorable should still be included as covenants. Letters of Intent would go to the Loan Committee, and in certain cases also to the Board. 4. A Consolidated Plan of Action. Detailed steps of implementation as well as admin- istrative and organizational arrangements should be embodied in an agreed Plan of Action, describing a program of implementa- tion, with time-bound steps. This would obviate the use of many separate minor covenants regulating implementation. Remedies could still be applied in case of violation of specific undertakings. - 39 - ANNEX C Page 3 of 18 5. Role of Lawyers. Legal Department lawyers should participate closely in drafting project descriptions and project implementa- tion arrangements for the benefit of clarity and consistency of Appraisal and President's Reports with the legal documents. The Board should be informed in due course about the intended change, at least with respect to changes in the General Conditions, which require Board approval. A schedule for implementing the recommendations will have to be set and advice given to staff on how to proceed. - 40 - ANNEX C Page 4 of 18 II. STRENGTHE4ING AGRICULTURAL RESEARCH AND EXTENSION - THE WORLD BANK EXPERIENCE (SecM83-879, September 1, 1983) The OED Report on Strengthening Agricultural Research and Extension - The World Bank Experience (Report #4684) dated September 1, 1983 (SecK83-879) was considered by the Joint Audit Committee on December 2, 1983 (JAC/ M83-11). The Report covers experience with- agricultural research and extension in ten member countries of the Bank. A. Summary of Major Findings 1. Bank efforts in support of agricultural research and extension (R&E) were significant and worth- while even though the efficiency of the approaches followed in &ddressing various countries' specific needs was variable. The Bank's overall experience with R&E has been positive. The lessons learned about the strengths and weaknesses in the approaches selected for supporting R&E makes the Bank today much better able to help member coun- tries improve the efficiency of their R&E reorga- nization than it was ten years ago. 2. R&E experience in ten member countries examined for this study shows three different kinds of approaches to the Bank's involvement in agricul- ture: to (i) support for research and/or exten- sion components in the context of Bank-financed agricultural and rural development projects; (ii) the same as (i) except that the national R&E organization were partly or fully associated with implementing the component; and (iii) full-scale research and/or extension projects. The review found that, regardless of the approach, the level of R&E services needed to achieve the study countries' development targets far exceeded the level of R&E services they actually provided. 3. The R&E function was often hindered by the coun- tries' difficulties in formulating and implement- ing the necessary policy action. The study also found marked inadequacies among resource alloca- tion to and among research and extension, - 41 - ANNEX C Page 5 of 18 reflecting weaknesses in the planning and monitoring process in these countries. With reference to research, the Bank was generally perceived as having had a positive impact on resource allocation to and within research systems. The Bank concentrated its support on applied research and promoted the establishment of monitoring systems. The Bank support for extension components in agriculture has helped meet urgent and realistic needs and has increased awareness in member countries of the potential benefits of technology transfer functions. 4. Regardless of the approach followed, the review found that Bank support for R&E has been con- strained by five factors: (i) lack of country objectives for agriculture; (ii) limited country input in the design of R&E components or projects along with unclear connections between R&E activi- ties supported by the Bank and other development activities in the sector; (iii) constrained Bank resources for helping countries address sector or economy-wide issues affecting R&E; (iv) institu- tional separation of research and extension and (v) the lack of clarity in or agreement on the definition of various stages in the process of technology development and transfer. The review found that, depending on which approach is selected for providing support for R&E, the Bank has succeeded to varying degrees in increasing awareness among policy makers of the need to address these constraints and in some cases in helping to resolve some of them. 5. The study outlines some of the common character- istics displayed for the more successful research and extension organizations in the study coun- tries: a. The more successful research organizations in the study countries displayed a combination of some of the following characteristics: (i) a strong central organization with over- all responsibility for most aspects of national research and funding control to back it up; - 42 - ANNEX C Page 6 of 18 (ii) some degree of autonomy from the routine bureaucratic structure of the Ministry of Agriculture; (iii) good links to the national planners and policy makers despite the semi- autonomous or autonomous status; (iv) service units either directly part of or closely related to the central research organization, which perform important planning, monitoring and evaluation functions; (v) planned decentralization of regional research; and (vi) active participation in translating research results into recommendations communicable to users. b. The more successful extension organizations in the study countries displayed a combina- tion of some of the following character- istics: (i) flexible extension methodology to fit the nature of the technology being extended, clients' needs, and the means available to the country; (ii) close coordination of the extension function with other technology transfer functions; (iii) active participation by extension personnel (i.e., subj.ct matter specialist), along with researchers, in translating research results into recon- mendations communicable to users; (iv) a formal feedback mechanism which reflects directly clients' points of view rather than observers' opinions of what those points of view are or should be; - 43 - ANNEX C Page 7 of 18 (v) clearly defined job descriptions for extension personnel, who are supervised and evaluated on the basis of these job specifications; and (vi) adequate attention to both the technical and communication/education skill requirements of extension personnel vis-a-vis the technology to be extended and the client group(s) to be served. B. Summary of Principal Recommendations 1. Design of Bank Support for R&E - The Bank experi- ence has been positive and the justification for continued support for R&E to build on the lessons of this experience is solid. These recommenda- tions are intended to sharpen the Bank's focus and to serve as a frame of reference for future design of Bank support for R&E. a. In countries with limited capability for project preparation, the Bank could improve the relevance of its support for R&E by: (i) more in-depth assessment of the socio- political and cultural constraints in order to design projects more consistent with the realisties in the country; (ii) more insistence on borrowers' active participation in the preparation and appraisal of R&E projects or components; and (iii) less emphasis on generalized approaches to supporting R&E or on specific R&E techniques: although Bank staff may well understand what the Bank intends by concepts such as applied research, R&E to benefit target groups, appropriate technology, training and visit, appro- priate organization, etc., R&E admin- istrators and consultants to governments eager to identify "acceptable" projects for Bank support may construe these concepts in ways that could jeopardize their potential utility to the country. - 44 - ANNEX C Page 8 of 18 b. The design of a R&E should start with a clear definition of the scope of the function(s) to be supported and an assessment of how it (they) would relate to other functions in the national technology transfer system. c. An assessment of a country's ongoing R&E efforts and its goals in the agricultural sector should precede the specification of Bank support to R&E in that country. d. Support for technology transfer extension should more systematically address how exten- sion and input supply and services should be balanced to best meet the needs of the target population. e. Little is known in the field of institution building that has been proven to work uader the varied circumstances of the member coun- tries. Thus the Bank needs to monitor closely the viability and sustainability of the training and visit technique of extension it has been disseminating so widely to the member countries. f. Support for extension should be designed to reflect the state of research-in the country. g. Support for R&E should reflect the heteroge- neity in the level of development of various sections of the agricultural sector in a given country and should be designed with enough flexibility to meet the varying demands of such sections. h. Support for national research should be spe- cifically designed to take maximum advantage of the work of International Research Centers. 2. Emphasis in Bank Support for R&E a. Research projects should emphasize: (i) manpower training, and the training of research program leaders in particular; - 45 - ANNEX C Page 9 of 18 (ii) development of the functions of research management and research station manage- ment; (iii) research on agricultural activities in low potential areas; (iv) research aimed at the integration of crop and livestock activities and farm management research; and (v) consideration of socio-economic, environmental and political factors in the design of research programs and the evaluation of their results. b. Extension projects should emphasize: (i) formal training for higher level exten- sion personnel in technical agricultural subjects as well as in communications; (ii) job descriptions for extension personnel which would be consistent with the country's administrative procedures and the existing reward system; (iii) relating training in the extension service to: - the country's overall demand for skilled manpower and its training capacity, and to its capacity to pay for extension both during the period of Bank support and afterwards; - the differences in the levels of sophistication of technology being utilized by various groups of farmers, and - the potential for trade-off between the use of manpower and communication media for extension; (iv) formalizing the professional links between research and extension; and - 46 - ANNEX C Page 10 of 18 (v) the establishment of channels for farmers, especially the less privileged among them, in particular, to influence the form and the substance of extension directly in addition or as an alterna- tive to feedback through the field extension agent. 3. Bank Supervision - Bank supervision of its support for R&E (i) could be improved through a greater input in monitoring the implementation of R&E components and projects; (ii) should go beyond the physical implementation of the project to cover the quality of research programs and institutional changes being introduced; and (iii) should closely monitor organizational changes and periodically reassess their usefulness to ensure that the agencies affected continue to work in harmony with other national institutes. C. Management Response A major impact study of the activities of the Consultative Group on International Agricultural Research (CGIAR) is being undertaken by a team of outsiders and is expected to be completed shortly. Management, and OED, advised against any specific action at this time pending completion of the impact study. - 47 - ANNEX C Page 11 of 18 III. HARVESTING THE WATERS - A REVIEW OF BANK EXPERIENCE WITH FISHERIES DEVELOPMENT (SecM84-240, March 19, 1984) Harvesting the Waters -- A Review of Bank Experience with Fisheries Development (Report #4984 dated March 13, 1984) was circulated on March 19, 1984 (SecM84- 240) and was considered by the Joint Audit Committee on March 28, 1984 (JAC/M84-3). The following is a summary of the Report's find- ings and recommendations: A. Summary of Findings: (1) The overall performance of the fisheries projects portfolio in the Bank has not been good and virtually all the problems encoun- tered can be traced back to: technical inadequacies, administrative and managerial weaknesses in implementing agencies, lack of a cohesive Bank approach to the sector, coupled with a shortage of in-house fisheries staff. As for fisheries components in other projects, with few exceptions, they have received scant attention and have not per- formed well. (2) The Bank's continued involvement in fisheries is only justified if the Bank is prepared to accord the same level of professional consid- eration to the formulation and supervision of fisheries lending as it does to the other sectors. (3) A more cohesive and forward looking program of subsector review and project identifica- tion would need to be developed in collabora- tion with FAO and other international and bilateral agencies if future fisheries opera- tions are to be more significant. (4) Future lending should place more emphasis on upgrading national fisheries research capabilities and/or the creation in national fisheries administration of units capable of - 48 - ANNEX C Page 12 of 18 converting research results into effective and enforceable fisheries management systems. (5) There is the need for improviug the reliabil- ity of national fisheries statistics which is essential for effective establishment of fisheries management. B. Summary of Recommendations: Specific recommendations are as follows: (1) The Bank should develop a cohesive subsector review program in support of its future involvement in fisheries operations and a firm decision regarding the scope of these activities is required. (2) All proposal for fisheries lending, whether in the form of main projects or components, should be referred to the Fisheries Develop- ment Advisor in AGR for review and advice, irrespective of size, cost or initiating sector. (3) As most proposals for irrigation, drainage or hydro-electric developments have implicit or explicit effects on fisheries development, they would also need to be referred to the Fisheries Development Advisor, whether or not they contain a fisheries component. (4) Supervision missions responsible for projects containing fisheries components should make specific reference to the component's progress in their reports; such missions should also include a fisheries expert at reasonable intervals. (5) The Fisheries Advisor should ensure that all in-house expertise relevant to fisheries projects is tapped as appropriate. (6) The Fisheries Advisor should also ensure that technical input by FAO Fisheries Department staff, in project design and preparation in particular, is arranged as appropriate. - 49 - ANNEX C Page 13 of 18 (7) Preparation and appraisal missions should evaluate the status of fish resources in project areas and the reliability of local fisheries statistics, and in cases where there is a lack of knowledge, more specific provisions for resource survey work, explora- tory fishing trials and improvement of statistical systems should be provided for. (8) Preparation and appraisal missions should give particular attention to institution building of national fisheries agencies and to making provisions for technical assis- tance, importing expertise and training of local staff as may be necessary. Such institution building objectives should not be limited to the execution of the project but should include building viable institutions for the management of the sub-sector as a whole. (9) Projects proposing use of fishing vessels different from types already in use in project areas should make a clear case for deviation in design; new designs should only be proposed on the basis of thorough and professional study of local conditions and users' preferences, to minimize the risk of non-acceptability. Ideally, these studies should take place under a Project Preparation Facility and result in pilot testing before the project is appraised. C. Management Response Management agreed with the highlights of the report and regarded it as adequately covering the main issues and problems in the Bank's fisheries projects. The Bank's basic strategy for future sector lending was set out in the 1982 Fishery Sector Policy Paper which would permit a flexible approach to supporting this sector in the future. - 50 - ANNEX C Page 14 of 18 IV. INSTITUTIONAL DEVELOPMENT IN AFRICA: A REVIEW OF BANK PROJECT EXPERIENCE (SecM84-457, May 22, 1984) Institutional Development in Africa: A Review of Bank Project Experience (Report No. 5085) dated May 17, 1984 was distributed on May 22, 1984 (SecM84-457) and was consi- dered by Joint Audit Committee on June 6, 1984 (JAC/M84-5). A management response for consideration by the Board is under preparation. This report is the first special study undertaken by OED to evaluate the Bank's experience in supporting institutional development (ID) in its member countries. In view of the priority attached to Sub-Saharan Africa in the Bank's present operations, and the importance of institu- tional constraints to development in Africa, the present review is confined to this region. The review attempts to assess the effectiveness and impact of institutional support measures attached to Bank-financed projects in two sectors, agriculture and transport, which have received most assis- tance from the Bank in the past. It does not attempt to assess the Bank's experience with non-project lending or the support given under country economic and sector work. The report recognizes the severity of institu- tional constraints in Africa and that ID is a long-term process that will require sustained effort and commitment by the countries themselves, as well as much external assis- tance, to succeed. However, the report does not set out to analyze the institutional constraints facing African coun- tries or to suggest appropriate actions; these have been well described in other reports. Instead, it focuses on the role of the Bank in promoting ID through its project lending, assesses how effective it has been in the past, and suggests what steps might be taken to make it more effective in the future. Summary Findings On the basis of an extensive review of evidence, the report concludes that, while there have been some notable successes, the Bank's past efforts supporting ID in Africa through agricultural and transport projects have generally not been very effective. A considerable part of the report is devoted to an examination of the apparent - 51 - ANNEX C Page 15 of 18 reasons for the success or failure of the Bank's project- related support for ID in the past. It concludes from the lessons of experience that five factors have affected success: (i) The projecz itself has some limitations as a vehicle for ID-particularly in its rela- tively short time horizon, the tendency for investment goals to divert attention from instit-tional concern,s and the difficulty of addressing country-wide institutional con- straints under a single project. (ii) The ID objectives of projects have not always been clearly defined or their ID components well designed and appropriate. In partic- ular, it has often been unclear whether technical assistance was intended to promote ID or included merely to help implement the project. (iii) the ID components of projects have tended to be less well prepared and appraised than the investment components. (iv) Successful implementation of ID components typically requires the borrower, the Bank and technical assistance personnel all to perform well; yet all these face particular problems in supporting the ID process. (v) Although ID is primarily the responsibility of the borrower, the Bank has frequently taken the initiative in ID and the borrower has not always been convinced of the need or prepared to provide the full support required for success. The report then goes on to evaluate the underlying process in the Bank, with a view to identifying areas for improvement. While recognizing that the environment for ID in Africa is unusually harsh, and may sometimes frustrate even the best efforts. The report suggests that three shortcomings in the regions' past performance have put a brake on effectiveness: (i) Institutional Analysis. The regions have often given insufficient attention to analyzing the strengths and weaknesses of target institutions and the environment as a - 52 - ANNEX C Page 16 of 18 whole. As a result, some opportunities for effective ID have been missed and some inappropriate measures included in projects. (ii) Planning of ID Work. In most cases, the Bank has not evolved an articulate strategy, or agreed long-term program, for developing the institutions involved. Until recencly, it has also given limited attention to the wider policy and institutional constraints that could not be tackled through projects. (iii) Processing PSID Measures. The review suggests that a lower standard of pre?aration and appraisal has often been applied to PSID measures than to the physical components of projects, and that many PSID measures have received little attention, or have even been ignored, during supervision. Conclusions and Recommendations The conclusions and recommendations of the report relate primarily to the Africa regions' efforts to suppo-t ID in past projects in only two sectors. It recognizes that, in some respects, performance in more recent projects has much improved, and that past efforts in some other sectors appear to have been more effective. Nevertheless, the report concludes that the undelying constraints that affected the Bank's performance in Africa still persist and that many of these, by reasonable inference, are of widexr relevance within the Bank. The report concludes, given the urgency of the need, that ID should be seen as central to the Bank's role as a development agency in Africa. Moreover, from a wider perspective, it points out that a progressive improvement in borrower capacity is also essential for greater cost efficiency in the transfer of resources, to ensure the sustainability of completed projects and to help develop a sound policy framework in member countries. The report therefore recommends that ID should be a major objective of Bank-financed projects in Sub-Saharan Africa, and that sufficient resources must be allocated to make an effective contribution. The report concludes that the Africa regions face a major task in responding to the challenge of ID in Africa, and accordingly recommends that management should draw up a - 53 - ANNEX C Page 17 of 18 phased action plan for expanding and upgrading the Bank's ID work in Africa. The report takes the view that it is not appropriate for OED to make detailed recommendations, but outlines five apparent problem areas which might be addressed in the proposed action program, and suggests some possible options for management consideration. The five priorities for action are: (i) Increasing Knowledge. The review suggests the need to expand the Bank's knowledge of countries and institutions, to develop the state of the art in ID, and to accelerate the dissemination of appropriate techniques and approach. (ii) Regional Capacity. It suggests that manage- ment should ensure that sufficient staff input will be available before undertaking any substantive ID work. To enhance staff capability it also suggests a more systematic effort to utilize staff with relevant skills and experience in operational posts, a major effort to provide operational staff with relevant training and guidelines for ID work, and the appointment of more specialist ID staff. The report also suggests that organi- zational changes will be required to enable the two Africa regions to undertake effective ID work on the scale and intensity required. (iii) Improved Procedures. The report emphasizes the need to upgrade the quality of ID work at all stages of the project cycle. In partic- ular, improved procedures are required to ensure that the priorities between the ID and other objectives of Bank assistance are clearly established, that ID components of projects receive the same care and attention in preparation, appraisal and supervision as t-a investment components of projects, and that the borrower is involved more fully in the analysis and diagnosis of institutional problems and the preparation and implementa- tion of project-supported ID measures. (iv) Management. The report considers that regional management should focus more sharply on the institutional aspects of projects, - 54 - ANNEX C Page 18 of 18 ensure that sufficient time and resources are allocated to them, and provide appropriate signals and incentives to operational staff to upgrade ID work. (v) Financing. Despite its limitations, the investment project is likely to remain an important vehicle for supporting ID, and the report concludes that there is much scope improvement. - 55 - ANNEX D Page 1 of 11 OED REPORTS ISSUED DURING FY84 Project Performance Audit Reports Public Utilities Zambia Kariba North Hydroelectric (Loan 701-ZA) SecM83-825 08/12/83 Turkey Istanbul Water Supply (Loan 844-TU) SecM84-3 12/27/83 Indonesia Third and Fourth Power SecM84-518 05/30/84 (Credit 399/Loan 1127-IND) Egypt Regional Electrification (Loan 1453-ECT) SecM84-557 06/05/84 Yugoslavia First Power Transmission (Loan 836-YU) SecM84-647 06/06/84 Nicaragua Eighth Power and Earthquake Reconstruction SecM84-611 06/19/84 (Credit 389/Loan 840-NI) Panama Water Supply and Sewerage SecM84-613 06/21/84 (Loan 1280-PAN) Algeria First and Second Power (Loans 997/1293-AL) SecM84-642 06/29/84 Transportation, Tourism and Urban Development Colombia Sixth Railway (Loan 926-CO) SecM83-832 08/22/83 Zaire River Transport, Rail/River Transport, SecM83-963 10/07/83 Second and Third Highways (Credits 255/571/292/536-ZR) Philippines Second Highway (Loan 950-PH) SecM83-1024 10/25/83 Senegal Feeder Roads (Loan 1221-T-SE) SecM83-1128 12/07/83 Mauritania Nouadhibou Port (Credit 588-MAU) SecM83-1181 12/19/83 Turkey Istanbul Urban Development (Credit 324-TU) -SecM84-36 -12/30/83 Philippines First Shipping (Loan 1048-PH) SecM84-103 01/31/84 Korea Third Highway (Loan 1203-KO) SecM84-291 04/04/84 - 56 - ANNEX D Page 2 of 11 Transportation, Tourism and Urban Development - continued Senegal Second and Third Highways SecM84-474 05/11/84 (Credit 366/Loan 1222-SE) Yugoslavia Sixth, Seventh, Eighth and Ninth Highways SecM84-555 06/07/84 (Loans 990/1143/1377/1535-YU) Ghana First and Second Highways SecM84-628 06/27/84 (Credit 438/Loan 1182/Credit 594-GH) Jamaica Second Highway (Road Improvement and SecM84-634 06/29/84 Maintenance) and Third Highway (Loans 899/1032-JM) Malawi Second and Third Highways SecM84-617 06/29/84 (Credits 523/758-MAI) DFC, Industry and Non-Project Lending Pakistan Industrial Credit and Investment SecM83-1148 12/02/83 Corporation Limited (PICIC) (Loans 961/1326-PAK) Uruguay Industrial Development and Export Expansion SecM84-17 12/30/83 (Loan 1176-UR) Kenya First Structural Adjustment Credit SecM84-135 02/10/84 (Credit 999-KE) Thailand Industrial Finance Corporation (IFCT) Secm84-417 04/24/84 (Loans 992/1327-TH) Yugoslavia First Industrial Credit SecM84-556 06/11/84 (Loans 1012/1013-YU) Zimbabwe Manufacturing Rehabilitation Imports Program SecM84-561 06/15/84 (Loan 1959/Credit 1120-ZIM) Bolivia Structural Adjustment Loan SecM84-599 06/25/84 (Loan 1865-BO) India First Industrial Development Bank (IDBI/SFC) SecM84-603 06/27/84 (Credit 356-IN) Morocco Maroc Phosphore and Expansion SecM84-633 06/29/84 (Loans 1017/1625-MOR) - 57 - ANNEX D Page 3 of 11 Agriculture Bangladesh First and Second Fertilizer Import Credits SecM83-934 10/06/83 (Credits 944/1044-BD) Dominican Republic Yaque del Norte Irrigation SecM83-986 10/20/83 (Credit 352-DO) Philippines Second Livestock (Loan 1225-PH) SecM83-987 10/21/83 Nigeria Bendel State Oil Palm and Ondo State Oil Palm SecM83-1059 11/15/83 (Loans 1183/1192-UNI) Thailand Rubber Replanting (Loan 1243-TH) SecM83-1081 11/18/83 Burma Forestry (Credit 493-BA) SecM83-1149 11/28/83 Thailand Livestock (Loan 1199-TH) SecM84-78 12/21/83 Nicaragua Agricultural and Industrial Rehabilitation SecM84-2 12/28/83 (Loan 1785/Credit 966-NI) Tanzania Kigoma Rural Development (Credit 508-TA) SecM84-19 12/28/83 Cyprus Paphos Irrigation (Loan 960-CY) SecM84-53 12/30/83 Congo Livestock (Credit 435-COB) SecM84-77 12/30/83 People's Democratic Republic of Yemen Wadi Hadramawt Sec84-192 02/16/84 Agricultural Credit (Credit 615-YDR) Sri Lanka Tree Crop Diversification (Tea) SecM84-315 04/13/84 (Credit 819-CE) Sri Lanka Dairy Development (Credit 504-CE) SecM84-430 04/16/84 Bangladesh Cereal Seeds (Credit 410-BD) SecM84-411 04/23/84 India Karnataka Wholesale Markets (Credit 378-IN) SecM84-536 05/14/84 Yemen Arab Republic Grain Storage and Processing SecM84-429 04/26/84 (Credit 636-YAR) Yemen Arab Republic Southern Region Uplands SecM84-463 04/26/84 Rural Development (Credit 545-YAR) Bangladesh Extension and Research (Credit 729-BD) SecM84-584 05/08/84 Bangladesh Coastal Area Rehabilitation and SecM84-567 05/15/84 Cyclone Protection (Credit 339-BD) - 58 - ANNEX D Page 4 of 11 Agriculture - continued Greece Groundwater Development (Loan 754-GR) SecM84-526 05/24/84 Yugoslavia First Agricultural Credit (Loan 1129-YU) SecM84-614 05/29/84 Ghana Oil Palm Development (Loan 531-GH) SecM84-578 05/31/84 India Madhya Pradesh Forestry Technical Assistance SecH84-537 06/06/84 (Credit 609-IN) Sudan Rahad Irrigation (Credit 364-SU) SecM84-559 06/13/84 Morocco Souss Groundwater Development SecM84-657 06/15/84 (Loan 1123-MOR) Cameroon Cocoa (Loan 1039-CM) SecM84-658 06/18/84 Togo Maritime Region Rural Development (Credit 638-TO) SecM84-586 06/18/84 and Rural Development in Cotton Areas (741-TO) Cameroon Second Semry Rice (Loan 1512/Credit 763-CM) SecM84-600 06/25/84 Indonesia Agricultural Research and Extension and SecM84-612 06/25/84 National Food Crops Extension (Loans 1179/1267-IND) Indonesia First Transmigration and Rural Development SecM84-598 06/25/84 (Loan 1318-IND) Pakistan Fertilizer Imports Credit SecM84-631 06/26/84 (Credit 1066-PAK) Arab Republic of Egypt-Upper Egypt First Drainage SecM84-659 06/27/84 (Credit 393-EGT) Senegal Sine Saloum Agricultural Development SecM84-644 06/29/84 (Credit 549/Loan 1113-SE) Ivory Coast Second Cocoa (Loan 1069-IVC) SecM84-618 06/29/84 Mexico - Rio Panuco Irrigation (Loan 969-ME) SecM84-645 06/29/84 - Rio Sinaloa Irrigation (Loan 970-ME) Tanzania - Geita Cotton (Credit 454-TA) SecM84-662 06/29/84 - National Maize (Credit 606-TA) - Tabora Rural Development (Credit 703-TA) - Tobacco Processing (Credit 658-TA) - Tobacco Handling (Credit 802-TA) - Cashewnut Development (Loan 1014-TA) - 59 - ANNEX D Page 5 of 11 Education Paraguay First Education (Credit 347-PA) SecM83-1185 12/23/83 Mauritania First Education (Credit 459-MAU) SccM84-35 12/30/83 Zaire First Education (Credit 272-ZR) SecH84-52 12/30/83 Botswana First Education (Loans 1274/1275-T-BT) SecM84-54 12/30/83 Central African Republic First Education SecM84-64 12/30/83 (Credits 308/1073-CAR) Peru First Education (Loan 949-PE) SecM84-96 12/30/83 Swaziland First Education (Credit 518-SW) SecM84-239 03/09/84 Swaziland Second Education (Loan 1478-SW) SecM84-248 03/16/84 Arab Republic of Egypt First Education SecM84-484 05/07/84 (Credit 681-EGT) Yemen Arab Republic First Education SecM84-485 05/09/84 (Credit 421-YAR) Yemen Arab Republic Second Education SecM84-464 05/18/84 (Credit 611-YAR) People's Democratic Republic of Yemen SecM84-558 06/11/84 First Education (Credit 494-YDR) Papua New Guinea First Education (Credit 661-PNG) SecM84-597 06/15/84 Gambia First Education (Credit 792-GM) SecMB4-562 06/19/84 Benin First Education (Credit 583-BEN) SecM84-663 06/29/84 Nigeria Third Education (Loan 929-UNI) SecM84-643 06/29/84 Population Tunisia First Population (Credit 238-TUN) SecM84-648 06/21/84 - 60 - ANNEX D Page 6 of 11 PASS-THROUGH PROJECT COMPLETION REPORTS Public Utilities Niger Telecommunications (Credit 634-NIR) SecM83-1018 10/27/83 Egypt Water Supply Engineering and SecM83-1126 12/05/83 Technical Assistance (Credit S-20-EGT) Syria First and Second Damacus Water Supply SecM83-1164 12/14/83 (Credit 401/Loan 1241-SYR) Bahamas Water Supply and Sewerage Project for SecM83-1180 12/22/83 New Providence Island (Loan 1320-BM) Philippines Sixth Power (Loan 1034-PH) SecM83-1184 12/22/83 Tunisia First Urban Sewerage (Loan 1088-TUN) SecM84-4 12/22/83 Malawi Nkula Falls II Hydroelectric SecM84-18 12/28/83 (Credit 691/Loans 1387-T/1388-MAI) Costa Rica Fifth Power (Loan 1126-CR) SecM84-232 03/15/84 Honduras Fifth and Sixth Power (Loans 841/1081-HO) SecM84-441 04/24/84 East Africa Third Telecommunications (Loan 914-EA) SecM84-576 06/11/84 Morocco Sewerage Engineering (Loan S-007-MOR) SecM84-602 06/14/84 Brazil Power Distribution (COPEL) (Loan 1257-BR) SecM84-632 06/27/84 Morocco Sidi Cheho Al-Massira Hydroelectric SecM84-616 06/28/84 (Loan 1299-MOR) Jordan Third Power (Loan 1688-JO) SecM84-650 06/28/84 DFC, Industry and Non-Project Lending Thailand Lat Krabang Industrial Estate SecM83-1080 11/17/83 (Loan 149 2-TH) Indonesia Fourth Technical Assistance SecM83-1093 11/18/83 (Credit 451-IND) Malawi Transport Engineering and Services Sec83-1110 11/18/83 (Credit S-7-KAI) - 61 - ANNEX D Page 7 of 11 DFC, Industry and Non-Project Lending - continued Swaziland National Industrial Development SecM83-1129 11/18/83 Corporation (NIDC) (Loan 1409-SWA) Romania Post Earthquake Construction Assistance SecM83-1094 11/18/83 (Loan 1581-RO) Ireland Second Loan to the Industrial SecM83-1140 12/02/83 Credit Company Limited (Loan 1160-IRE) Sudan Technical Assistance (Credit 614-SU) SecM83-1147 12/06/83 Algeria Industrial Credit Project to Banque SecM83-1165 12/16/83 Algerienne de Developpement (Loan 1158-AL) Rwanda Banque Rwandaise de Developpement (BRD) SecM84-123 02/10/84 (Credit 655-RW) Turkey First Dev.et Yatirim Bankasi (Loan 1024-TU) SecM84-124 02/10/84 Lesotho - Lesotho National Development Corporation SecM84-231 03/14/84 (LNDC) and Basotho Enterprises Development Corporation (BEDCO) (Credit 720-LSO) Turkey - Turkiye Sinai Kalkinma Bankasi A.S. (TSKB) SecM84-251 03/14/84 (Loans 1078/1430-TU) Algeria SNMC Expansion (Loan 1185-AL) SecM84-241 03/19/84 Cameroon Small and Medium-Scale Enterprise SecM84-314 04/12/84 (Credit 575-CM) Tunisia GAFSA Phosphate (Loan 1042-TUN) SecM84-615 06/15/84 Cyprus Development Bank (Loan 1744-CY) SecM84-651 06/12/81 Tanzania Mwanza Textile (Loan 1128-TA) SecM84-687 06/29/84 Transportation, Tourism and Urban Development Senegal Sites and Services (Credit 336-SE) SecM83-1025 10/31/83 Papua New Guinea Second Port (Loan 1551-PNG) SecM83-1186 12/08/83 Ecuador Second Guayaquil Port (Loan 1255-EC) SecM84-20 12/13/83 - 62 - ANNEX D Page 8 of 11 Transportation, Tourism and Urban Development - continued Federal Islamic Republic of the Comoros First Highway SecM84-57 12/30/83 (Credit 852-COM) Mexico Regional Airports Development (Loan 1022-ME) SecM84-65 12/30/83 Liberia Third Highway (Loan 1156-LBR) SecM84-104 01/23/84 Togo Third Highway (Credit 693-TO) SecM84-87 01/23/84 Spain Second Port (Loan 884-SP) SecM84-148 02/08/84 Yugoslavia Montenegro Earthquake Rehabilitation - SecM84-149 02/13/84 Highways (Loan 1759-YU) Cameroon Second Douala Port SecM84-163 02/14/84 (Loan 1321/Credit 657-CM) Tanzania First National Sites and Services SecM84-230 02/15/84 (Credit 495-TA) Haiti Fifth Transport (Credit 807-HA) SecM84-238 03/12/84 Tunisia Tunis District Urban Planning and SecM84-287 03/27/84 Public Transport (Loan 937/Credit 432-TUN) Central African Republic Third Highway SecM84-313 04/16/84 (Credits 847/1074-CA-Supplement) Honduras Seventh Highway (Loan 1341/1342-T-HO) SecM84-666 05/14/84 Korea Sixth Railway (Loan 1542-KO) SecM84-585 05/22/84 Gambia Infrastructure and Tourism (Credit 602-GM) SecM84-517 05/29/84 Burma Second Rangoon Port (Credit 671-BA) SecM84-516 05/30/84 Uganda Second Highway (Credit 164-UG) SecM84-560 06/14/84 Panama Highway Maintenance (Loan 1565-PAN) SecM84-575 06/22/84 Mexico Seventh Highway (Loan 968-ME) SecM84-579 06/26/84 Niger Third Highway (Credit 612-NIR) SecM84-649 06/28/84 Ethiopia Fifth and Sixth Highways SecM84-668 06/28/84 (Credits 332/552-ET) - 63 - ANNEX D Page 9 of 11 Agriculture Indonesia Smallholder and Private Estate Tea SecM83-1023 10/20/83 (Credit 400-IND) Malaysia Rural Credit (Loan 1900-MA) SecM83-991 10/24/83 Spain Second Livestock (Loan 1411-SP) SecM83-1037 11/07/83 Ethiopia Second Wolaita Agricultural Development SecM83-1182 12/19/83 (Credit 486-ET) Zaire First Livestock Development (Credit 398-ZR) SecM84-51 12/30/83 Romania Flood Recovery (Loan 1169-RO) SecM84-134 02/13/84 Algeria Meat Industry (ONAB) (Loan 1739-AL) SecM84-150 02/17/84 Bangladesh Shallow Tubewells (Credit 724-BD) SecM84-515 05/09/84 Ghana Ashanti Cocoa (Loan 1181-GH) '-cM84-473 05/24/84 Senegal Second Sedhiou Crops (Credit 647-SE) SecM84-596 06/11/84 Tanzania Fisheries Development (Credit 652-TA) SecM84-654 06/18/84 Upper Volta West Volta Agricultural Development SecM84-711 06/20/84 (Credit 706-UV) Chad Sahelian Zone (Credit 739-CH) SecM84-601 06/29/84 Indonesia Jatiluhur Irrigation Extension SecM84-629 06/29/84 (Credit 514-IND) Education Tanzania Fourth Education (Credit 371-TA) SecM83-904 09/15/83 Paraguay Third Education (Credit 667-PA) SecM83-913 09/19/83 Uganda Second Education (Credit 258-UG) SecM83-932 10/05/83 Liberia Second Education (Loan 1266-T-LBR) SecM83-103( 11/04/83 El Salvador Second Education (Loan 1007-ES) SecM83-1127 11/28/83 Nicaragua Second Education (Loan 1244-NI) SecM83-1183 12/20/83 Ethiopia Fourth Education (Credit 553-ET) SecM84-112 01/20/84 - 64 - ANNEX D Page 10 of 11 Education - continued Cameroon Second Education (Credit 320-CM) SecM84-89 01/20/84 The Hellenic State (Greece) Fourth Education SecM84-143 02/10/84 (Loan 1539-CR) Colombia Third Education (Loan 920-CO) SecM84-145 02/10/84 Philippines Mass Media Pilot (Loan S-8) SecM84-283 03/23/84 The Hellenic State (Greece) Second Education SecM84-449 04/20/84 (Loan 859-GR) OTHER OED REPORTS Annual Report on Operations Evaluation R83-282 08/24/83 Strengthening Agricultural Research and Extension - SecM83-879 09/01/83 The World Bank Experience Ninth Annual Review of Project Performance R83-302 09/16/83 Audit Results Impact Evaluation Report - Dominican Republic; SecM83-941 10/06/83 Livestock Development (Credit 245-DO) Concordance to Project Performance Audit Reports SecM83-1079 10/31/83 Issued by the Operations Evaluation Department (March 1972-June 30, 1983) Impact Evaluation Report - Malawi: Lower Shire Valley SecM84-16 12/23/83 Agricultural Development (Credits 114/363/823-MAI) Impact Evaluation Report - Kenya: First and Second SecM84-144 01/30/84 Forestry Projects (Loans 641/1132 and Credit 565-KE) Harvesting the Waters -- a Review of Bank Experience SecM84-240 03/13/84 with Fisheries Development Impact Evaluation Report - Benin: Hinvi Agricultural SecM84-302 04/05/84 Project (Credit 144-BEN) Impact Evaluation Report - Thailand: First Education SecM84-458 05/07/84 (Loan 471-TH) Impact Evaluation Report - Ivory Coast: Oil Palm SecM84-462 05/07/84 and Coconut Development (Loans 611, 612, 613, 759, 760, 1036 and 1382-IVC) - 65 - ANNEX D Page 11 of 11 OTHER OED REPORTS (Continued) The World Bank Built-In Project Monitoring and SecM84-466 05/11/84 Evaluation: Rural Development in Northeast Brazil Institutional Development in Africa - A Review SecM84-457 05/17/84 of World Bank Project Experience (Vols. I and II) Impact Evaluation Report - Gambia Agricultural SecM84-656 06/13/84 Development (Credit 333-GM) Impact Evaluation Report - Senegal Settlement Project SecM84-653 06/29/84 in the Terres Neuves Region (Credits 254/578-SE)