Report No.7415 Annual Report On Operations Evaluation (FY88) September 1, 1988 Office of the Director-General, Operations Evaluation Document of the World Bank This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be discibsed without World Bank authorization. ABBREVIATIONS USED DAC - Development Assistance Committee (OECD) DGO - Director-General, Operations Evaluation ECDP - Evaluation Capability Development Program EDI - Economic Development Institute IFC - International Finance Corporation JAC - Joint Audit Committee OED - Operations Evaluation Department OEU - Operations Evaluation Unit (IFC) PCR - Project Completion Report PPAR - Project Performance Audit Report PPR - Policy, Planning and Research Staff FOR OFFICIAL USE ONLY THE WORLD BANK Washington. D.C. 20433 U.S.A. Office of Director-Ceneral Operatons Evaluatin September 1, 1988 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT Annual Report on Operations Evaluation Attached, for information, is this year's Annual Report on Operations Evaluation. The report reviews evaluation activities in FY88, highlighting changes that have taken place in the wake of the Bank's reorganization and identifying issues that will be addressed in the current year. As a new feature, this report includes a detailed account of the follow-up by operational managers to the findings and recommendations of recent evaluation studies. Attachment This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. FOR OFFICIAL USE ONLY ANNUAL REPORT ON OPERATIONS EVALUATION, FY88 TABLE OF CONTENTS Page No. I. EVALUATION IN THE REORGANIZED BANK ............ 1 Report on the Effectiveness of Dissemination and Feedback ............ 2 PCR Backlog ............................. 2 Project Monitoring and Evaluation ....... 5 Evaluation of Bank Research ........ 5 II. OPERATIONS EVALUATION DEPARTMENT .............. 7 Project Evaluations...................... 7 Annual Review and Studies ............... 10 Internal and External Dissemination ..... 11 Economic Development Institute .......... 14 Evaluation Capability Development Program 14 Use of Resources .. III. OPERATIONS EVALUATION IN IFC .................. 18 IV. CONCLUSIONS ................................... 19 Annexes A - Responsibilities for Operations Evaluation. 21 B - Report on Dissemination and Feedback ...... 23 C - Monitoring and Evaluation in Projects ..... 83 D - OED Reports Issued During FY88 ............ 95 E - Operations Evaluation Reports and Papers Reviewed by the JAC ........... 105 F - Ongoing OED Studies ....................... 107 G - OED's Dissemination Program ............... 109 H - IFC Investment Assessment Reports and Proj-ct Completion Reports Filed in the Office of the Director-General, Operations Evaluation ..................... 115 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. ANNUAL REPORT ON OPERATIONS EVALUATION (FY88) 1. This report reviews the Bank's evaluation activities in FY88, as they relate to the Bank's ongoing and future operations. The first chapter discusses evaluation in the context of the reorganized Bank, highlighting two issues that have been of particular concern to the Bank's management and the Board the effectiveness of dissemination and feedback, and the growing backlog of Project Completion Reports (PCRs). Recent activities of the Operations Evaluation Department (CED) are discussed in Chapter II, while Chapter III reviews ex-post evaluation in the International Finance Corporation (IFC). I. EVALUATION IN THE REORGANIZED BANK 2. Under the new organization of the Bank, the evaluation function--together with other central functions--is seen as an essential counterpart to the delegation of authority and decision- making to line managers and as an important means of enhancing institutional memory and cross-fertilization of experience. In this context, evaluation responsibilities were strengthened across the Bank. Operations evaluation continued to receive strong support from Senior Hanagement, and the mandate of the Director-General, oyelanuub Evaluation (DGO) and of OED were recertified (Annex A). The President endorsed the establishment of Bank-wide working groups to review the issues associated with the preparation of PCRs and the means of reinforcing dissemination and feedback, including reinstitution of a formal system of management response to major OED studies. 3. From OED's perspective, an important change that arose from the reorganization was the elimination of central review functions of the former Operations Policy Staff, and the associated role of intermediation that it performed in disseminating OED findings across the Operations Complex along with the lessons drawn from its own studies and from project implementation experience. As a result, relations with the Policy, Planning and Research Staff (PPR) are acquiring a new character, more focussed on mutual support in developing and applying experience in policy, guidelines and research. These relations have been developed productively in a variety of ways across PPR without apparent duplication of efforts, a concern at the time of the reorganization. 4. As another consequence, OED faces more directly operational units across the Bank which are at the same time the source of UED's evaluation material and its prime users. This new relationship raises expectations on both sides: on the part of Operations, for more readily actionable OED messages; on the part of OED, for more thoughtful analysis of lessons in PCRs as well as for their timely production, as stressed in this report. At the same time, it should -2- be emphasized that intermediation is no longer centered in one place, but rather a responsibility of a numba of units across the Bank, not only in PPR or at the center in the Operations Complex, but also in the regional technical departments, with which OED will increasingly work. The new approach followed this year in reporting on the effectiveness of dissemination and feedback reflects responsibilities under the new organization. Report on the Effectiveness of Dissemination and Feedback 5. There are three stages in the dissemination and feedback process: (i) dissemination, which involves dispersing evaluation findings to operational managers and staff, and to interested parties outside the Bank; (ii) feedback, which involves absorbing the lessons of past experience into Bank operational strategies, policies, and economic and sector work, as well as into the design and implementation of new operations; and (iii) reporting to the Board on both activities, which can include planned follow-up ty management to a recent OED study or actions that may have been taken. 6. This year, a new approach was followed to provide more tangible evidence on the impact of ex-post evaluation on the Bank's operations. Bank managers were asked to indicate to the DGO the follow-up actions taken in response to specified OED evaluations of operational activities ior which tey were responsible. The tocos was more on what was actually learned and used to enhance quality than on when and from what source the lessons were learned--Project Implementation Reviews, PCRs or OED's evaluation work. 7. Although the inquiry was selective in its coverage, it included structural adjustment lending, country reviews, rural development, land settlement and other issues in agriculture. In addition, the four Regional Offices were asked to review the impact of OED audits and PCRs in the context of a few project cases. 8. The responses of managers appear in Annex B. They demonstrate the continuing concern for quality in operations and reveal an internal feedback process at work in much of the Bank, notwithstanding the initial post-reorganization discontinuities in management and institutional memory. The responses have been carefully reviewed in OED and will be followed up with Management or within OED. 9. This review has been worthwhile and should be repeated annually to monitor the adequacy of follow-up and to keep the Board and Management informed of progress. PCR Backlog 10. Project evaluations through PCRs continue to be the backbone of the Bank's operations evaluation system. They constitute a critical self-examination by borrowers and operational staff who -3- evaluate the outcome of Bank/IDA-supported projects, as a rule within six months of the final loan/credit disbuisement. The basic purpose of PCRs is twofolds accountability to the shareholders and Management of the Bank, and learning from experience. PCRs are the primary source of information for project performance audits and are essential building blocks for the Annual Review of Project Performance Results. for special studies undertaken by OED and elsewhere in the Bank, and for the dissemination of the lessons of operational experience to both Bank staff and borrowers. 11. Last -ar, the D-0 suggested that Management review the content, timeliness and cost-effectiveness of PCRs in the light of developments in recent years and as a means of strengthening evaluation in the Bank. Such developments included: (i) the introduction of selective audits by OED in 1982, when full audits were reduced to about 60 percent of completed projects initially and to close to 40 percent in the last year; (ii) the increasing participation of borrowe-s in tte preparation of PCRs as required by loan agreements since the late seventies; (iii) the growing perception that PCRs were overly descriptive and uneven in the quality of their analytical contents and findings; and (iv) increasing delays in the preparation of PCRs that were creating a large and growing backlog. A Working Group was appointed last year to review these issues. OED, which appointed one of its staff to the Working Group, is generally supportive of its preliminary findings. More specific comments will have to await finalization of the report. 12. Meanwhile, OED has continued to assess the extent of the backlog that had accumulated over recent years. At the end of FY88, PCRs had not been completed for 365 loans and credits, supporting 321 projects, that had been fully disbursed before the end of CY87 (Table 1). Twenty percent of these projects had been completed during the first half of FY88; another 53 percent during FY87 and FY86; and the remaining 27 percent even earlier. The backlog includes 24 structural and sector adjustment operations. Table 1: BACKLOG OF PROJECT COMPLETION REPORTS Years after Final Disbursement or Closing Date Region 0.5-1 1 - 2 2 - 3 3 - 4 4 + Total Africa 30 44 15 7 13 109 EMERA 9 18 5 4 13 49 Asia 12 25 12 11 20 80 LAC 15 31 19 5 13 83 Total 66 118 51 27 59 321 13. This issue was brought to the attention of Management and the Board last year, as most of the backlog predated the reorganization, -4- and it could be feared that work program priorities would lead to a further deterioration. The matter was raised again in connection with the OED Work Program, when it became clear that the reduced PCR program for FY88 and the priority criteria proposed in last year's Annual Report, including in particular the priority for adjustment lending, would not be adhered to. In connection with the FY89 budget discussions, Management reaffirmed the priority it was planning to give to structural adjustment operations within the current PCR program. 14. Programming of PCRe, which has long been deficient, was particularly late last year. Actual delivery was substantially less than programmed, with a shortfall that exceeded the 20 percent customary ir. previous years. Most PCRs reached OED late in the fiscal year, accentuating the perennial bunching (Table 2). Programming for FY89 has improveds the PCR program was put together Region by Region under the direction of the Office of the Senior Vice President, Operations, and it was issued and communicated to OED in the first month of the new fiscal year, much earlier and in more detail than in previous years. Tab:e 2: BUNCHING OF PCRs Fiscal PCRs -------Percentage Received------- Year Received Last Quarter June Only FY88 180 61 49 FY87 222 36 19 FY86 197 47 26 FY85 190 45 26 FY84 178 51 31 15. While the FY89 program of 247 PCRs would surpass the output of any previous year and would be in line with the expected completion rate of projects, it would leave the backlog essentially unchanged. The implications of this continued backlog for the integrity and effectiveness of the Bank's evaluation system cannot be over-emphasized. As the delivery of PCRs lags, their operational value and relevance decline. Where the delays are excessive, tne linkages between Bank staff and project implementation teams of borrowers are broken, and important data and findings are lost. This situation in turn affects the relevance of OED's Annual Review of Project Performance Results and delays OED studies. 16. The PCR Working Group is expected to recommend ways to eliminate the backlog, emphasizing the need for continu!ty between supervision and the PCR process. The DGO endorses this emphasis, which implies producing PCRs on a current basis and prompt liquidation of the backlog. Not to address this problem would prolong the undesirable circumstances under which the results and lessons of the - 5 - more distant past continue to dominate evaluation findings and therefore diminish the value of a system which is already inherently slow, given the long gestation of most Bank operations. It should be emphasized that writing off some of the backlog is not an option as it would impair the integrity of the evaluation system. Moreover, it is unlikely that objective criteria can be devised for selecting out operations which could ensure that accountability was not jeopardized or that valuable lessons were not missed. 17. There may. however, be avenues for redilring cost that are consistent with the principles of full coverage ; I transparency of project results. OED has agreed to support a detailed review of the backlog by Operations, which will assess more specifically the possibilities and time-frame for its elimination. Project Monitoring and Evaluation 18. The learning of lessons is not limited to ex-post evaluation. It starts during project implementation, when supervision by Bank staff ensures feedback throughout the project cycle. Host lcan/credit agreements now include provisions that commit borrowers to monitor the progress of projects and to evaluate their results and impact. These progress and evaluation reports provide important inputs into PCRs. Annex C shows the present status of project monitoring and evaluation in Bank-financed projects by Ma4or sector. Evaluation of Bank Research 19. Since the inception of the Bank's research program, the Research Committee or its equivalent has been required to evaluate all completed centrally-funded research projects. Here, too, a backlog of unevaluated projects has emerged over recent years. At present, more than 80 completed research projects await evaluation. No evaluations were carried out in FY88, in part because efforts were concentrated on reorganizinj, redirecting and staffing the Bank's centrally funded research program. However, the moratorium on evaluations during this past fiscal year is also the result of a decision by the Research Committee to undertake a major review of the existing research evaluation system, its value to the Bank, and ways of improving its effectiveness. The Joint Audit Committee (JAC) will be kept informed about the development of a new research evaluation system. 20. In the past, OED has participated in the evaluation of individual research projects. An effort is now underway to diversify this interaction to take greater advantage of common objectives and elements of OED and research programs, as both aim to improve the Bank's future contribution to the development of member countries through lessons from experience. In fact, OED's studies program often contains a significant element of research while much Bank sponsored research contains a large element of evaluation. A good example of the latter is the ongoing review of "Managing Agricultural Development -6- in Africa (MADIA)". The question of OED's involvement in the evaluation of Bank research will be considered in light of the Research Committee's recommendations, and the availability of resources taking into account other OED activities. - 7 - II. OPERATIONS EVALUATION DEPART'M 'T 21. Following the heavy turnover of higher level staff before and after the Bank's reorganization, staff rotation fell substantially in the latter part of FY88, and relatively few changes are expected for FY89. Altogether, eleven higher level staff were recruited in FY88--that is 28 percent of the authorized positions--while the same number of evaluation officers were transferred to operational assignments or retired from the Bank. At the end of June, there were two vacancies, which have since been filled. One further staff dcparture since then was anticipated by appropriate recruitment action. Thus, the department is fully staffed for the first time in many years. Table 3: OPERATIONS EVALUATION STAFF FY85 FY86 FY87 FY88 Higher Level Staff Authorized positions 39 39 39 40 Staff beginning FY 34 34 38 38 Recruitment 9 9 8 11 is a DepaiLtes 95 81 Staff end FY 34 38 38 38 Vacaneties end FY 5 1 1 2 Support Staff Authorized positions 21 21 21 21 Staff end FY 21 20 21 20 Vacancies end FY 0 1 0 1 22. Despite the scarcity within the Bank of some of the required skills, OED had little difficulty attracting staff of the high standard required, so that the current staff profile adequately matches the diversity and complexity of the department's work program. 23. With over 50 percent of the evaluation officers having joined the department in the last two years and an average staff tenure of about five years, no major staff rotation is expected in the next two to three years. Project Evaluations 24. OED's output of project evaluation reports declined from 220 in FY87 to 175 in FY88 reflecting the drop in the numbers of PCRs received and their bunching at the end of the fiscal year (para. 14), -8- as well as the impaL of high staff rotation in OED during the past year. Table 4s PROJECT EVALUATIONS FY82 FY86 FY87 FY88 PCR9 received /a 159 197 222 180 Evaluation reports issued -a 122 219 220 175 of which - PPARs 122 ]b 125 109 71 - PCRs passed through - 94 111 104 Audit ratio 100 57 50 41 Project evaluations in process c 173 156 158 163 /a Number of evaluated projects. /b Last year when OED audited all projects. /c From receipt of PCR by OED to completion of final evaluation report. 25. Of the 175 project evaluations sent to the Board in FY88, 71 were project performance audits (PPARs). In addition, 56 audits were in progress, a somewhat higher number than at the end of FY87. The remaining 104 evaluations were PCRs that had been passed through by OED staff without a full audit (Annex D). The audit ratio was 41 percent, compared with 50 percent proposed in the Work Program for FY88. During FY89, the audit ratio is expected to recover, reaching about 45 percent and consisting of a minimum 40 percent audit ratio for projects and 100 percent audit of structural adjustment operations, as proposed in OED's Work Program. 26. The sectoral composition of the evaluation reports is shown in Table 5. The variations in audit ratios were partly the result of small numbers in some sectors which over several years, tend to be distributed more evenly. More importantly, the audit ratios reflect the criteria OED staff used to select projects for audit. They continue to give preference to innovative operations, large and complex projects, operations in new sectors and countries, and audits requested by Executive Directors. - 9 - Table 5: EVALUATION REPORTS ISSUED IN FY88 ----Audit Ratios----- Sector PPARs PCRs Total FY88 FY87 FY86 Agriculture 26 35 61 43 47 71 Human Resources 10 8 18 56 38 59 Transport & Urban 15 18 33 45 54 50 Energy & Public Utilities 7 23 30 23 65 39 Industry/DFCs 6 17 23 26 35 48 Structural Adjustment 4 0 4 100 100 100 Technical Assistance 3 3 6 50 25 75 Total 71 104 175 41 50 57 27. Grouping of audits for the same or adjacent countries to save travel time and afford missions of two rather than one staff, and clustering of audits by sector in the same country to increase their thematic value and the replicability of findings, remain essential objectives. Clustering and grouping suffered a setback in FY88, however, when they dropped to 38 percent of total audits, compared with 49 percent in the previous year. This situation illustrates the inherent conflicts in the objectives for OED's management of the PCR inventory: on the one hand, timely review of PCRs and production of audits, leading to. lower year-end inventories, the current situation; on the other hand, holding PCRs long enough to permit more clustering, a practice that leads to higher inventories and perceived delays. 28. While an increase in the flow of PCRs arising from a reduction in the backlog would temporarily allow more clustering, more permanent solutions have been under consideration in OED for some time. The most radical and potentially valuable one is to delink PCR review and audits by passing through all PCRs promptly after they have been received, reviewed and certified, and then auditing the proportion agreed to with the Board at the most opportune evaluation time--including some preliminary operating and impact experience-- and when clusters can be maximized. In the two years since this idea was first discussed within OED, it has received broad support from the department's management and staff. Presenting it to the JAC and the Board hag, however, been postponed until now in view of the Bank's reorganization and the subsequent establishment of the PCR Working Group, and because of the additional strain on management, staff and resources that would have been associated with one more initiative. 29. Implementing this approach would involve some transitional cost, especially in the first year, as resources allocated to review of PCRs would increase by 40-45 percent over current levels resulting - 10 - in some diversion of resources from other OED work. In addition, more intensive management would be required, as opposed to the current reactive process, to set and apply criteria for audits and clustering and to meet increased reporting requirements to satisfy the JAC and the Board about the discharge of the audit function. 30. We propose at this time to begin testing the new approach on a small scale by passing through a few extra PCRs intended for clustering while maintaining the number of audits called for under the FY89 work program. Whether more could be done in FY90 will be outlined in the next Work Program, in light of JAC/Board reaction now and new facts on hand then. Another way to broaden the scope of clustering would be to audit comparable operations in several countries at the same time. This approach, raises, however, difficult issues of confidentiality and processing, especially where comments are called for on comparative performance, which require careful consideration. Small-scale experiments are under way. Annual Review and Studies 31. arnual Reviews of Project Performance Results together with impact evaluations and special studies continue to be used by OED as principal instruments for synthesizing and disseminating the lessons of experience. The Annual Review covering the results of PPARs and PCRs issued in CY86 was issued in October 1987 and published in April 1988. The Review for CY87 has been issued on August 17, 1988. 32. Last year's Annual Report focused attention on the limitations of the Annual Review--as it has evolved over the previous 13 years--and undertook to review its design, a process which, given the long cycle, could only begin producing results in 1989. These limitations stem from the excessive expectations placed on the analysis of a largely heterogeneous set of operations, linked together only by the fact that they were evaluated in the same calendar year, though they were started and closed at different times and under different Bank policy and economic circumstances. The growing diversity of the Bank's output in the 1970s and 1980s have made the analysis increasingly complex and the returns on this considerable effort lower. 33. Preparatory work on the 1988 Review was started last January, about nine months earlier than for previous Reviews. Given the issues involved and the intended changes, an approach paper will be prepared for consideration by the JAC before drafting of the Review begins. In the meantime, a panel of senior Bank staff will be appointed to advise the DGO on the preliminary design of the next Review. 34. Besides the Annual Review, OED issued the following eight studies during FY88: " World Bank Experience with Rural Development 1965-1986 o A Review of Bank Assisted Free-Standing Technical Assistance Credits in Indonesia - 11 - * Evaluation of Structural Adjustment Lending in Turkey * Educational Development in Thailand: The Role of World Bank Lending * Renewable Resource Management in Development - Lessons from World Bank Experience in Agriculture and Forestry * Agricultural Marketing: World Bank's Experience * Conditionality in World Bank Lending: Its Relation to Agricultural Pricing Policies * An internal study on project cost discrepancies prepared in support of the Annual Review, which was circulated to Bank scaff. 35. Another two studies were scheduled for completion in the FY88 work program. One of them--dealing with the sustainability of power projects--was discontinued to avoid duplication of effort with the Colombia Power Sector Review. After the latter is completed and in the light of its findings, consideration may be given to designing a multicountry power sector review. The other study--a review of the envizonmental impact of infrastructure projects in Brazil--was delayed by discussions with the Government. An approach paper is being prepared and will be forwarded to. the JAC after a preparatory mission to Brazil in the Fall. 36. During the past fiscal year, efforts were invested in strengthening the pipeline of studies in process, which has risen to 30 at year's end, compared with 22 a year earlier. Of these, 12 can reliably be expected to be completed in FY89, as contemplated in the Work Program. Approach papers are now prepared in much more detail, including file search where appropriate and vetting with other Bank staff, before being submitted to the JAC, along with more reliable cost and completion time estimates, as required by the rising complexity of the studies. Eleven approach papers have been submitted to the JAC, and another eight are well advanced (Annex E and F). Of the larger studies due for completion in FY89, three had reached the drafting stage by the end of FY88, including two country reviews (Senegal and Tanzania) and the power sector review in Colombia. It is expected that at the end of FY89, the inventory of work in progress will have been somewhat reduced and will include the reviews of supervision, procurement under SALs, the Bank's approach to industrialization, and human resource development in Africa. Internal and External Dissemination 37. During FY83, dissemination and feedback continued to receive particular attention. As suggested a year ago, the Bank's Management appointed a Task Force to review internal dissemination and feedback - 12 - after the reorganization and to recommend their strengthening and the reinstitution of a formal system of management response. The Task Force, which consisted of managers and senior staff from PPR and Operations, was supported by the leader of OED's own internal task force on the same subject. Its report is currently being reviewed by Management and comments are therefore premature. What can be said at this point, however, is that its recommendations concerning OED's contribution to the process are fully consistent with OED's own efforts and goals in the context of the new organization of the Bank. 38. The elimination of the central review function of the former Operations Policy Staff (OPS) has important implications for OED's role in dissemination, as well as for its reporting on dissemination and feedback. In the past, OPS critically reviewed OED's output, and combining it with experience gleaned from other sources, disseminated it sector by sector as part of its own review process as well as its policy and guidelines functions. Now, in addition to its interaction with PPR, OED faces its Operations *market' more directly, requiring closer interaction with operations staff across the Bank, the formulation of messages that are actionable or trigger problem- solving analyses, and supporting the delivery of such messages more actively inside and outside the Bank. Annex G, an update of what was contained in knex 7 of OED's Work Program, provides further background. 39. Interaction with Bank staff has been further stressed in all phases of OED's FY88 activities. Comments are increasingly sought in the course of preparing the Work Program, now set in a broader and three-year perspective, as well as for approach papers on studies before they are submitted to the JAC. This consultation contributes to the relevance and quality of the work proposed and facilitates, because of the expectations generated, subsequent dissemination and feedback. In some cases, PPR and/or Operations offered funding or staff contributions to studies of particular interest to them. The merit of each case will be presented to the JAC in connection with the relevant approach paper for approval. Participation of OED on the PCR and dissemination and feedback Task Forces, closer review and discussion of PCRs, and cooperation with the Economic Development Institute (EDI) and the research complex, all enhanced this Bank-wide interaction. 40. Formulating odissemination ready" messages is another ongoing effort which involves: (i) prioritizing and targeting findings and recommendations to specific audiences; (ii) product design, as noted !n connection with the plans to cluster more audits (para. 27) and to improve the design of the Annual Review (para. 32); and (iii) use of professional editors for OED's major studies, to enhance their clarity and focus before they are submitted to the Board. 41. To assist in the delivery or retrieval of these messages, OED plans to hold more seminars to disseminate the findings of its studies, both before and after their discussion with the JAC and the - 13 - Board. In addition, sectoral data bases will be further developed to facilitate access to project information, such as cost overruns, implementation delays or economic rates of return. In FY88 OED added data bases for transport and education to its existing agriculture data base. 42. OED is considering other dissemination instruments, such as the preparation of abstracts or publication of an OED digest, which could improve access to evaluation findings, an objective supported by the Task Force on Dissemination and Feedback. 43. OED also widened the scope of its external dissemination activities, which not only benefit from, but contribute to, the internal dissemination. A start has been made by publishing selected OED documents more widely. Following the recommendations of the OED publications committee established in early 1987, seven OED reports were published during FY88. In addition to two Annual Reviews (translated into French), they included the following studies: * Sri Lanka and the World Bank A Review of a Relationship * The Aga Khan Rural Support Program in Pakistan An Interim Evaluation * Rural Development World Bank Experijnce, 1965-1986 (translated into French and Spanish) * Cotton Development Programs in Burkina Faso, Cote d'Ivoire and Togo (translated into French) * The Jengka Triangle Project in Malaysia Impact Evaluation Report. 44. The rural development study illustrates the range and costs of dissemination. From August 1987, when the study was completed, to the end of FY88, dissemination costs totaled about 30 staff-weeks and US$67,000 in cash expenditures--more than one-third of the cost of the study itself. These costs covered such activities ast (M) professional editing before Board review; (ii) internal staff seminars before and after Board review; (iii) desk-top publishing; (iv) translation into French and Spanish, including review by OED staff; (M) round-table discussions with journalists, arranged by the Bank's External Relations staff which resulted in some press articles; (vi) preparation of an article for publication in Finance and Development; and (vii) an international seminar sponsored by OED in Paris last February, at which senior evaluation and rural development experts from the DAC countries and several multilateral agencies shared their experiences on rural development. - 14 - 45. Summing up, the priorities for internal dissemination will be to continue building relations with operational staff through seminars and workshops, and preparing abstracts, initially on an experimental basis. Externally, in addition to ongoing relations with evaluation units of other donor agencies, OED will work closely with EDI, as outlined below, and will cintribute to strengthening the network of evaluators in developing countries, in particular through the Evaluation Capability Development Program (para. 49). Economic Development Institute 46. An obvious commonality of interests is already manifest in the long-standing working relationship between OED and EDI. It has includedt OED contributions to the mid-term evaluation of EDI's current five-year program; participation of OED staff in EDI seminars on monitoring and evaluation; and modest contributions to the development of case material. The scope for expanded cooperation will be examined and it is expected that the recent appointment of the DGO to EDI's Advisory Board will support this effort. In the coming year, this effort will be experimental and although still small scale, more structured. 47. From time to time in the past, EDI has used OED reports as training materials for courses and seminars, notably in agriculture, education and national economic management. OED and EDI are now beginning to work collaboratively on a series of studies that will utilize OED findings more systematically for training in EDI and its partner institutions. Experiments with alternative ways to develop teaching case studies based on OED findings will be carried out jointly by OED and EDI staff and consultants. These will include both attempts to transform OED reports into teaching materials, and EDI participation in a few OED missions to prepare teaching materials in parallel with the OED report. 48. EDI policy seminars and courses are normal outlets for OED's output, appropriately combined with contributions from other sources. OED will look more systematically at the prospects for suggesting, in advance, topics, audiences and, when possible, cofinanciers for consideration in EDI's programs. Relatively little use has been made of OED's pool of experience for lecturing at EDI due to the difficulties in matching work and travel schedules. The possibilities for doing more are being reviewed. In addition OED places a particular emphasis on providing staff support to EDI courses and seminars contributing to the ECDP including, but not limited to, EDI's courses on monitoring and evaluation, for which there is a clear OED priority as well as budget resources. Evaluation Capability Development Program (ECDP) 49. The ECDP aims to help interested developing countries create or strengthen a central capacity for the ex-post evaluation of - 15 - development policies and programs and to advise and support implementing agencies in preparing completion reports, including PCRs for Bank-supported projects. Some bilateral donors and multilateral agencies have indicated their willingness to support this program, although the stage of formal cofinancing has not yet been reached. During the past year, the interest of several borrowing countries was established, and a number of activities (visits, seminars, preliminary design) was undertaken. 50. While adjusted to divergent country circumstances, an overall approach has emerged. The first phase of the program in a given country aims to establish the country's commitment to ex-post evaluation, a process that normally involves a fairly short visit by senior OED staff. The second phase usually takes place at Bank headquarters with broad participation of OED, Regional and EDI staff. It involves exchanges of views with the country's senior evaluators about their country's evaluation capabilities and needs, and how OED can help strengthen ex-post evaluation through work program formulation and implementation, methodology, guidelines, institutional advice, or as a catalyst for other contributions in the areas of training, support for PCR preparation and funding. In subsequent phases OED will undertake, with the help of consultants, to design a framework program and make arrangements for preparation, delivery and funding of its components by the most qualified partners acceptable to the country. In addition, OED is prepared to involve country staff in its own work at headquarters in a few cases, or in the country itself, to help build documentation and information exchanges through a network of evaluators and generally to ensure sustainability over time. 51. Only a few countries have been approached since the program was started in FY87, to allow time for developing a workable approach and to avoid overcommitting future resources and delivery capacity. This may have been too prudent a course to follow in light of the slow progress in the initial set of countries reflecting their particular circumstances, including pending organizational changes, priority of project implementation in allocating scarce staff and local funds, etc., which have delayed the detailed design phase beyond FY88. While OED expects to break new ground in FY89 in most of these countries, it is also prepared to add new countries. Use of Resources 52. During FY88, the use of authorized higher level positions was somewhat constrained by the high turnover of staff. The shortfall was, however, largely made up by a greater use of consultants. Work proceeded through the reorganization with only relatively minor slippage in some of the audit work and studies as new staff was taking over. Deducting the cost of overhead activities, the department effectively used 37.8 staff-years to carry out its program, almost 10 percent more than in FY87. The increase reflects primarily a drop of 3.1 staff-years in overhead expenditures which peaked in FY87 when: - 16 - an administrative officer was appointed; a senior adviser wad used temporarily to provide continuity in the face of a rapid turnover in management. Management and administration were actually reduced in FY88 compared with earlier years, reflecting the abolition of one of OED's three divisions. Within this item, more time was dedicated to administration and budgeting, with a noticeable improvement in PCR data, time recording, costing and budgeting control. Management stricto sensu, which also includes internal and external relations, while better supported and more efficient, i iow constrained in view particularly of the new initiatives outlined in this report, a matter which is now under review in OED. Table 6 details the actual sources and applications of staff and consultant resources in FY88 compared with budget as well as actual figures for FY87 and FY86. Table 6: STAFF AND CONSULTANT RESOURCES (Higher Level Staffyear Equivalent) FY86 FY87 FY88 FY88 --------- Actual--------- Budget Sources Authorized Higher Level Positions 39 39 40 40 Actual Use of Positions .35.8 37.2 36.2 37.8 Use of Consultants 10.5 11.3 12.5 11.3 Total Resources 46.3 48.5 48.7 49.1 Overhead 10.7 14.0 10.9 11.3 MHanagement & Admin. (4.3) (5.7) (3.7) (4.0) Leave, Training, Conferences (6.4) (8.3) (7.2) (7.3) Resources Available for OED Work 35.6 34.5 37.8 37.8 Applications Project Evaluations 14.6 11.2 9.3 11.0 of whicht SAL Audits (1.0) (1.7) (0.3) (1.4) Annual Reviews & Annual Report 3.8 3.9 4.4 3.8 Studies 13.5 14.4 18.1 15.7 Dissemination 1.0 1.5 2.6 2.5 ECDP - 0.3 0.7 2.0 Other Activities a/ 2.7 3.2 2.7 2.8 Total Application 35.6 34.5 37.8 37.8 a/ Evaluation support to Operations, PPR and EDI, departmental retreats, methodology and work program development. - 17 - 53. Actual expenditures for PCR reviews and audits (including adjustment audits) and for the ECDP were less than was budgeted for reasons explained in paras. 24 and 51, respectively. In addition, the ECDP budget expenditure was significantly understated for several reasones travel expenses were minimal in the case of OED staff, as they were charged to the evaluation work with which ECDP work was combined, or, in the case of country staff, were paid from technical assistance programs. In addition, management inputs, which were significant at the identification and conceptual stage, and contributions received from Bank operational staff were not charged to ECDP. 54. The savings, totaling about 3.0 staff years, were reallocated to advance the preparation of the studies program (2.4 staff-years) and the 1988 Annual Review (0.6 staff-years). Dissemination activities were about as budgeted and well above previous years, an expansion reflected in paras. 37 and 43. - 18 - III. OPERATIONS EVALUATION IN IFC 55. IFC's operations evaluations focus on studies of groups of projects to identify underlying patterns and lessons that may be of broad application. The studies are intended to assist IFC in the selection, appraisal, structuring and supervision of its investments. During FY88, the Operations Evaluation Unit (OEU) of IFC submitted to the JAC a review of IFC's work in the agricultural processing and storage sector; an abridged version was sent to the full Board to permit dissemination to member countries. The JAC discussed the report, and IFC's Portfolio Committee met, with the DGO present, to discuss the findings. Twenty-seven Investment Assessment Reports on completed IFC projects were issued in FY88; they are filed in the Office of the DGO (Annex H). 56. To encourage greater dissemination and feedback of the lessons arising from special studies, OEU has been discussing the findings with individual departments or divisions, has prepared brief summaries of the main findings in a form that can be sent to present and potential clients or others outside the Corporation, and has circulated to IFC's management and staff a comparative summary of findings with respect to IFC's management and market assessments, based on the studies already carried out. In addition, OEU recently began to provide comments on investment proposals through the Economic Advisor, to the Investment Review Committee for consideration before project appraisal. These comments are intended to provide guidance from IFC's past experience, based essentially on the findings of IFC's operations evaluation work. 57. During FY88, OEU also substantially completed a review of IFC's work on agricultural production projects, the second of a series of studies on IFC's experience in the agribusiness sector. This report will be circulated this fall to the JAC and the full Board. Work continued on a review of IFC's participation in the textiles sector. IFC is focusing on textiles and agribusiness because of the poor performance of its investments in these sectors, the importance of its fiancial exposure, and the expectation that IFC will continue to make substantial investments in these sectors. - 19 - IV. CONCLUSIONS 58. The need for a strong evaluation function as a necessary complement to the increased accountability of operational managers is being stressed in the new organization of the Bank, and the mandate of the DGO and the Operations Evaluation Department has been reaffirmed. As a result, evaluation is being strengthened across the Bank as well as in OED which is now fully staffed to meet the demands of its Work Program. 59. In the Bank's new organizational structure, OED faces more directly the operational units across the Bank which are the primary source of its evaluation material and the recipients of its output. Relations with PPR are also acquiring a new character, more focused on mutual support in developing and applying experience to policy, guidelines and research. OED is taking opportunity of these changes, stepping up its interaction with other units Bank wide, to ensure the continued relevance of its products, support their dissemination and report on their impact. 60. Specific proposals put forward in last year's Annual Report have been followed up by OED, ?PR and Operations. They include: * More participation of Bank staff in the review of OED's work program for FY89, now set in a broader perspective and three- year time frame; * A new approach to the Annual Review of Project Performance Results which will be discussed with the JAC later this fall; * A Working Group to review the format, content and process of preparing PCRs to improve their timeliness, their value to Bank staff and borrowers, and their cost-effectiveness; * A Task Force to review the effectiveness of dissemination and feedback, focusing on measures to strengthen the integration of evaluation findings into the design and implementation of new operations; * The strengthening of OED's own dissemination program through seminars, data bases, publication and translation of selected reports. 61. A new and rewarding approach was introduced this year to report on the effectiveness of dissemination and feedback. The responsible managers were asked to report on follow-up to past OED output and their responses are reproduced in this report. They show a continuing concern for quality in operations and reveal an active internal feedback process at work in much of the Bank in the post- reorganization period. - 20 - 62. Two issues have come to the fore in recent months and will be acted upon in the current year: * One issue concerns the PCR backlog that accumulated well before the reorganization. OED estimates that, as of June 30, 1988, the backlog included 321 projects, financed with 365 loans and credits. The view expressed in this Report is that the PCR backlog needs to be liquidated promptly and fully so as not to compromise the integrity of the system and the learning from experience. OED has already agreed to help Operations in its analysis of the backlog and their assessment of the means by which it could be eliminated. Future PCRs would be prepared promptly so as to maintain continuity with project supervision. * To enhance the timeliness and operational relevance of project evaluations, this report proposes to delink PCR reviews and audits by passing through all PCRs promptly after they have been received and reviewed by OED, and to audit later the proportion agreed with the Board at the most opportune time. This approach would provide greater scope for clustering projects in audits, and would increase the analytical quality and relevance of Annual Reviews. ANNEX A Page 1 of 2 - 21 - ANNUAL REPORT ON OPERATIONS EVALUATION (FY88) RESPONSIBILITIES FOR OPERATIONS EVALUATION The Director-General 1. The Director-General, Operations Evaluation (DGO) has overall responsibility for the Bank's evaluation function. This includes: - assessing whether the Bank's programs and activities are producing the expected results; - incorporating OED's assessments and findings into recommendations which will help improve the efficiency and effectiveness of the Bank's programs and activities, and their responsiveness to member countries' needs and concerns; - appraising the Bank's operations evaluation system and reporting on its adequacy for use within the Bank and by member governments; and - auditing periodically ac'ions taken by the Bank in connection with the findings of studies by the Operations Evaluation Department, and reporting thereon to the Executive Directors and the President. 2. The DGO reports to the Board and has an administrative link to the President. He is appointed for renewable five-year terms and can be removed only by the Board. To further ensure his independence, he may not join the Bank staff upon termination. The incumbent took office March 3, 1986. The Operations Evaluation Department 3. The Operations Evaluation Department (OED) is the staff arm of the Director-General. It is headed by a director appointed by the DGO. The incumbent took office August 15, 1986. The principal functions of OED are to: - assist the Director-General in making periodic assessments of the adequacy and effectiveness of the operations evaluation system in the light of the objectives and programs of the World Bank; ANNEX A Page 2 of 2 - 22 - - carry out performance audits on selected completed projects and conduct evaluation studies focusing on selected issues and sectors; - encourage and assist member countries to develop their own operations evaluation system; and - disseminate evaluation findings regarding Bank operations both within the institution and to the wider development community. 4. OED has a staff complement of 61, including 40 higher-level staff, currently distributed into a front office and two divisions: Agriculture, Infrastructure and Human Resources; and Policy-Based Lending, Industry, Public Utilities and Urban Sectors. International Finance Corporation (IFC) 5. IFC's Economics Department is responsible for the operations evaluation function within IFC. It carries out this function through an Operations Evaluation Unit (OEU) consisting of two higher level staff and one research assistant, under the general oversight of the DGO. OEU maintains close relations with OED, consulting with OED staff at various stages of the evaluation process and observing OED's criteria for independent evaluation. 6. The purpose of operations evaluation in IFC is broadly: - assessing whether IFC's programs and investments are producing the expected results; and - feeding these findings back into recommendations which will help improve the performance of IFC's investments. Joint Audit Committee 7. The Joint Audit Committee (JAC) of the Board of Executive Directors oversees, inter alia, the Bank's operations evaluation procedures and activities. It reviews OED's annual work program as well as selected OED reports to identify problems or policy issues for consideration by the Executive Directors. In addition, a subcommittee to the JAC reviews a select number of project evaluation reports and assesses the adequacy of the project evaluation system. ANNEX B Page 1 of 58 - 23 - ANNUAL REPORT ON OPERATIONS EVALUATION (FY88) REPORT ON DISSEMINATION AND FEEDBACK Table of Contents Page Structural Adjustment Lending ............................ 2 Country Revieva ............................................. 4 The World Bank in Pakistan ......................... 4 The World Bank and Sri Lanka ....................... 11 Program Lending ............................................. 14 Bangladesh Import Program Credits I - X ............ 14 Rural Development ........................................... 16 Cotton Development in West Africa ........................... 19 Cost Recovery in Irrigation ................................. 20 Land Settlement ............................................. 22 Smallholder Livestock ....................................... 24 Monitoring and Evaluation ................................... 26 Economic Rates of Return t.r.... ...... Go***. ............ 29 Project Evaluations ............................. ........... 30 EM NA ...................... ........................ 31 Africa ............ o..... ........... .........o........ 40 LAC . .... .. .. . . .. . . .. . ............-o o ** -- o oo o..... 45 Asia ...o. ...... .......................... .... o.. 48 ANNEX B Page 2 of 58 - 25 - ANNUAL REPORT ON OPERATIONS EVALUATION (FY88) REPORT ON DISSEMINATION AND FEEDBACK This Annex reproduces the responses of operational managers who reported to the Director-General, Operations Evaluation (DGO) on the effectiveness of dissemination and feedback in the context of a selected nuber of evaluation studies, audits and PCRs. No attempt has been made to change the text of these memoranda; only introductory or transmittal statements were removed. The responses are arranged in two parts% the first part illustrates the operational feedback to the evaluation studies while the second part contains regional responses to selected audits and PCRs. I. OED STUDIES A. STRUCTURAL ADJUSTMENT LENDING Structural Adjustment Lending A First Review of Experience OED Report No. 6409, September 24, 1988 Evaluation of Structural Adjustment Lending in Turkey Overview of SAls I - V OED Report No. 7205, April 13, 1988 Annual Review of Project Performance Results, 1987 Chapter VII: Structural Adjustment Lending OED Report No. 7404, August 17, 1988 The First Review of Experience (September 1986) covered the earliest fifteen structural adjustment loans and credits, made to ten countries in the fiscal years 1980-83. The last operation covered in this group, Thailand Second SAL, was approved by the Board in March 1983. This early period was particularly experimental and many of the findings that were brought out in the report, which could benefit later processing had in fact been fed back into the process even before the report was published. The report rightly drew attention to the limited capacity of the implementing borrower in respect of institutional development and the inadequate time allowed to translate objectives into monitorable steps to guide implementation. Since that time a gradualistic approach has been adopted and care has been taken to analyze the institutional constraints sufficiently beforehand, and often ANNEX B Page 3 of 58 - 26 - to incorporate technical assistance components or process parallel loans to bring about the institutional reforms. Other findings regarding cofinancing and aid coordination, and attending to the social impl'cations of SALs have also been taken up in subsequent operations. All adjustment operations now contain a financial table showing the foreign exchange gap and the contributions expected from other sources besides the Bank. Serious efforts are now made to tap other sources of finance in the realization that expanded funding is often essential to support structural reforms. A number of initiatives have been started to assess the social dimensions of poverty and the use by various groups of public services as well as to gauge the .arious parameters of demography, income distribution, subsidy benefit, etc. Such efforts are bearing fruition slowly and are being integrated in the design of structural reforms in the countries concerned. The finding that front- loading of conditionality would expedite implementation of adjustment operations has been taken into the new design and increasingly these operations are front-loaded which guarantees them a greater measure of success. The report on Turkey's five SALs has added a valuable perspective in that this was the only case where structural adjustment took a full course. This report was issued in April 1988 so there has been little time to incorporate afresh many of its findings. The report described the progress achieved under the five SALs as representing "genuine and long lasting improvements." It has also brought out a number of lessons regarding the sustainability of the process, institutional reforms, the need for mid-course correction and the crucial importance of a macroeconomic framework. The overview audit of the Turkey SALs was discussed at Bankwide seminar chaired by the Vice President, Economics and later by the Joint Audit Committee. The Bankwide seminar was a most lively meeting where opposing points of view were advanced with great conviction about the alleged shortcomings of design and whether these could have been anticipated beforehand when design had to be done quickly and often with a poor information base. But the discussion itself made participants aware of the many issues that have to be mastered for a successful design of such operations and these are expected to be reflected in later operations. The chapter prepared for the next Annual Review on SAL evaluated in calendar year 1987 covered nine structural adjustment operations and two supporting technical assistance projects in five countries (Korea, Malawi, Mauritius, Turkey and Yugoslavia). The chapter reflects the widely divergent resource endowments and policy regimes of these countries. The ass-,ssment of impact is bedeviled by changed exogenous conditions, with Malawi and Yugoslavia achieving the ANNEX B Page 4 of 58 - 27 - least by way of results. The chapter stressed again the lessons that structural adjustment needed time and patience especially for institutional building and policy analysis before a program can be successful. Another important point is the limited absorptive capacity of some of the borrowers for the reforms, which also urges selectivity in conditionality. Government commitment to reforms has been identified as sine qua non for success, and the importance of carefully sequencing reforms has been stressed. A major adjustment Lending Policy Paper by PPR is under preparation drawing on all experience gained so far. This incorporates OED findings and will guide the future processing of these operatiors. It should be mentioned that all three OED reports have been distributed widely throughout the Bank, were extensively commented upon by the relevant staff, and figured prominently in seminar and discussion groups, particularly in the EMENA Region because of the focus on Turkey. In this respect a joint EDI/EMENA seminar was organized in March 1988 in Izmir, Turkey which was attended by representatives from Algeria, Tunisia, Egypt, Jordan, Pakistan and Turkey to discuss structural adjustment. A similar seminar was held also in March 1988 in Morocco. The Turkey Report was avidly read by Turkish officials and was thoroughly reviewed in Ankara in January 1988 by a two-week OED mission. In other countries the dissemination process was not that active. OED reports on structural adjustment have attracted considerable attention both within and outside the Bank. Many of the lessons derived fro this work had already been learned by Bank staff themselves and were only brought out in OED reports because of the time lag that causes the audit to focus on early operations, not the very last ones. The findings are being incorporated in fresh operations to the extent possible. Among the important findings that had substantial operational impact are the necessity of strong borrower commitment; the advantage of a strong analytical basis for policy prescriptions; the importance of selectivity of reforms so that borrower capacity to implement is not overtaxed; greater scrutiny of the public enterprise system and better understanding of the privatization process; the need to attend to financial sector reforms to improve performance of the productive sectors; and the need to front-load conditionalities to facilitate and expedite implementation. ANNEX B Page 5 of 58 - 28 - B. COUNTRY REVIEWS The World Bank in Pakistan Review of a Relationship, 1980-1984 OED Report No. 6048, January 27, 1986 The OED Report has been widely distributed to Bank staff working on Pakistan. Its issuance was timely for orienting and guiding the many new Bank staff that have been assigned to work with Pakistan since the reorganization. A seminar was held with the authors of the Report and Bank staff in Washington shortly after its issuance and there has been wide distribution of the report in Pakistan. Although the repor# has been received with interest at the highest levels of the Pakistan Civil Service, the Government has preferred that it not be disseminated to the public, either in written published form or through seminars. The report contained many recommendations. In order to assess the report's impact, there follows a summary of its recommendations and a brief discussion of the actions that have and have not taken place since its issuance. Erratic lending levels have imposed unwelcome discontinuities and constraints on the Bank's project and sector work in Pakistan. Bank planners for Pakistan should "...remain keenly aware of the program- shaping effects that inhere in the steadiness or not of the total financial ration the institution provides". Bank lending from FY85-87 has been relatively steady--in the $600-700 million range, with about 1/3 of the lending program in IDA resources. Bank strategy in Pakistan is to link lending levels to macroeconomic performance despite the risk that lending may fluctuate when conditions are not met. The Bank's policy in this regard has been articulated to the Pakistan authorities and in this sense should not be regarded as erratic. During the 1970s several key positions--regional vice president, country programs director, country economist--experienced a turnover that was undesirable. There have been significant headquarters changes in the key personnel assigned to work on Pakistan as a result of the reorganization. A more normal rotation pattern is now in effect. ANNEX B Page 6 of 58 - 29 - The Bank needs stronger resident representation in Islamabad. The resident staff needs further reinforcing and should be given more responsibilities and discretion. A stronger resident staff should be able to replace the itinerant headquarters' missions that have reached 200 a year. The respective roles of headquarters, missions and field offices must now be seen in light of the reorganization, which the report did not and could not have contemplated. A Bank management team made an evaluation of the resident mission after the issuance of the report, but its work and recommendations were overtaken by the reorganizaton. A Bankwide task force is now at work evaluating the role of resident missions. The role of the resident mission has been enhanced in the country dialogue with the presence of a resident representative well equipped to conduct such a dialogue cnd the support of a headquarters assigned mission economist. In other respects the functions of the resident mission have not changed, nor have the number of headquarters missions been reduced. The requirements of project preparation and supervision have left little room for the reduction of headquarters missions. Changes in the function of the resident mission will be considered after the Bankwide task force completes its work and the effects of the reorganization are better understood. There is common consent for the Bank to take the lead in aid coordination. To be successful, it will be necessary to have stronger Bank representation in Islamabad and Pakistani Government participation as the associate lead coordinator. The Bank continues its work as the lead coordinator for the Pakistan Aid Consortium. In this capacity it has been the leader in undertaking macroeconomic analysis and conducting the macroeconomic dialogue with the country. The Bank's lead in this area has been increasingly shared with the IMP, particularly since Pakistan has come under consideration for the Structural Adjustment Facility. The Bank has taken the lead in various sector work and has been particularly active in energy, where an interagency donor's group headed by the Bank meets twice a year and large scale cofinancing has been undertaken. The Bank has also assumed an important, but less formal leadership role in other sector--e.g., agriculture, population, finance and education. A case may be made that the Bank lacks the staff resources to lead sector work and financing in all sectors and that this burden should be shared with other major donors. Some discussions along these lines took place at the last Consortium meeting and it is possible that donors such as the Asian Development Bank and the U.S. will take a leadership role in coordinating work and financing in selected sectors in the future. ANNEX B Page 7 of 58 - 30 - The report is critical of technical assistance efforts and suggests the Bank assist in providing "... a more integrated, pooled mode of technical cooperation.' The UNDP is mounting a special effort to coordinate better technical assistance. The Bank does not have the expertise nor the staff resources, nor the mandate, to undertake a lead role with respect to technical assistance. Social, Institutional and Human Resource Development. The Bank needs to take a more active role in supporting Government efforts these sectors. The Bank has given special emphasis to social sector lending in Pakistan. It has made the largest primary education credit in its history in FY87 for a primary school education project principally directed at the education of females. Other education projects are planned. The Bank has been less successful with its population credit, but efforts are underway to lend in the health sector and incorporate a family planning element in a project in that sector. Efforts have begun to incorporate women in development components in various projects- -e.g., the agricultural credit and Afghan income generation projects. The Bank's urban projects have been directed at the needs of the poor. Future projects in rural electrification, urban development, rural water and shelter development all will support the Government's efforts directed at the social sectors. The social sector volume of the 1986 CEH and the Bank's sector work in education have also been instrumental in helping direct the Government's attention to social problems. Hopefully, the ongoing Bank sector work in population and women in development will produce similar positive results. The Bank should interest itself more in questions of local participation, centralization-decentralization, and federal-provincial issues. Except for the Afghan refugee income generation project, the Bank has not found the resources, the time, nor the opportunity to get involved in questions of local participation, or grass roots organization. The Bank's comparative advantage to work in these areas has also been questionable. The report correctly states that the Government of Pakistan has strong reservations about donor intrusion into federal provincial issues. This has constrained the Bank from having a direct involvement in the matter of federal-provincial financial relations, but the Bar..c has continued to exhort the Government to deal with these matters within the context of Pakistan's pressing fiscal problems. ANNEX B Page 8 of 58 - 31 - The report recommends a series of measures that the Bank should suggest to the Government to help mobilize additional resources. With the exception of the measure relating to federal- provincial financing relations, the Bank has conducted a dialogue with the Government on all of the measures suggested in the report. In addition, the Bank has gone further than the report by making resource mobilization efforts and reduced fiscal deficits conditions for expanded Bank lending. Institutional improvements affecting irrigation, water logging and salinity should be made to reap the benefits of the Indus Valley investments. The Bank irrigation rehabilitation project and proposed Agricultural Sector Loan will promote the development of private tubewells and the Bank lending program for water related projects has been conditioned on the principles of cost recovery. Bank financed projects have provided for institutional strengthening of provincial irrigation departments in order to improve the operation and maintenance of irrigation systems. An innovative agricultural environment project directed at protecting Pakistan's water resources is currently under preparation. Agricultural Credit. (i) The Bank has not faced up to its own criticism of the emphasis of the Agricultural Development Bank of Pakistan's (ADBP's) on mechanization in agricultural lending. (ii) The credit system is "... dependent on a continuous hemorrhage of donor and Government resources." The Bank's FY86 Sixth Agricultural Development Bank project addresses both of these problems. One of the project's principal objectives is to further strengthen ADBP as a sound, effective and self- sustaining lending institution. The project also aims to reduce loans for farm mechanization and diversify ADBP's portfolio. Agricultural pricing. The Bank's impact on the development and reform of the pricing system has only been moderately effective. Progress has been made in agricultural pricing. Host agricultural prices (at producer level) are broadly in line with border prices. Agreement was reached on the gradual removal of fertilizer subsidies in connection with the negotiations for the proposed Agricultural Sector Loan. ANNEX B Page 9 of 58 - 32 - Agricultural Research and Extension. There is a lack of effective articulation between national and provincial research and an excessive fragmentation of provincial research. The Bank's efforts to reform agricultural extension through the Training and Visit System has not appropriately taken into account Pakistan's agricultural and bureaucratic setting. The Bank's proposed Agricultural Research project for FY90 will address the agricultural research issues raised in the report. There is a need for an in-depth review of the extension system. Resources pernitting, the Bank may undertake such a review in the future. Rural Development, Land Tenure Reform, and Human Capital. The Bank has played only a marginal role in support of rural development programs and land tenure reform. It has also ignored the problems of rural education and health. The report correctly notes that the Bank has largely ignored the problems of rural development, land tenure and rural human capital in its programs. The Bank's FY87 primary education loan and its plarned education and family health lending will inter alia address the problems of the development of human capital in rural areas. The problems of land tenure reform are in a politically sensitive area, which the Bank has thus far avoided entering. The Bank's programs in irrigation, credit, research and extension work, as well as the planned FY90 rural water program will address other problem areas in rural development. Trade Policy. The report recommends various Bank actions, the most important of which are to urge the Government to rely more on tariffs and less on quantitative restrictions, and to rationalize the incentives provided by the trade and industrial policy regime. These subjects were extensively treated in a comprehensive FY88 Bank Trade Policy Sector Report. - The findings of the report have also been incorporated into the CEM, and the PFP/SAF process. The report has been fully discussed with the Government and many of its recommendations are likely to be incorporated in its economic program. Industry. There should be: (a) a more even flow of lending from the Bank for industrial development; (b) broader participation of intermediate credit institutions in Bank loans for industry; (c) more industrialization deregulation; and (d) a firm Bank policy stand on the Pakistan Steel Hill. The Bank has had a growing and continuous flow of resources to small, medium and large industry over the last three years. The Bank's Five Year Lending program provides for a continuation of this trend. ANNEX B Page 10 of 58 - 33 - Bank loans for industry now utilize various DFla and commercial banks as lending channels. The Bank did an industrial deregulation study which is discussed with the Government in FY88. The Government has made good progress in this area and many of the recommendations of the study have been adopted. Given the existence of the Pakistan Steel Hill, the Bank financed a study under the Technical Assistance credit which makes recommendations to improve the performance and efficiency of the Pakistan Steel Hill. Energy. No new recommendations were made. Conclusion. The OED report has been useful both in giving historical perspective to the Bank's operations in Pakistan and in suggesting how the Bank may operate more effectively in the future. The foregoing review demonstrates that many of the recommendations have been adopted. The report has had a particularly strong influence on increasing the Bank's involvement and support for social sector development in Pakistan. Its recommendations in the industrial, agricultural, trade and energy sectors coincided with Bank programs and initiatives that were underway, or being contemplated. In so doing, however, they have reinforced the Bank's actions in these areas. The report's concern with fluctuations in the Bank lending program is at odds with current Bank policy linking lending with macroeconomic performance. To the extent this policy represents a disagreement with the report's recommendations, it is one that has been adopted only after careful consideration and with an awareness of the risks of the discontinuities it may cause in Bank lending. The Bank has also not followed some of the report's recommendations with regard to headquarters/resident mission relations and functions. The role of the resident mission will, however, continue to be reviewed and the report's observations and recommendations are likely to be consulted again in the future. Although there have been many changes in the Bank, Pakistan and the world economy since the report was issued in January 1986, it continues to be reievant and important reading for Bank staff working on Pakistan. ANNEX B Page 11 of 58 - 34 - The Wortd Bank and Sri Lanka A Review of a Relationship OED Report No. 6074, February 24,1988 A seminar to discuss the OED report was held in January 1987 in the Bank in which Bank staff working on Sri Lanka and the current Senior Deputy Governor of the Central Bank (ther Alternate Executive Director at the IMF) participated. It also included other Bank staff who had previously worked on Sri Lanka. This seminar provided some useful discussions particularly on the content of the Bank's assistance program to Sri Lanka and a retrospective review of some of the key decisions made by the Bank. The discussion of the OED report with the country officials was scheduled to take place in July 1987. However, because of political disturbances it was canceled at the last minute. Subsequent attempts have not materialized primarily because of assignment of key officials to other pressing matters and the continuing political disturbances in the country. However, many government officials have read the report and the Bank staff have had informal discussions with them. The OED report has been useful both in giving historical perspective to the Bank's operations in Sri Lanka and in suggesting how the Bank may operate more effectively in the future. The review detailed below demonstrates that many of the recommendations have been adopted. The Bank has since then increased its involvement in policy based lending and provided a stronger focus on key sectors (agriculture and industry). Because of the Bank reorganization, the proposals in the areas of strengthening of field office and minimizing staff rotation need to be reassessed (see below). The recommendations in the report on desirable changes in the Bank's approach are categorized under two headings: (i) Improving the Cost Effectiveness of Policy Reform; and (ii) Increasing the Effectiveness of Interaction. Following are the actions that have and have not taken place in each of these categories: Improving the Cost Effectiveness of Interaction The report recommends focusing the Bank's program more clearly on the malor current and future problems of Sri Lanka. The FY88-92 lending program provides a heavy focus on issues in agriculture and industry sectors which the Bank and the Government agree are the most important areas where the Bank's assistance would be most effective. This focus is also supported by the Policy Framework Paper jointly prepared by GOSL, IMF and the Bank. ANNEX B Page 12 of 58 - 35 - Relating the total size of lending more explicitly to policy changes in the targeted sectors. The current lending program includes specific policy-based loans in the agriculture (Agriculture - Tree Crops Sector Credit, FY91) and industry (Public Enterprises, FY90) sectors. If the Government were to implement the recommended policy reforms in these sectors, it would generate substantial additional foreign resources not only from the Bank, bvt also from other donors as they are in agreement with the Bank on Lljor policy issues affecting these sectors. Reducing macroeconomic conditions attached to Bank lending. This could mean for instance, abandoning the condition that an IMF standby be signed, that there be general exchange rate reform and so on. We do not agree with this recommendation (and the OED report itself is not entirely consistent--see below). Our strategy has been to work with other donors in developing a macroeconomic policy framework. For example, the Bank and the IMF staff worked closely including joint missions in preparing the policy framework paper supported by the IMF's SAF. This collaboration is not for the purpose of enforcing cross conditionalities, but rather to ensure that investment and adjustment lending are undertaken in the context of a sound macroeconomic policy environment. Strengthening Bank work in the areas of concentration. One of the costs of borrowing from the Bank is the difficulty of dealing with poorly focused Bank missions. As the objective of the Bank's reorganization is to strengthen the country focus, this problem should be alleviated. However, we doubt that "poorly focused missionso was a serious issue in the past. Mobilizing additional resources from other donors in support of policy changes. See above. Allocating additional Bank resources to Sri Lanka if major policy changes are made could be a very effective step. We have been urging the Government to adopt major policy changes in several areas. Our FY89-91 lending program includes three structural/sectoral adjustment operations which constitute 332 of IDA's program for those years. If the Government were to adopt these changes, it would imply a substantial increase in transfer of Bank resources to Sri Lanka. Conversely, our lending levels would decline in the absence of policy adjustments. ANNEX B Page 13 of 58 - 36 - Increasing the Effectiveness of Interaction Continue low-key but reduced involvement during periods when conditions are not favorable for maior Bank initiatives. Our strategy has always been to work closely with the Government. However, in instances where we disagree on major policy issues, our activities are likely to decline which was the case during 1974-1976. Even in such instances we continued to work with the Government in the hope that there would be a breakthrough someday. At present the Bank's relationship with Sri Lanka is excellent. This positive environment has led to an active dialogue between the Government and the Bank. Strengthening the Bank office in Colombo, which needs to play a key role in keeping the Bank in touch with developments in Sri Lanka. The respective role of headquarters, missions and field offices must now be seen in light of the reorganization which the report did not and could not have contemplated. A Bankwide task force is now at work evaluating the role of resident missions. Any suggested changes would obviously have to be evaluated in terms of budgetary implications. Expand and strengthen Bank interaction at the highest political level of the Sri Lankan Government. Because of the excellent relationship with Sri Lanka, accessibility to the highest political officials is available to Bank staff when requested. Major policy issues are normally discussed with the key Ministers as well as the President. The proposed Public Sector Restructuring Project has been discussed with the President. Major concerns about economic performance have also at times been communicated to the President. However, it may be useful for us to make a greater effort in discussing key policy issues with a large number of Ministers. Reduce the frequency of rotation of Bank staff. It is difficult, if not impossible, for the Bank to participate effectively in policy dialogue until they are familiar with the Sri Lankan economy and the constraints under which the Government operates. By that time they may face rotation to another assignment. Valuable energy of senior Sri Lankan officials is consumed in educating successive generations of Bank staff. Staff rotations are not entirely controllable. There have been significant changes in key personnel assigned to work on Sri Lanka as a result of the reorganization. A more normal rotation pattern should now emerge. ANNEX B Page 14 of 58 - 37 - C. PROGRAM LENDING Bangladesh: Review of Experience with Import Program Credits I - X OED Report No. 6080, February 24, 1988 This is in response to your inquiry on the status of dissemination and feedback of the findings and recommendations of the OED Report (No. 6080-BD) on Program lending to Bangladesh, including how key issues identified in your report are being addressed through our CESW and lending operations. In particular, you requested information on the progress of four activities which were expected to address many of the issues raised in the OED report. The folloswing prragrapho outline progress achievpd to date in these areas, as wel) as actions taken by us to incorporate OED's findings and recommendations into our economic analysis and lending operations. Agricultural Sector Study. Regional staff advised you that a major sector study initiated in 1986 would help clarify agricultural sector issues and define IDA's future sector assistance strategy for the next decade. The study was initiated in FY87 as planned; but its finalization was hampered by the Bank's reorganization last year and subsequent dispersion of several staff members responsible for the study outside the Asia I Department. Secondly, the Government of Bangladesh itself embarked on a major review of its agricultural policies, with UNDP assistance, in late 1987. In order to avoid duplication of effort and to coordinate our sector work with the UNDP study, it was accordingly decided that (a) the preliminary analysis of our sector review would be made available to the UNDP team; (b) we would undertake additional sector work (which has already begun) to cover identified gaps in our analysis on factors constraining foodgrain production, particularly on the role of input-output pricing policies, institutional credit and support services and water (minor and major irrigation) managEment, and that (c) we, together with the Government, UNDP and other multilateral and bilateral donors would collectively conduct a joint review of completed analytical work in early 1989, to help define critical issues and an appropriate agricultural development and assistance strategy for the next decade. It has been further agreed that such a review would take place next spring within the framework of the Bangladesh Aid Group, either at a special session of the Aid Group or at a Local Consultative Group (LCG) Meeting. Financial Sector Study and Agricultural Credit. We have been seeking to address the problems of agricultural credit identified in the OED report and the more pervasive issues of widespread lack of financial discipline and institutional weaknesses in several ways. The Financial ANNEX B Page 15 of 58 - 38 - Sector Review completed early this year has identified critical issues and actions necessary to bring about a significant strengthening of the financial system. This study in fact will provide the analytical basis for a planned Financial Sector Adjustment Operation in FY89, which among other things will aim to address the issues of credit discipline, rest3ring the viability cf financial institutions, liberalizing interest rate policies and improving monetary management. Secondly, through our lending operations over the rast two years, we have actively pursued the issues of improving loan recoveries and financial discipline in both the agricultural and industrial sectors. For example, as a part of our Industrial Sector Credit, we reached agreement with the Government on collection targets for commercial banks and DFIs during 1987 in both agricultural and industrial sectors; and these targets were satisfactorily achieved. The Covernment also introduced a passbook system under which access to agricultural credit would be provided only to borrowers of good standing. While the Government has continued to implement this policy, the credit recovery drive led to negative net credit availability to agriculture, with some adverse impact on minor irrigation equipment sales and agricultural production in 1987. This 1.ssue, particularly its linkages with the minor irrigation strategy and pricing policies, would be addressed through our ongoing sector work on agriculture noted in para. 2 above. We also propose to support this approach with a proposed agricultural credit operation in FY90. Industrial Sector Credit (ISC) and Public Enterprises (PE) Autonomy. One of the objectives of the ISC was to improve PEs' operational efficiency through enhancing operational autonomy while establishing clearer accountability and performance criteria and improving central monitoring of their activities. Accordingly, the Government in 1987 initiated a UNDP-financed project for developing performance evaluation and financial MIS systems for public enterprises. A monitoring cell was set up in the Autonomous Bodies Wing of the Finance Ministry to implement thit project; and the Government selected two enterprises for a pilot exercise for implementation of the performance evaluation and monitoring system. While the pilot exercise was completed in only one enterprise in FY87, it was extended to three additional enterprises i FY88. The Government also promulgated a Public Enterprise Management Ordinance in 1986 designed to provide greater autonomy to public enterprises in setting prices. Notwithstanding these initiatives, however, actual progress in implementing these reforms has been limited. Improving the autonomy and performance of public enterprises war one of the objectives of the Industrial Sector Credit, which aimed primarily at rationalizing trade policy, liberalizing the investment approval process and improving the efficiency of the industrial credit ANNEX B Page 16 of 58 - 39 - system. During the preparation of the credit it was recognized that (a) the reform of the trade and investment approval system was a much higher priority, (b) the ISC could not effectively deal with too many issues, (c) considerable additional analytical work was needed to provide the underpinning of a more substantive effort to reform public enterprises, and (d) public enterprise issues could be addressed through a separate credit later on once the necessary sector work is completed. Accordingly, we have initiated in FY88 a review of public enterprises which would initially focus on generic issues of public enterprise autonomy, management, efficiency and employment aspects; a second stage of the study planned for next spring will evaluate specific problems at the enterprise level in selected subsectors. It is expected that this work will provide the basis for a more thorough reform of public enterprises which we hope to address through both a SAL in FY90 and a Public Enterprise Credit operation in FY91. Jute Sector Study. This study was completed in early 1986. Given the complexity of the issues and suggested recommendations, the Government indicated its desire for initiating a thorough internal review of the report and for formulating a strategy to address the identified issues. An inter-ministerial committee headed by a senior Member of the Planning Commission was set up for this purpose; and this review is expected to be finalized in July. The follow-up of this work within the Bank has been hampered by the recent reorganization of the Bank and the transfer of staff responsible for the initial sector work. However, following the completion of the Government's review in the near future, we intend to discuss with the Government the scope and arrangements for implementing the main recommendations of the report. D. RURAL DEVELOPMENT Rural Development World Bank Experience, 1965-1986 OED Report No. 6883, October 16, 1987 I refer to your memorandum of June 2, 1988, and wish to report on the dissemination, both within and outside the Bank, of findings presented in the OED Report on Rural Development - World Bank Experience, 1965-86. As the Report was formally published only in April 1988, it is as yet too early to assess its operational impact. I shall, however, comment on the anticipated follow up and informal feedback which PRE has received from the Regions. ANNEX B Page 17 of 58 - 40 - Dissemination. Prior to the Report being submitted to the Board in December 1987, AGR organized a seminar for Bank agricultural managers and staff to discuss the OED tindings and recommendations. The seminar was co-chaired by yourself and the Director of AGR and did much to sensitize operational staff to the issues. You will recall that it was pointed out at the seminar that, over the period covered by the report, the Bank had learned much about rural development and that many of the OED recommendations had already been incorporated into project design. Following the December Board presentation, AGR, in cooveration with COD and regional technical management revicwed OED recommendations in detail and prepared a management response which was submitted to the Board ia March 1988. In February 1988, the Report, together with an informal statement basee on the anticipated management response, was presented by OED and AGR to the Development Assistance Committee of OECD, at a seminar attended by the major bilateral and multilateral agencies in Paris. In May 1988, AGR presented the findings of the Report to a meeting of the UN Administration Coordination Committee Task Force on Rural Development, held in New York and attended by representatives of all appropriate UN agencies. The Asia Technical Department circulated its assessment of the OED Report, the management response, and ideas for future strategies for the Region, In April 1988. The Latin America Region convened a meeting chaired by the Vice President and attended by all senior staff and OED representatives to discuss the Report findings. As a direct result of this extensive dissemination of the report other agencies are determining their courses of action on rural development. For example, USAID has held a seminar to discuss a similar evaluation of its experiences. Follow-up Action. The twenty-one findings of the OED Report were addressed under eight headings in the management response. The current status on each is as follows: (a) Defining Rural Development Projects and Monitoring the Portfolio. This finding was accepted and a new definition was proposed. However, changing the existing definition has implications not only for the Bank in its reporting of poverty-lending but for other agencies which have adopted the earlier definition, and for modifying historical databases. A task force is to be set up in AGR to review the definition and an Operational Directive will be issued in FY89. ANNEX B Page 18 of 58 - 41 - (b) Government Commitment and the Importance of the Macro- Economic Environment. The Bank fully agreed with OED's findings which are already being incorporated into dialogue with Governments. (c) Resource Transfers and the Scale of Rural Development Projects. OED's views on separating rural development from the overall nature and size of country lending programs were not fully accepted; the preferred strategy is to adopt a long term programmatic approach which allows for flexibility in adjusting implementation as experience accumulates. This strategy is being followed in all Regions. (d) Adopting a Process Approach to Project Design and Implementation. Management felt that it would be undesirable to mandate a process approach but will consider an experiment with selected projecta to see if procedures require modification. (e) Closer Involvement of Beneficiaries. Direct involvement of beneficiaries is being fostered in several ways: through close participation in extension; establishment of water users associations; and involvement in on-farm research. The recent appointment of additional sociologists to the staff of PPR and the Regions is in line with the OED recommendation to strengthen the Bank's sociological skills. (f) Technical Assistance. Management agreed with the findings regarding the limitations of relying on long- term technical assistance for project implementation. Growing awareness of the need to set objectives according to the limits of local capacity is shown by the fact that extension initiatives in AFrica no longer rely on technical assistance. The implications of this approach on supervision needs, on increased training requirements and on implementation time frames has, however, still to be assessed. (g) The Role of NGOs. It is already the Bank's policy to encourage the use of NG0s. ANNEX B Page 19 of 58 - 42 - (h) Training of Bank and Borrower Staff. Management recognizes that new staff must have extensive exposure in the field to the problems of rural development. Training courses and seminars organized by PPR and the Regions will emphasize practical experience and lessons learned in on-going projects. Feedback from Bank Staff. Regional responses to the dissemination of the OED report have stressed that generic problems related to rural development are being gradually overcome but that it is unwise to lay down specific guidelines. Flexibility in approach is essential and the learning curve will continue for the Bank and for Borrowers. Situation specificity, at region and country levels, must be recognized and the issuance of inflex.ble directives would be counter- productive. E. COTTON DEVELOPMENT IN WEST AFRICA Cotton Development Programs Burkina Faso, Cote d'Ivolre, Togo OED Report No. 6878, June 30,1987 The OED report "Impact of Cotton Development Programs in Burkina Faso, Cote d'Ivoire and Togo (Bank Report No. 6878 of June 30, 1987) has been well read by those involved in cotton development in West Africa. The report was produced during a turbulent market period and agriculture divisions working in most West African countries were in a state of constant discussion and dialogue within the Bank, with governments and with donors, particularly the French. The views of the report have been debated by staff working on cotton in the various countries and the views of the report have been taken into account when developing improvements for the different social, political and administrative systems found in the countries concerned. The bottom line has been to reduce costs and improve profitability to increase African competitivity. In October 1987 the French Ministry of Cooperation called a meeting of donors to discuss cotton policy in francophone Africa. They had produced their own report and had copies of the OED report. The Region prepared its own position paper following a meeting of staff concerned. The meeting in Paris achieved its object of sharing experience and points of view (with the French, Germans, EEC, UNDP, IFAD, African Development Bank and FAO). It was not meant to come to conclusions as the complexity of each country's situation was recognized. ANNEX B Page 20 of 58 - 43 - Subsequently (and as a result of two years previous study) the Bank has prepared reports or projects on the Cotton sectors in Benin, CAR, Togo, Mali and Burkina Faso. Discussions on reducing costs have been held with the Ivorians and changes have been made in Chad and Senegal as part of our ongoing projects. No guidelines have been issued on the position paper but discussions have fully internalized the matter. However a divisional cotton paper is being prepared summarizing restructuring efforts in cotton producing Sahelian countries. The OED report has been timely and helped in a thorough analysis of the Cotton sector in francophone West Africa. The lessons are being spread to some of the anglophone countries of Africa where production has mushroomed again following the currency realignments, in particular Tanzania and Nigeria. F. COST RECOVERY IN IRRIGATION World Bank Lending Conditionality: A Review of Cost Recovery In Irrigation Projects OED Report No. 6283, June 25, 1986 Cost recovery in irrigation projects is a complex and controversial issue with which the Bank has been wrestling for many years. Bank policies are spelled out in a 15-page OPN 2.10 on "Irrigation Water Charges, Benefit Taxes and Cost Recovery Policies", written in 1980, which notes the multiple objectives involved (economic efficiency, income distribution and public savings), and calls for primarily direct cost recovery from the project beneficiaries, through water charges or their equivalent. However, implementation of these policies has been a problem in many countries, as the OED report points out, and this has led many Bank staff, as well as OED, to question these policies. The OED report was significant input into this debate. It received widespread and high-level attention within the Bank, including a memorandum from the SVPOP calling for special attention to this issue in the review of new lending, and the annual review of project implementation. He also noted that "additional operational guidance is necessary to clarify our approach to the various kinds of charges levied on farmers in irrigated areas, how these are to be counted in the total cost recovery objective, and the ways which have proven effective to ensure that financing for irrigation maintenance and investment is reasonable and sustainable." A work program on this was developed by the Agriculture Department, but then interrupted at an early stage by the reorganization. ANNEX B Page 21 of 58 -44 - The OED report made some specific suggestions which are consistent with Bank policies. (a) Whenever feasible, farmers' groups should be given a larger role in O&M. This is widely accepted in principle, and pursued in many countries. For example, recent irrigation loans in Brazil have provided for farmer participation on State Irrigation Councils. In Pakistan, supervision missions have been pressing the provincial implementing agencies under the second On-Farm Water Management Project to give the Water Users' Associations in the renovated watercourses a greater role in OkM. But in various countries, local cultural, bureaucratic and political factors are often a constraint. (b) Water charges should be indexed. This is generally accepted by Bank staff. Where it cannot be implemented, the Bank normally insists on an annual review of the water charges. But the two main substantive recommendations raise significant issuess (a) New loans should not be made if the O&M on existing projects was unsatisfactory. There is unanimity in the Bank on the critical importaiia of improving O&M: however, the decision as to whether at any given point this can best be achieved by refusing to make new loans until O&M is satisfactory, or by designing new projects to include more specific financial, organizational and material measures for improving O&H, needs to be made based on country conditions. But should O&M remain poor despite such measures, cessation of lending may well be necessary for the Bank to retain credibility on this issue. (b) Less emphasis should be given to direct water charges for cost recovery, and more to a review of the overall direct and indirect tax burden on the farmers, in the context of the sector dialogue. This recommendation has considerable merit, but the issues of equity, efficiency, public savings, and earmarking of funds for O&M need a fuller review. Any changes in policy cannot be implemented until OPN 2.10 is revised, as the OED recommendation is at odds with the thrust of that instruction to staff. While the OED report is far from the final word in the debate on the complex issues regarding irrigation water charges and O&M, its broad analysis of the Bank's experiences provides an important input to that debate. We intend to give this subject priority in the coming ANNEX B Page 22 of 58 - 45 - year, with a view to revising OPN 2.10 and providing clearer guidelines to the Regions on both cost recovery and how to proceed when O&M is unmatisfactory. G. LAND SETTLEMENT The Experience of the Wold Bank with Government-Sponsored Land Settlement OED Report No. 5625, May 1, 1985 AGR regarded OED's review of lessons from land settlement projects as an important exercise. From the outset, AGR cooperated with OED in selecting the consultant anthropologist, and collaborated further with OED in defining the terms of reference and assisting the execution of the study. Upon completion of the OED review, AGR took the following main actions: Dissemination and Staff Training (a) In November 1986, AGR's Sociology Adviser organized a Bankwide sociological seminar, at which the main author of the OED study presented the key findings of the review. The seminar was attended by a cross sample of Bank staff from all regions. (b) AGR organized two training courses, jointly with PERTH, for three days each in 1986 and 1987 in Annapolis for Bank staff from all regions on involuntary resettlement issues. During both training courses the findings from the OED study on land settlement were presented as part of the course. Specific cases discussed by OED, including the Jengka Triangle projects, were brought up in the presentations. In the first of these two courses, the main author of the OED study was again invited to lecture. Subsequent follow up has confirmed that the two training courses effectively influenced the operational work of participating staff. (c) A summary of the main findings and recommendations of the OED review was circulated, under cover memo from the Director of AGR, to all agricultural staff in the Bank. This was done in recognition of the unusual relevance and immediate importance to Bank operations of the sociological and economic findings of this review. ANNEX B Page 23 of 58 - 46 - (d) AGR is currently preparing a second edition of the volume "Putting People Firsts Sociological Variables in Rural Development", within which a chapter discusses the sociological model for land settlement projects. This chapter will now incorporate the main findings of the OED settlement review. In fact, the author of the OED study prepared this contribution to the book which is scheduled for publication in early 1989. Feedback into Bank Policy During the last two years, AGR has been engaged in rewriting the Banik's policy on involuntary resettlement. There are many similarities and common features between the regular land settlement operations and the resettlement of displaced people on alternative lands. In preparing the new OPN 10.08 published in October 1986 (entitled "Operational Policy Issues in the Treatment of Involuntary Resettlement in Bank-Financed Projects") AGR took into account the OED findings. In the recent World Bank Technical Paper No. 80 (May 1988) titled "Involuntary Resettlement in Development Projects - Policy Guidelines in World Bank-Financed Projects" the OED conclusions on land settlement experiences have been carefully considered and incorporated. As suggested by OED, new technical guidelines for Bank staff and borrowers have been prepared, consisting of a set of three instruments: (a) guidelines for preparing and appraising resettlement projects; (b) guidelines for the economic and financial analysis of resettlement components; and (c) guidelines for monitoring and evaluating resettlement operations. All three are published as annexes in the World Bank Technical Paper No. 80 on involuntary resettlement, available to staff and borrowers. The first printout of this paper (1200 copies) sold out in four weeks and a second one is being presented, testifying to staff interest in feedback. AGR advisers have systematically used the findings of the OED review in their work with regional operational staff on specific settlement and rural development projects. AGR and COD are envisaging the preparation of a new OHS during F189 or FY90 to formulate Bank policy principles, objectives and approaches in lending for land settlement operations. ANNEX B Page 24 of 58 - 47 - Proposals to OED In 1987 AGR proposed that OED should prepare a sequel to their evaluation of land settlement, in the form of a special OED evaluation study or synthesis of Bank experiences with involuntary resettlement operations. We understand that such a study might be included in OED's future work plan. AGR is prepared to cooperate with OED in this study as part of the intended PPR-0ED cooperation. It is probably adequate to use this opportunity to give additional feedback to OED regarding their work program. The review of settlement projects was one of the better OED reports in the last 2-3 years, because it went in depth in a precisely defined subsector. This formula is superior to the attempt of writing very broad-based reviews covering 20 sectors and 20 years at the same time, which lend themselves to much less precise and convincing conclusions. We recommend that OED replicate the pattern used in the settlement review, hiring good social scientists with proven analytical skills and field experience, to undertake similar analysis on a subsector by subsector basis and to focus on the social/cultural issues addressed less effectively in regular PPAR9. H. SMALLHOLDER LIVESTOCK The Smallholder Dimension of Uvestock Development A Review of Experience OED Report No. 5979, December 18, 1985 Upon your request, the following memo provides some elements on the dissemination and feedback of the above mentioned report and the audit report of the Senegal, Guinea and Zaire (Ituri) livestock projects. The Report "The Smallholder Dimension of Livestock Development' was widely distributed and disseminated to Bank staff (through two seminars for Bank staff) and to other donors, and data from the report were used at a number of international meetings. To my knowledge, no specific efforts were made to distribute the report to African decision makers. Its main conclusions in the technical field, e.g., shifting Bank support from beef production to the development of livestock as a part of the overall farming system and giving more emphasis to dairy, animal traction and other species than cattle, are being implemented in a number of projects (Chad Livestock, Ethiopia Livestock IV under implementation and Uganda Livestock, Cameroon Livestock III, and Niger Integrated Resource Management, under preparation). ANNEX B Page 25 of 58 - 48 - The Report's main recommendations in the institutional field, i.e., more emphasis to institutional weaknesses in the supply of services and the development of new technologies, again from a key element in present Bank lending. Indeed, the institutional aspects are central in the majority of the more recently appraised projects (Chad Livestock, Cameroon Livestock III, Guinea Livestock and Kenya Animal Health to mention a few). Animal production research is receiving increased attention in specific Bank-funded research projects, for example in Camero3n, Burkina Faso, Senegal and Kenya. The Report's plea for a general increase in the Bank's support for livestock development was not heeded. In spite of the Report's recognition of livestock as the best possibility for raising smallholder income and living standards and as an excellent instrument for poverty alleviation, and in spite of a general improvement in the performance of more recent projects, livestock lending has decreased from an average of 3-4 projects per year in the seventies to 1-2 per year now. The number of Bank livestock specialists has been halved over the same period. The combined audit on the Senegal, Guinea and Ituri livestock projects was circulated to staff involved in the projects concerned and discussed in a seminar with the consultant responsible for the audit. The PCR of the Senegal project (the PPAR had not yet been prepared) was one of the background documents in a workshop held for African livestock project directors at the Senegal project site, which provided a good forum for exchange of ideas on the development of traditional village livestock development. The audit reinforced the prevailing trends in livestock lending and stimulated Benk staff and especially managers in favor of livestock development through group formation, be it at the village level, as reviewed in the Eastern Senegal case, or at a village and regional level, as in the case of Ituri. This central PCR and PPAR theme or group formation now forms a key element of at least eight livestock projects in the AFrican region (i.e. Chad, Cameroon, CAR, Guinea, Mali, Mauritania, Niger, Senegal and Zaire). These groups are constituting a key element in the privatization of input supply (veterinary drugs and basic animal health care principally) of these countries, and are becoming a standard implementing unit in the new generation of resource management projects. Group formation and their role in the privatization of animal health care has also been generally accepted by African decision makers and other donors, following a series of meetings (Bujumbura (1984), Blantyre (1986) and Senegal (1986)] with African heads of veterinary services and other donors, in which Bank staff played an important role. ANNEX B Page 26 of 58 - 49 - Specifically, the Eastern Senegal PCR and PPAR results are helping in the design of the third phase project, which is expected to take village association and grazing management concepts of the Eastern Senegal project to a much broader scale. This, after the second phase (designed and appraised substantially before the audit and at the time of general dissatisfaction with livestock), had been significantly reduced in its livestock component. The Guinea PCR and PPAR helped in focussing the attention on the institutional constraint, which are now being addressed under the Guinea Livestock Sector Rehabilitation Project. The main recommendations contained in the Ituri PPAR and PCR are the need to further assist the regional livestock producer federation to integrate livestock more in overall crop development and to address marketing are reflected in the follow up project. In summary, the two reports have, each at their level, contributed to the thinking of those involved in livestock development. It has in this respect reinforced existing trends and actions, rather than open new approaches, as inherent in the audit process. In spite of their favorable recommendations, the reports have not, however, been able to arrest the decline in livestock lending. I. MONITORING AND EVALUATION Built-In Project Monitoring and Evaluation An Overview OED Report No. 5781, June 28, 1988 Although its title does not indicate so, the above study is limited to M&E experiences in the Agriculture Sector only. Dissemination of OED Findings 1. The draft was commented on in detail by OPS/AGR and Agriculture Sector project managers. The final report was also reviewed at these levels. 2. AGRME (the unit in OPS responsible for M&E in Agriculture) circulated a brief questionnaire to all Agriculture Operations staff in order to update the OED findings in the context of ongoing operations. Follow-up interviews were held with a sample of over 50 such staff in which the OED findings were discussed in relation to current M&E experiences. 3. Although the study was restricted to the Agriculture Sector, it was decided to prepare a management response covering M&E for all sectors. A task force was set up representing sector departments in ANNEX B Page 27 of 58 - 50 - OPS to review the OED findings in the context of experiences in other sectors. These sectors also conducted enquiries with a sample of relevant operations staff. 4. A Management Response was prepared and cleared by a meeting of Operational Vice-Presidents, chaired by the SVPOP, in January 1986. This Response was then sent to the Joint Audit Committee and discussed by the Committee in March 1986. 5. In accordance with the recommendation in the Management Response a multi-sectoral Working Group was established on which all sector departments and Regions were represented. This Working Group had the task to coordinate M&E efforts in future operations. For just over one year the Working Group functioned well and sponsored various training seminars, undertook a revision of the relevant OMS 3.55 and prepared a cross-sectoral roster of consultants. Feedback and Impact of OED Findings The main recommendations of the OED study were: 1. Basic concepts need clarification separating the monitoring function (indispensable) from evaluation (variable in scope and selective). 2. Revise Bank publications and directives accordingly. 3. Transfer AGRME functions to regions leaving central adviser in OPS. 4. Greater attention to M&E required during supervision. 5. Regional specialist to assist in upgrading M&E skills of general operational staff. Management regarded Recommendation 1 as having already been accepted and reflected in the widely available Bank handbook issued in 1982. However, greater inter-agency consistency has been achieved since 1982 and a two-volume publication by the Bank with FA0 and IFAD collaboration was planned. The message of these books would be even more emphatic than OED on the separation of the monitoring and evaluation functions. Recommendation 2 is partly covered by these books (both are now published and achieving wide dissemination and high sales). The Working Group undertook a revision of the OMS 3.55 and relevant OPNs. This work was overtaken by the reorganization, but will be reflected in the new OMS on supervision currently under preparation. The Working Group ANNEX B Page 28 of 58 - 51 - itself lapsed at the time of the, reorganization of the Bank in 1987 and the abolition of the Central Unit (AGRME) which provided the secretariat for the Group. Recommendation 3 was rejected in the Management Response which argued that the mix of skills required was such that each Region was unlikely to achieve the staffing level required. The central unit to be maintained for a further interim period with a broadened mandate including the provision of the secretariat for the Working Group (see above). However, the report of the Reorganization Committee accepted the OED finding: consequently the central unit was abolished and the Technical Departments were expected to make adequate provision for M&E skills. Africa and Asia Regions provided for one specialist in the Agriculture Division of the Technical Department. EMENA and LAC made no provision in terms of posts. Moreover, contrary to OED's recommendation no advisory post was retained in PPR so that there is no focal point for cross-regional coordination or the preparation of Bank-wide guidelines. As forecast in the Management Response it is considered that the problems of inadequate resources to advise on M&E and assist Operations staff (as discussed by OED) are now becoming more serious, and certain gains achieved in restructuring M&E efforts within projects are in danger of being dissipated. Recommendations 4 and 5 stem from the advantage OED sought by regionalizing the M&E specialist skills. As indicated above, after one year it seems that M&E is receiving less attention during implementation in many projects although progress has been made in certain specific areas such as development of micro-computer assisted management systems (Asia) and development of guidelines for M&E of extension projects (Africa). In the Africa region, the demand for staffweeks of M&E support far outstrips the weeks available from M&E specialist. The response of the division is to focus its available input on advisory and training activities for Bank and borrower staff (including the preparation of guidelines) and the coordination of consultant inputs. The responsible division in Asia is concerned at the lack of adequate monitoring and is undertaking a review of the needs. The other Regions agree that the regionalization of M&E resources has not worked for the reasons given in the Management response. In summary, the OED findings were extensively reviewed; existing training and publication programs reflected the findings where appropriate (which were in any case consistent with current trends); and directives are under review. But in one important area regarding the provision of advisory and specialist skills the regionalization proposal as accepted by the Reorganization Committee and as accepted (or neglected) by the Regions would, on the basis of one year of experience, ANNEX B Page 29 of 58 - 52 - be as seriously questioned by line managemeat today as it was at the time. M&E remains a somewhat neglected area of Bank input. J. ECONOMIC RATES OF RETURN Agricultural Project Economic Rates of Return: A Statistical Analysis Internal OED Paper, June 18, 1987 Dissemination The original dissemination of the above report was uneven. Not all agricultural Division Chiefs received it. Some circulated it to all higher-level staff in their divisions, others just to their economists, and still others not at all. The subtitle, "A Statistical Analysis", may have put off many potential readers more interested in reports with obviously actionable conclusions. It is also possible that the timing of tha report--coinciding with the most intense period of reorganization--may also have disrupted its dissemination. Recently, there has been some further dissemination, impelled by questions raised at the Board and by the circulation of an EAS paper, "Project Evaluation in Practice: Uncertainty at the World Bank", which broadened OED's analysis to include the other sectors for which economic rates of return (ERRs) are calculated. But, while relatively few staff have actually read the OED report, very many more have heard about its findings of (a) general over-o,timism at appraisal about ERRs for agricultural projects, and (b) a low correlation between appraisal and completion estimates of ERRs. Several meetings have been or will be held on the subject. For example, a seminar is being arranged by AFTAG and Division I of OED involving the agricultural divisions in the Africa Region, to review the main findings of the OED and EAS reports as they relate to agriculture in Africa. Reactions There have been a wide range of reactions to the report's findings. Based on the report, some staff concluded that we must do our economic analysis much more carefully and objectively; others felt that doing ERRs at all may be a waste of time. One memo called the report's findings 'important, dramatic and startling"; another considered them "well known ... by all experienced project officers and their managers.6 Still others questioned the statistical methodology and significance of the findings. Thus, the report can mainly be credited with stirring debate, rather than resolving questions. ANNEX B Page 30 of 58 - 53 - Further Analysis As noted above, the OED report inspired the author of the EAS paper to extend the analysis to other sectors. His draft in turn is being strengthened statistically, and supplemented by case studies (being done mainly by two summer interns in EAS and COD), in an effort to better explain the ERR gap. In addition, the agricultural component of the FY88 ARIS for the Africa Region is expected to consider explicitly the conclusions of both the OED and EAS reports. In a related area, AGR is carrying out a research project on the treatment of uncertainty in project analysis. Utilization The report made no recommendations, but nevertheless is having some impact on behavior. For example, one Division Chief noted that "a heightened sense of conservatism has ben induced in the economic evaluation of our projects" and another that "the OED report has heightened the awareness of the task managers for the need for a more realistic assessment of the project's economic impact ..." However, any major effect must await completion of further analysis that enables us to better understand the significance and causes of the OED report's findings. II. PROJECT EVALUATIONS The Bank's Regional Offices were requested to review the impact of OED audits in the context of a few salient project cases. The questions to be addressed were: (i) what dissemination efforts have taken place internally and externally in connection with these projects; (ii) what actual feedback, if any, has taken place in regard to research and policy, guidelines and operational impact; (iii) what has been accomplished with regard to the main conclusions, irrespective of whether OED or others in the Bank were responsible for originating the finding; and (iv) what fell short of being accomplished and why (for instance disagreement with OED's findings). The regional responses are shown below. ANNEX B Page 31 of 58 - 54 - A. Europe, Middle East & North Africa Regional Office Turkey - First Railway (Loan 0893) PPAR No. 5768, June 30, 1985 The PPAR on the First Railway Project (Ln. 893-TU) highlighted several failures in this Project, which could be classified under two broad categories: (A) those which stemmed from the institutional weakness of Turkish Railways (TCDD) and the Government's policy framework in which it functioned; (B) those which were intrinsic to project design and supervision. Significant failures under Category (A) were (i) late and insufficient tariff increases; (ii) unwillingness to set up a cost-based tariff structure; (iii) inability to provide a commercial orientation to TCDD's role; (iv) reluctance to make staff reductions, and (v) refusal to abandon uneconomic investments. The audit concluded that these serious violations of covenants can be traced to a lack of agreement in the early stages of the project cycle, between the Bank on one side and the Government and Railways on the other, about the objectives of the project and the means of achieving them. This misalignment of perceptions was the result of insufficient attempt to sensitize and commit the borrower and the guarantor to support the project as they should have. Under Category (B), the project was inordinately delayed due to severe procurement problems; it did not address adequately the operating-efficiency issues and the spare parts needs for satisfactory maintenance and it did not optimize the effectiveness and timeliness of consultants' work. In the Second Railway Project (Ln. 2739-TU) these concerns have been largely taken care of. The appraisal and supervision processes have brought about a better understanding by the Government (including the Ministry of Planning) and the TCDD of the policy and institutional reforms called for, by setting out their rationale and the modalities of implementation in a very detailed Action Plan. This time both the Government and TCDD management have been made to feel that this is their project and should therefore support it. The spare parts issue has been given full attention; selection and monitoring of consultants has been tightened up; procurement issues are resolved with utmost diligence, and only a six-month slippage is currently estimated. Nevertheless, this project is not without risks, perhaps of a different nature. Cout overruns appear inevitable, principally due to the drop in the value of the dollar. Readjustments of costs and shopping lists appear necessary. Traffic projections and financial performance are threatened by an inherent weakness in the couplings of ANNEX B Page 32 of 58 - 55 - TCDD's existing wagons. The problem has been diagnosed and a solution is available. On the whole, the feedback channel has been effective but with new risks cropping up, more perceptive and sustained supervision is necessary, as suggested in the last ARIS review. Morocco - Third Education (Loan 1200) PPAR No. 6845, June 28, 1987 Dissemination Efforts. The process of finalizing the PCR and the audit report for the Third Education Project in Morocco was long (the PCR was issued by the Region in April 1984 but the audited version, which highlights different conclusions was not issued until June 1987). Because of the extensive discussion, which took place between the EMENA region and OED during this period, all those working on the Morocco Education sector were well informed about the issues raised in the audited report. Feedback. Judging from the consonance between the following conclusions of the audited PCR (boldfaced below) and the directions of the lending operations subsequent to the completion of the Third Education Project in 1984, the feedback process appears to have worked well as follows: (a) "Special workshop facilities are not necessary for successful implementation of the primary school practical activities program.' COMMENT: The Morocco Education Sector Reform Program of FY86 includes a large scale primary school construction program which does not include workshop facilities. (b) uProject components should not be developed with new ministries without an extensive briefing of ministry staff as to Bank procedures and implementation expectations.' COMMENT: The Vocational Training Project of FY85--the first Bank financed training project with the Office for Vocational Training and Employment of the Ministry of Equipment and a small but important component in the Ministry of Planning--includes major borrower training efforts (involving both the UNDP and EDI) in project identification, preparation and appraisal methodology. ANNEX B Page 33 of 58 - 56 - (c) "Project components which venture into new project areas should only be developed if sufficient staff resources are available to assure adequate supervision, both on the part of the appropriate subject specialists provided by the Bank, and on the part of the Government agency responsible for inter-ministerial coordination." COMMENT: The two vocational training projects of FY85 and FY87 are sharply focused, have been well supervised by vocational trainers, and are being implemented on time. The Education Sector Reform Program of FY86 has also been well supervised by appropriate sector specialists from the Bank. See also comment under (b) above. (d) 'In the event of Government Administrative decisions which impair the achievement of project objectives, the Bank should be prepared to take firm action to encourage the Goernment to assure satisfactory Implementation and performance.* COMMENT: Following a 1987 Government decision, which jeopardizes a major objective of the Sector Reform Program (excessive student intake in higher education), the Bank cancelled its preparations for a sequel operation to that program. (e) 'The most important lesson for the Bank is that, despite the eagerness of the Government to expand and upgrade educational and training programs across a broad range of activities and the empathy that Bank staff may have for the need for such expansion and upgrading, projects should be restricted in size and scope to correspond to the Borrower's capability to implement efficiently and the Bank's capability to supervise properly. The numbers of ministries or agencies involved should be kept to a minimum [sic] of two or three if they are experienced with Bank financed projects; to a lesser number of they are not experienced.0 COMMENT: All Bank financed education and training projects in Morocco have since FY85 been implemented by one, or (in one instance) two agencies; all have been sharply focussed on 1-3 related major themes; and all are (despite budget problems, which impair disbursement performance) being rapidly implemented. ANNEX B Page 34 of 58 - 57 - Egypt - Cairo Gas Distribution (Credit 1024) PPAR No. 6860, June 24, 1987 Internally, the report has been studied by gas specialists and energy economists. In EMENA, the lessons learned have been taken into consideration in dealing with ongoing gas projects in Poland, Pakistan and Egypt. On the pricing issue, these projects are no more single- product oriented and will address the relative prices of other fuels. Furthermore, particular attention is given to the price structure in the subsector and tariff rationalization, if required, becomes a project condition. With the removal of price distortions, another issue raised in the Report, i.e., price discrimination against commercial users of gas, should not arise. Externally, the Egyptian authority responsible for the implementation of the project. Petrogas, has benefitted from the lessons learned. In this respect, the preparation missions for Egypt Gas Investment Project have observed that Petrogas has: developed the capability for safe operation of Cairo gas distribution; and established a joint venture to carry out the engineering and construction of the network as well as consumer conversions. The PCR should also have identified another lessons which relates to installation of gas piping in apartment buildings. Although the SAR had recommended that the building construction code be modified to make installation of piping a requirement in new buildings located in areas of potential gas usage, no action was taken. Consequently, a substantial portion (about 25 percent) of the cost of the ongoing network expansion project will have to be allocated for installation of gas pipes in dwellings. Tunisia - Sixth and Seventh DFC and Small-Scale Industry Pilot (Loans 1189/1504/1505) PPAR No. 6565, June 29,2987 The OED audit on Tunisia DFC raised a series of issues that, even if identified in the context of three specific operations (BDET 1189/i504/1505), are quite relevant for DFC operations in general. They have been part of the core subjects discussed in all recent financial intermediation projects in the Region focusing particularly on the following: ANNEX B Page 35 of 58 - 58 - (a) streamlining of approval and follow-up procedures required from participating DFIs; (b) improving the profitability of the loans to encourage the participation of commercial banks; and (c) enhancing the Bank's follow-up work and using lessons learned from DFC operations for Policy Dialogue. Also, these topics will be covered in the FY89 Sector Overview Paper that 91TTF will write through the year. Pakistan - Fourth Education (Credt 0892) PCR No. 6827, June 19, 1987 The Fourth Education Project, for which a credit of US$10 million was approved in April 1979, became effective in October 1979. The Credit was closed in January 1986. The PCR was prepared by Unesco, issued in June 1987, and the project was not audited by OED staff. This project was the first primary education project in Pakistan, and it was designed explicitly to be an experimental one. The lessons that were learnt during the implementation of this project have been incorporated in the design of the Second Primary Education Project, which was approved in June 1985 and became effective in January 1986. The lessons learnt from both of these projects have been incorporated in the Third Primary Education Project, which was approved in June 1987, and are being studied in the preparation of the fourth operation focussing on primary education, scheduled as a FY90 project. This note will focus on the impact of the conclusions and recommendations regarding this first project. Its impact is seen most clearly in the second and third project, but the relevance to the ongoing work will also be discussed. Project Background. This experimental project was to determine the relative contribution of selected inputs to improving quality and access in primary education. The inputs and questions to be studied were whether: (a) construction of additional facilities would reduce overcrowding and provide additional access; and low cost, experimental facilities could be designed; (b) provision of female teacher residences, and hiring of assistant teachers can aid recruitment of female teachers; (c) increased training of teachers resulted in improved pupil achievement? (d) improved supervision reduced teacher absenteeism; (e) increased contact between parents and teachers resulted in greater ANNEX B Page 36 of 58 - 59 - enrollment and less dropout; and (f) per student costs could be reduced through the use of assistant teachers, larger cidss and school sizes. The project was managed by a parallel structure of provincial implementation units (PIUs) and a federal coordinating unit (FCU). Management and supervision of the construction component, which accounted for half of the project costs, was given to the provincial public work departments. Project Impact According to PCR. The PCR reports that overall increased enrollment, reduced repetition, increased pupil-teacher ratics, and improved student achievement scores were achieved. Specifically, it reports: (a) severe managerial problems, with staffing gaps, and high turnover; (b) lack of proper supervision of construction, resulting in poor designs (the experimental designs were never prepared) and questionable quality of civil works, with which dissatisfaction was uniform; (c) an unsuccessful inservice teacher training program; (d) improved supervision and morale due to the hiring of learning coordinators and supervisors, resulting in reduced teacher absenteeism, better communication between teachers, parents, and administrators, resulting in better quality teaching; (e) provision of staff housing did not significantly contribute to increased female teacher recruitment, but provision of school boundary walls, hostels and recruitment of assistant female teachers did; and (f) inappropriate teaching and learning materials; nevertheless, the distribution of free textbooks contributed to increased enrollment and reduced wastage. PCR Recommendations. The PCR chapters on Conclusions and Recommendations, and Lessons Learnt note that: (a) more information on the appropriate, cost-effective input mix to overccme chronic primary education problems in Pakistan is now available; (b) project management needs strengthening, by removing the job insecurities and incorporating staff in the regular civil service. Civil works specialists who can oversee the construction component and also need to be added, or a private engineering consultant hired to carry oat all supervision; (c) the teacher training approach needs to be changed radically, giving responsibility to the existing teacher training institutions, and improving their methodology through a continuous, axperimental process in conjunction with curriculum development; (d) the learning coordinator and supervisor roles need to be more clearly defined in tasks, and communication channels with the regular structure, and a comprehensive investigation of the influence of the various ratios of supervisory staff to schools in the different provinces needs to be carried out; (e) curricula need to be modified according to local needs, integrated with training and supervision efforts; (f) action research capacity needs to be further strengthened either in the research cells or by contracting to universities; (g) monitoring needs to be greatly improved; (h) the ANNEX B Page 37 of 58 - 60 - provision of female teacher residences should not be abandoned, but fine-tuned to local circumstances, and (i) linkages with the Provincial Departments of Education should be strengthened. Diesemination. Although these conclusions and recommendations have not had any formal dissemination, many of the recommendations have been taken into account in the project design in subsequent projects, and as such, internal learning took place among Bank staff. Since reorganization, renewed efforts have been made to improve our understanding of primary education through in-depth sector work. This will support the preparation of the next (FY90) operation. Zmpact Learning input mix. Unfortunately, the PCR does not list what the optimal input mix is that was "identified, during the project. The experience in the second and third project suggests that elements such as better training, appropriate curricula, supervision, availability of female teachers, school buildings, learning materials all contribute in varying degrees depending on the local situation. Experimentation is, however, an ongoing process, and the Bank's approach at the moment in Pakistan tends to emphasize a more cautious, incremental approach to change. While the second and third project still attempt to improve all of the above elements simultaneously, the new project may only emphasize a few policy changes which we know to have a positive impact (e.g., provide funds in the recurrent budget for learning materials), learn from those, and then affect others. Project management. The second project continued with the parallel structure of project management, the PIUS and FCU, which were to be greatly strengthened. Links with the provincial education structure were also enhanced, as the district education officers (DEO), managers in the normal structure at the district level, were to be much more involved in project implementation. In reality, supervision missions have noticed very little real integration of "project-paid staff" and regular staff, although both do report to the DEO. Duplication of functions occurs frequently. The Departments of Education in the provinces do not regard this project as "theirs". And the PIUs and FCU resist the recommendations of the recent supervision missions to restructure the project in such a way as to achieve this integration. In the third project, project management has been incorporated into the regular structure of the Department of Education. Again, this was much resisted by FCU. ANNEX B Page 38 of 58 - 61 - With hindsight, the recommendation of the PCR to strengthen the PIUs should have been discarded. Instead, more attention should have been given to strengthening the regular structure through a process of incremental instit tion building and also with a view of assuring that the project would be suctained. The PCR notes very pointedly that there is little evidence that the conclusions reached under the project would be sustained. A parallel management structure (with power, because it controls funds) is not the best way to institute changes in the administration of the education system. Civil works. While the first project was meant to be experimental, its civil works component was in fact so large (constructing over 1,000 classrooms and some 300 other facilities) that its scope could not be called experimental. The extremely negative experience with the local design, construction and supervision capacity, led the designers of the second primary education project to experiment with a completely different concept, again on a large scale. Civil works were all to be prefabricated, procured through a turn-key contract, under ICB. Construction management was to be provided by a nationally recognized firm. Given the lack of experience with prefabrication in Pakistan, this approach, after a delay of two years, has again been modified. In Sind, the School Construction Unit of the Department of Education, which managed the construction component of the first project and increased its capacity meanwhile, has been asked to manage and supervise construction. Prefabricated structures are still to be constructed, first on an experimental basis at 500 sites, in the other two provinces. Under the third project, civil works are expected to be carried out by the Local Government and Rural Development Department, which constructs over 1,000 schools per year, and a newly created Civil Engineering Cell (CEC) in the Department of Education. Supervision is again by a nationally recognized consultant firm. With hindsight, Bank staff are now debating the need for the CEC to take over construction and maintenance of school buildings later in the project. The organizational arrangements for this component may well be modified. Teacher training. In spite of the strong recommendation to incorporate teacher training in the regular structure, exactly the same structure was continued under the second project. Despite some progress, problems continue. Teacher training was designed to be integrated with curriculum development under the second project, but because this process of curriculum development is in fact also taking place in a parallel structure, divorced from the regular institutions dedicated to curriculum development, teacher training organization remains problematic, with difficulties in integrating training to actual classrooms developments. ANNEX B Page 39 of 58 - 62 - Supervision structure. Based on the positive evaluation of the learning coordinator, and supervisor, these two layers of civil servants are being added nationwide. Recent supervision missions are now calling for the in-depth evaluation of the contribution of these new staff members, as they add to the recurrent costs of the education system, and the observed problems of ill-defined tasks and roles have not yet been solved. Anecdotal evidence suggests that there is wide variation in their effectiveness, but theoretically the concept is sound, and it seems to solve a problem in the bureaucracy of having overburdened DEOs who cannot give professional guidance to teachers. The concept needs much more careful evaluation and fine-tuning during future supervision and preparation missions. Curricula. The recommendation to improve curricula has been taken very seriously by the designers of the second project. Again, the pendulum swung to wholescale experimentation with an unproven method. The concept of learning modules was introduced. While in theory a concept that could improve learning and teaching in Pakistan, it was not a known method, and it was not being considered by any other provincial curriculum bureaus or the Federal Curriculum Wing. These organizations responsible for curriculum development in the country were also not formally involved in the second project. Unfamiliarity with the concept, unwillingness to attract foreign technical assistance, etc., have led to delays in having any approved learning modules. These learning modules are now being tested in the classrooms and evaluated by the provincial curriculum bureaus and federal curriculum wing. It remains to be seen whether they will be incorporated at any larger scale, but at least this gives the proper responsible organizations the chance to decide on the merits of the concept. In contrast, the third project supports both teacher training and curriculum development activities of the provincial organizations responsible for these matters, and attempts to improve the on-going activities by supplying additional financing and technical assistance. Learning materials. While the provision of free textbooks was evaluated positively, especially as a measure that contributed to greater female student enrollment and retention, this has not been replicated under follow-up projects. The province of Baluchistan has a policy to provide free textbooks as an incentive; under the third project, a donation of paper by Canada is expected to decrease the price of books. This is one of the policies which Bank staff intend to discuss as part of the next operation, as research worldwide shows that the provision of textbooks, more than any other factor, is correlated with higher student achievements. A survey of Pakistani parents also indicates that the price of books may be a deterrent to send children to school, and this is more so for girls. ANNEX B Page 40 of 58 - 63 - Action research. Strengthening of action research capacity was to take place along the lines suggested by the PCR: more staff and technical assistance to the FCU and PIUs. In reality, this has not happened. For the third project, contracting out the research is actively being pursued by supervisien missions. Monitoring. Enhanced monitoring capacity of PIUs and FCU was built into the second project. The third project also includes funds for a complicated monitoring design. A recent supervision mission, in collaboration with a USAID funded action research program for the development of MIS, has attempted to scale down the elaborate designs to a minimal, useful and manageable system that is coordinated throughout the country. Female teacher residence. The PCR recommends to fine-tune the construction of female teacher residences to local needs and cultural patterns; instead it became common wisdom that such residences are a failure.. Neither the second or third project provides female teacher residences. Yet, during tha preparation of the new operation, providing female teacher residences, properly constructed in safe places, is again being considered as part of an incentive package. The construction of boundary walls, in contrast, has become a standard feature of school construction programs, financed by the Bank or locally. Conclusion The PCR has certainly been useful in the design of follow-up projects, although not all its conclusions are now accepted by the staff. Experience gained in the design and implementation of the Fourth Education Project and the two ongoing projects will be used in the design of the next operation. B. Africa Regional Office Cote d'Ivoire - First Telecommunications (Loan 0981) PPAR No. 6874, June 26, 1987 Ghana - Kpong Hydroelectric and Third Power (Loans 1380/1381; Credit 0689) PPAR No. 5731, June 25,1985 Kenya - Third and Fourth Education (Loan 1184; Credit 0797) PPAR No. 6712, Apri 15,1987 Mali - Liveslock (Credit 05338) PCR No. 6846, June 25,1987 Nigea- Lafia and Ayangba Agricuftural Deve1opment (Loam 1454/145) PPAR No. 5972, December 16, 1985 ANNEX B Page 41 of 58 - 64 - The Annex summarizes the answer to the questions mentioned in your memo of June 3, 1988 namely (i) dissemination inside and outside the Bank, (ii) actual feedback, (iii) accomplishment and pessible shortfalls for the projects cited above. The results confirm that PCRs and PPARs are systematically reviewed by Regional staff who have participated in the project at hand and/or are responsible for the sector and the country concerned. PCRs and PPARs are circulated within the issuing SODs and to the relevant TD Divisions. Managers and staff in a given sector will continue to play a key role in disseminating findings across countries and departments through direct involvement in different countries and through guidance and professional exchanges. Dissemination in the Bank across country groups and across sectors relies at present on TD staff, on COD Project Advisors, and, at the RVP level, on the Senior Operations Adviser. The Country Team may also provide a useful forum for dissemination of key findings across sectors in a given country. PCRs and OED findings are also disseminated in the respective country with project agencies and with relevant Government departments through discussions for the preparation of follow-up operations and in the course of supervision of such operations. As far as feedback is concerned they have not found substantive disagreement with OED findings in the Region. The record of acceptance within countries is also good. The sample suggests the PCRs and OED findings cover substantive and useful recommendations concerning project design, institutional matters as well as in sector policies of direct relevance to the project at hand. The findings of PCRs and PPARs are systematically backfed into follow-up operations. In many cases, the lessons learned from a given operation emerge through supervision before the issuance of the PCR. OED findings are also incorporated in ongoing projects through supervision and have been influential in determining the scope of thematic supervision efforts. Finally they have pointed out areas of policy deficiencies related to implementation issues complementing PIRs which have a shorter term perspective. Thematic Reports OED thematic reports are systematically circulated in the relevant sectoral groups (SOD3 and TD) and are the object of careful review and coments. These reports have been the object of seminars in the Bank (MADIA) and with donors. The West Africa Cotton Report was an important input for a special donors meeting called by the French ANNEX B Page 42 of 58 - 65 - Ministry of Cooperation. The lessons learned are being spread to other African countries (Tanzania, Nigeria). The impact of OED Reports on operational policies has been substantial; they have supported major policy reorientation in particular in rural development lending. These findings have been disseminated through internal fora and subsequent policy papers. The conclusions have generally been internalized, although sometimes with a time lag. Suggestions The key requisite for the effective dissemination of findings of PCRs and PPARs across country groups and across sectors is greater ease in communication. Shorter and issue-oriented PCRs would be a first step in the right direction. A brief Region specific annual compendium by OED on the PPAR/PCRs and other OED findings by sector and generic topic would be useful. o PCRs AND PPARs 144 o Ghana Kpong Hydro go C" and ECO Third Power Cots d'Ivoire Telecom. Mail Livestock -00 Dissemination effort - In-depth review by Regional - Review by Regional Staff; - Review by Regional 4, ft and A4 Internally Staff; findings discussed findings discussed by Consultant involvedz Lesson externally with project entities VRA & project entities ONT and learned already incorporated ECO. relevant Ministries. In follow-up project; subsequent sectoral policy recommendations taken up with Govt. In the framework of SPN. - Feedback research and - DED findings and - Recommendations accepted. - Recommendations accepted. policy, guidelines and recommendations accepted. operational impact - No well built without prior participation of - Findings of PCR and PPAR - Special attention to beneficiaries; Drugs are - What has been accomplished Instrumental in shaping implementation capacity and sold full cost with with regard to main sector policies and procurement; action plan to comission for herders who conclusions investment program through improve ONT's accounting and do the retailing; Input Into Power Rehabilitation financial system; management broader policy paper on $a Project; re-orlentation t improvement program for ONT. 'Livestock In West Africag. * better utilization of existing capacities rather than expansion and consideration of thermal option. PCRs AND PPARs 0 Nigeria Lafla and Ayangbe Kenya Third and Fourth Education Projects on - Dissemination effort - Review and coments by Regional Staff; - Regional STaff; DED findings disseminated N internally Findings discussed with Government through discussions with Government. externally Officials and agency staff FACU and ADP. Two sectoral consultations were held. - Feedback research and policy - Recommendations accepted; In the country - Recommendations were accepted but focus of guidelines and operational feedback positive. sector dialogue has now shifted to sectoral Impact pol icies. - What has been accomplished - DED findings being Implemented through - All recon dations have been addressed with regard to main supervision In particular; unification of under Fifth and Sixth Education Projects. conclusions extension services; emphasis on users Major OED recommendations in the PPAR cover charge and community/beneficlary (a) measure to accelerate use of specialist maintenance for rural water supply; services; corresponding allocation privatization of water supply; introduction disbursed 901 and 501 under follow-up of thematic superviion strategy. projects; (b) PIU (Project Implementation Unit) has been strengthened and coordination improved; (c) the adequacy of output of agricultural technicians being reviewed by a Presidential Working Paper. ANNEX B Page 45 of 58 - 68 - C. Latin America & the Caribbean Regional Office on the general issue of feedback of the results of evaluations of individual operations under the new organization, the Technical Department is responsible for dissemination of such findings to LAC sector staff. Our initial activity in FY88 has been to disseminate the main findings of the PCRs on Bank operations produced in the previous year. This was done normally by circulating a brief note to the responsible country department managers; in some cases other vehicles were used, such as a discussion at a sector retreat or a section in a regional sector strategy paper. We continue to regard early dissemination of PCR findings as a high priority activity since our implementation reviews show that managerial, institutional and financial problems are the leading cause of poor project performance. We feel that LAC staff can learn more about addressing such problems by keeping current on experiences in other LAC countries. We plan to extend this dissemination process in FY89 to include new material in PPARs (i.e., significant points not covered by the earlier years' dissemination of PCRs) and OED's annual reviews. We would rely on the latter to summarize salient points from the experience of other regions. Once the above system is working smoothly (which it should by the end of FY89) LAC would have a reasonable collective awareness of the lessons of the PCRs and PPARs in each sector: the Senior TD Advisors in each sector who produce the summaries or conduct dissemination meetings, SOE managers and senior sector staff in the CDs would have participated. These key individuals could pass on their knowledge as necessary to newer or more junior staff in the course of operational work. The Senior Operations Adviser is considering the extension of the coverage of this system to include non sector-specific operations (there have been relatively few of these in LAC so far). The particular operations mentioned in your memorandum are reviewed below. It must be stressed, however, that, with the exception of Panama, these were all handled under pre-reorganization procedures, with responsibility for dissemination being shared by the former regional Projects Divisions and the respective Central Projects Department. Under the previous organization dissemination was much less of a problem since we had larger and more homogeneous groups of sector specialists who found it easy to keep current with the implementation of LAC projects in countries other than those they worked on directly. The absence of formal arrangements at the time should therefore not be taken as an indication of a lack of effective dissemination. ANNEX B Page 46 of 58 - 69 - Panama - COFINA Development Banking Projects (Loans 1641 and 1955) PPAR No. 3062, December 22, 1987 This PPAR (circulated in 1988) contains much interesting material and will be the subject of a LAC seminar in September. Colombla - Las Mesitas Hydroelectric (Loan 1628) PPAR No. 6638, January 20,1987 The main lessons in the PCR and PPAR relate to: (a) The need for more thorough geological prospecting for hydroelectric projects and the follow-up of execution by a Board of Consultants; (b) The importance of paying special attention to the capability of management and of the organization to handle all its operations, in addition to the project; and (c) The need to improve Bank staff continuity in handling projects. Our experience with this and other projects supported by the Bank in Colombia and Central America at about the same time, which encountered Eavere cost and time overruns, has persuaded us of the need for particularly rigorous geological investigation and for close follow- up during implementation where similar conditions are present. The second lessons applies to most Bank operations. Dissatisfaction with the handling of institutional capacity in our lending has resulted in a LAC program to strengthen the ability of sector staff in this area over time. The third lesson is equally applicable to most operations, although some, like loan 1628-CO, may require particular attention to this aspect. The abolition of the previous guidelines for maximum stay in a VPU vill help in maintaining continuity. It is unclear whether the new organization will, on balance, help or hinder it. Proposed criteria for promotion to senior non-managerial positions which emphasize cross- regional experience would contribute to increased rotation and reduced continuity. The leveliag-off of growth in the Bank's staffing should work in the opposite direction. ANEX B Page 47 of 58 - 70 - Brazil - Paulo Alfonso IV Hydroelectic (Loan 1009) PPAR No. 6578, December 31, 1988 The main lessons in the PCR and the PPAR are: (a) The inadequacy of technical appraisal based on feasibility-level studies for this type of project; (b) The need to plan resettlement appropriately and well ahead of the initiation of works, and to give consideration to carrying it out through a separate--but not independent--project; and (c) The need to provide adequate resources (in terms of time and skills! for supervision, taking into account the size and complexity of projects. As the PPAR points out, the first lesson (derived from similar experience with many earlier projects) led in 1978 to the change in Bank guidelines on the readiness of such projects which require detailed designs to be done prior to loan approval. We continue to learn more about resettlement, but there is general agreement that Bank practice has improved substantially. A recent example is the $142 million Itaparica resettlement project (Loan 2883-BR) where the Bank financed resettlement required by a hydro project downstream from the Sobradinho area referred to in the PPAR, as a free-standing operation. The Bank was not involved in the financing of the hydro project, but it was approached by the Brazilian Government for support in connection with the resettlement, following delays in its preparation by Brazilian agencies. The Executing Agency for the loan is the same regional power company, CHESF, which was responsible for loan 1008-BR. A considerable effort went into analyzing with CHESF the lessons of that loan with a view to avoiding unnecessary hardships, especially for the portion of the population being resettled to rural areas. More generally the Bank also introduced specific conditionality on resettlement in the Power Sector loan to Brazil (Loan 2720-BR). Budgeting for supervision continues to be tight, but the new organization has clarified responsibility for budget allocation decisions--country department division chiefs and directors are in charge of making tradeoffs between supervision and other uses of resources available for country work. The extent to which this results in improved supervision will depend on the vigor with which higher levels of management exercise their oversight, and the degree to which they reward good supervisioa as compared with lending and ESW. ANNEX B Page 48 of 58 - 71 - Mexico - Technical Training Project (CONALEP 1) (Loan 2042) PPAR No. 7136, March 1, 1985 This project was evaluated as highly successful by OED as well as in the PCR, and its evaluation provided a number of interesting general conclusions concerning, inter alia, the major benefits to project design and implementation accruing from (a) basing a project on a thorough sector study; (b) concentrating implementation in one well managed agency; and (c) selective but continuous and carefully staffed Bank supervision. Evaluation results for this project have been widely disseminated among sector staff through otal presentations and circulation of the PPAR and (earlier) the PCR. Dissemination was facilitated by the interest generated among staff by the recent preparation of another large training project in Mexico (the Manpower Training Project Loan, 2876-ME, which is partly linked to CONALEP) and during several discussions involving PHREE and sector staff from all regions to identify interesting experiences upon which to base the forthcoming Bank policy paper on vocational education and training. Project experiences have also been disseminated beyond the Bank, including presentations by the Director-General of CONALEP and Bank staff at the annual meeting of the International Vocational Education and Training Association (IVETA) in December 1987. The lessons learned have been fully incorporated in the follow-up project (CONALEP II, Loan 2559-ME) and in the Manpower Training Project mentioned above. Innovative and successful features of the CONALEP system, such as the standard process for initiating substantial changes in training programs, have been described and recommended to borrowers in other countries. D. Asia Regional Office India - First National Dairy Project (Credit E24-IN); State Level Dairy Projects In Karnaaka, Rajasthan and Madhya Pradesh (Credits 482, 521 and 522) PPAR No. 6857, June 30, 1987 Draft PCRs for the above projects were prepared by Government in association with the Bank and FAO in 1986, and OED visited the projects in May 1986. While a draft OED report was not available until early 1987, OED findings were discussed with the preappraisal mission for the Second National Dairy Project prior to departure of the mission (from the Initiating Executive Project Summary stage). During the preappraisal mission the revised draft PCRs and the uOverviewO of the ANNEX B Page 49 of 58 - 72 - four PCRs prepared by the region were given informally to Government and were thoroughly discussed. Dialogue referring to PPAR/PCR findings was continued throughout the appraisal process and we found the package a useful supplement to mission field findings especially as a reference source in the dialogue on cooperative sector policy. There were no major differences of opinion between the appraisal mission and PPAR findings. Central Government and the project implementing authority, the National Dairy Development Board (NDDB), also agreed with the PCR/PPAR main conclusions. The Chairman of NDDB, advised us verbally that he was particularly supportive of the report's conclusions on necessary policy reforms at state government levels. These concerned the need for creation of independent, farmer managed cooperative businesses, free from government control, owning their own assets, managed by elected representatives and including appropriate cooperative by laws and business management procedures. Central Government also backed up this approach and Dr. Kurien's letter to OED included a copy of a letter from the Prime Minister to all stage governments providing support to this policy direction. NDDB also accepted the PPAR's opinions on technical and management adjustments of the "Operation Flood' (dairy development) program. The PPAR and PCRs have thus been an operationally very useful input during appraisal and negotiation of the Second National Dairy Project. The findings support the main thrust of the project which relates to dairy cooperative policy adjustments plus improvements in some technical and management aspects of the program. PPAR conclusions are also extensively reviewed in paras. 1.20 to 1.25 and Annex 3 of the SAR (Report 6897-IN of November 18, 1987). This PCR audit is thus a very positive example of an exercise that has influenced project design of a follow-up project right from the identification stage. It should nevertheless be pointed out here that a number of state governments are not fully in agreement with the audit and appraisal recommendations on cooperative policies, particularly in the area of complete farmer control. A progressive shift in such thinking at the levels of individual states is thus the Second National Dairy (ND II) projects's main thrust and is backed up by institutional and cooperative policy onlending conditions to be applied by NDDB during appraisal of subprojects. Policy adjustments will take time and we will need to be realistic about how much will be achieved in some states by ND II project completion. Initial progress, however, is encouraging. Beyond the specific project related dialogue with the Indians, assessment of the full impact of the PPAR and appraisal exercise ANNEX B Page 50 of 58 - 73 - findings is more difficult and may be premature. Findings are consistent with and in part have helped influence India Department orientation concerning cooperative development, emphasis on private sector management principles and recognition of the limits of Governments' role in development. The PPAR was also provided to the EEC which is providing commodity aid supporting Indian dairy development. Some queries concerning key features for successful cooperative or dairy development have also been received from other Bank departments. Central exercises such as the one to which this note responds may also serve to disseminate feedback. India - Railway Modernization and Maintenance Project (Credt 844) PPAR No. 7035, November 3,1987 This account focuses on specific follow-up dissemination by staff working on Indian Railways. The PPAR's conclusions may well have influenced the design of railway projects in other countries, but we have not way of monitoring that. The PPAR drew three major conclusionaz (i) the audited project was the first specific project loan to IR; previous operations had been general sector/program loans. The PPAR concluded that this attempt to effect specific changes, while meeting with mixed results, was a more appropriate format than the earlier operations; (ii) wile the modernization of five pre-specified workshops was probably justified, the more general objectives of overall workshop rationalization and introduction of unit exchange components were not attained; and (ii;) the principal reason for the latter was the failure to assess beforehand the labor redundancy implications of these initiatives. These conclusions were discussed with Indian Railways and influenced (i) the approach to supervising an ongoing follow-up project in workshop modernization and (ii) the design of a new project in track modernization. On (i), the Eank engaged consultants to undertake a detailed review of the status of overall rationalization and introduction of unit exchange spares, and recommend how these objectives might be better implemented. These reports are just to hand and will be discussed with Indian Railways during the next supervision mission, ANNEX B Page 51 of 58 - 74 - scheduled for September/October. On (ii), IR requested the Bank to finance, in a new project, modern tracklaying and maintenance equipment to replace existing more labor-intensive methods. The Bank requested, and obtained, a statement from IR on how the labor implications of this component would be dealt with; at negotiations, agreement was reached on arrangements to monitor this issue (see SAR $7047, April 15, 1988 para. 2.08). More generally, IR agreed to the Bank's suggestion to include a manpower planning component to help IR assess the labor implications of future technology and other influences (e.g., changing traffic composition) and identify appropriate adaptive measures (ibid., para. 3.11). India - Singraull Thermal Power Project (Credft 685) PPAR No. 6784, May 14,1987 Copies of the PPAR were distributed to the concerned authorities in the Central Government through the Department of Economic Affairs of the Ministry of Finance and to the Borrower, the National Thermal Power Corporation (NTPC). Copies were also distributed to the staff in the former South Asia Energy Division (part of which stayed in the new Transport and Energy Division, India, after the reorganization of 1987). The report was, however, not discussed in any formal meetings with the Government, the Borrower or the staff. The reason for not discussing the PPAR in a formal way was probably that the implementation of the project was successful and the project objectives were substantially achieved. As indicated in the PPAR, "the project is an excellent example for the association of success with the traditional virtues the Bank persietently advocates in connection with public utilities projects" (para. vi, page vi of the PPAR). The fact that there were no formal discussions of the report does not mean, however, that the lessons learned and comments contained in the PPAR have not made an impact in the design of subsequent projects. The most important items raised in the PPAR which have helped a more focused approach by the Government and by the Bank are: (a) the potential cumulative environmental effect of massive thermal power developments combined with concentrated industrial activities. This aspect was not properly assessed at the appraisal stage. Presently environmental impact studies are being carried out under the Second Singrauli Project with a broader scope than originally envisaged. The studies would also tackle the issue related to resettlement of population which the appraisal report did not treat in sufficient detail. ANNEX B Page 52 of 58 - 75 - More recently, in projects like the Talcher Thermal (in early phases of implementation) and the Bihar Power Project (under preparation), the Bank and the Borrower have taken a much more detailed approach to these aspects; (b) the estimation of project costs and particularly of local costs. This aspect has been substantially improved in recent projects because of: (i) a better knowledge of the local market through subsequent projects similar to Singrauli; and (ii) a much more detailed engineering being raquired before appraisal can take place. The Department is presently requiring project engineering to be carried out at bidding level as a minimum before appraisal; (c) the underestimation of time required to process .procurement. For essentially the same reasons given in (b), estimates of time needed for procurement processing are now more realistic and a very detailed procurement planning is presently being carried out between the staff and the Borrower as a pre requisite of appraisal. An excample of this detailed planning can be seen in the recently approved Uttar Pradesh Power Project; and (d) the threat to NTPC's financial stability posed by the increasing unpaid bills owed by the State Electricity Boards (SEBs). This is being attacked in two fronts: (i) through provision in the operations with the SEBs as for instance the recently approved Karnataka and Uttar Pradesh Power Projects. In these projects appropriate conditionality has been introduced in terms of the acceptable level of unpaid bills and of the securities to be established in favor of NTPC to guarantee the payments; and (ii) through covenants in the agreements with NTPC for recent projects regarding the maximum allowable level of accounts receivable. This matter is now subject of close monitoring by the India Department. ANNEX B Page 53 of 58 - 76 - Bangladesh - First Popilation (Credit 533) PPAR No. 8303, June 30, 198 FINDINGS - OED AUDIT FEEDBACK FROM GOB, IDA AND COFINANCIERS 1. Overriding concern on cofinancing 1. Cofinancing is an effective way at the expense of adequate to narrow down ideological attention to project design and perception and evolve a common dialogue with Government. strategy. 2. Program and project design and development of the Second Population and Third Population and Family Health Projects were participatory efforts between Government and donor agencies. 2. Number of separate projects in Execution of project affected lieu of the complex multisectoral more by weak bureaucracy, project would improve effective- outmoded governmental procedures, ness. etc., than by complexity of project. 3. A gradual and more focussed Magnitude of population problem approach, in lieu of present calls for maximum possible input; projects' too large scope and project design based on best size. known measures to control fertility; continuous monitoring and feedback and an action research approach adopted to improve effectiveness of the project. 4. Delegation of more authority to Accepted under continous review. resident population office. 5. Smaller and more focussed super- Being adopted. Donors are shar- vision mission. Delegation of ing supervision responsibilities supervision responsibilities to to an increasing extent. Annual cofinanciers. Review Mission is large because of numerous project cofinanciers. 6. Underutilization of facilities. Facilities being monitored and measures being discussed and adopted by Government to reduce underutilization. 7. Poor maintenance. Remedial action initiated. Main- tenance component build into Third Project. ANNEX B Page 54 of 58 - 77 - FEEDBACK FROM GOB, IDA AND COFINANCIERS FINDINGS - OED AUDIT Not accepted by IDA. Present design is considered appropriate for current and future needs. 8. Review of facility design particularly Family Welfare Training capacity under constant Centers. review; capacity not large compared to the need to upgrade 9. Overcapacity in training a knowledge and skills of the large moratorium on construction of work force through refresher training facilities is needed. courses, which had only short initial training. Periodical changes in program content - MCH and FP necessitates additional training. Accepted - Remedial actions, worked out with GOB. 10. Multiple administrative reorganizations, sometimes with Accepted - Necessary modifica- detrimental effects. tions introduced in Third Project. 11. Information, Education and Com- munications activities to be more consumer oriented. Incorporated in designs of PCR subsequent projects. 12. Conscious choice of project complexity - steps taken to increase implementation capacity Accepted. Incorporated in Third in line with that complexity. Project with respect to Family Welfare Assistants and women's 13. Testing of project elements programs. before their large expansion. Accepted. Incorporated in design of subsequent projects. 14. In social sector, infrastructure and program development should Accepted. New Maternala nd Child receive equal emphasis. Health Specialist position incor- porated in Third Project as per 15. Resident specialist useful wishes of Cofianciers, who fund position. the services. Accepted. Incorporated in design of subsequent programs and projects. 16. Crisis response to population issues does not suffice - longer timeframe and persistent efforts needed. ANNEX B Page 55 of 58 - 78 - Indonesia rrigation Projects Loans: 1100, 1434, 1435, 1579,1691 PPAR No. 6864, June 30,1987 AS5AG Division (then AEPA4) had extensive discussions with OED staff in November-December 1986 on several issues and recommendations contained in the draft Project Performance Audit Report for the above- mentioned irrigation projects, and the results of these discussions were substantially reflected in the final PPAR dated June 30, 1987. Subsequently, the PPAR was distributed to all the relevant GOI agencies, including the Directorate General of Water Resources in the Ministry of Public Works, BAPPENAS, Ministries of Finance, Agriculture, and Home Affairs, and the provincial level agencies. The substance of the findings and recommendations of the PPAR were extensively discussed with GOI officials, both in the context of supervision of ongoing projects with O&M and cost recovery compDonents (e.g., East Java Irrigation, Central and West Java Irrigation, and Second Provincial Irrigation) and in the course of preparation/appraisal of the Irrigation Subsector Project in 1987. GOI was in broad agreement with the major recommendations of the PPAR, i.e., the need to improve O&M services, strengthen Water Users' Associations, promote efficient water use, and introduce direct cost recovery (for O&M costs) from ultimate beneficiaries. An evidence of GOI's agreement, and subsequent commitment to implement the recommendations of the PPAR, was in the Statement of Policies issued by GOI, as a part of the Irrigation Subsector Project which was approved by the Board in November 1987. The findings of the report were also disseminated to FAO/CP staff, who have been preparing the PCRs for a number of irrigation projects. The FAO/CP staff have been paying particular attention to some of the findings/issues highlighted in the PPAR, to evaluate if these issues are being resolved in the course of implementation of the subsequent ongoing irrigation projects. The report was distributed to all the operational staff within the sector division at headquarters and the Resident Mission in Jakarta, and its recommendations were discussed in Division meetings both here and at RSI. The staff are therefore well cognizant of the recommendations of the PPAR, which have been reflected in the preparation of the recent Irrigation Subsector Project, and also in the monitoring and review of the implementation of the ongoing irrigation projects. The following table details the specific measures being taken under these projects to implement those recommendations. ANNEX B Page 56 of 58 - 79 - Philippines, Second Manfla Water Supply (Loan 1615) PCR No. 7153, March 4, 1988 The Project Completion Report was read by ORD but the project was not audited. As such there were no additional lessons learned beyond what was already included in the rqport and thus known to divisional staff. MODEL FEEDBACK MATRIX Findia/Recommendation Promote Improve Improv. analysis Efficient 0AM Strengthen Cost of fermers' Water Use Schemes WAs Recovery ability to pay 1. Findings not accepted 2. Findings still under Review 8. Finding accepted x X - has been ACCEPTED X X X earlier - no longer operationally relevant 4. Actions Initiated a. Formal Research Study/ Subsector Loan X X X X X b. Informal Review of Ongoing Projects X X c. Production of Policy or Technicul Paper d. Modification of Guidelines/Directives X X X e. Staff Reminded to Follow Guidelines K X X X In the matrix cells, where applicable, specify the Bank unit responsible. ANNEX B Page 57 of 58 - 80 - As a general observation, we find that the OED reports may sometimes be useful in cataloging main lessons learned for posterity, but that they rarely bring new knowledge. Lessons are learned during project implementation and are normally fed back into new project preparation long before completion report, and much less the audit report, is written. Whenever an OED report brings out a particularly interesting topic it is discussed during one of our regular division meetings. Indonesia - First Population Project (Credit 300) OED Report No. 5675 The main findings of the OED report in terms of lessons learned were thats (a) a longer time frame beyond the project period is needed for institution building, especially when a program is relatively new; (b) matching management capacity and project diversity is needed and that over-extension of management capacity resulted in implementation problems; and (c) that priorities and the interests of cofinancing agencies as well as their relevant administrative rules and regulations should be clearly underr,ood at an early stage. The design of the Second Population project incorporated some of these lessons. On the first item, it continued the institution building efforts to strengthen the National Family Planning Coordinating Board (NFPCB) and extend its family planning program outside Java and Bali. However, this project also recognized that large scale programmatic improvements, such as institutionalization of population education, extended beyond a five year project period and that the institutional strengthening would need to be continued. Consequently, institutional development has been an element in all subsequent operations. On the second lesson outlines above, the Second Population Project attempted to respond to the criticism of poor management, particularly in delays in the implementation of the hardware component of the project, by specifically assigning responsibility for the implementation of the civil works component to the NFPCB Secretariat. This was successful and helped expedite the completion of the component. However, as the managerial responsibility for the software component was not as clearly specified, delays were encountered in its implementation. On the third lesson, the audit recommendations do not appear to have been given serious consideration. Following completion of the project, concern was expressed by donor agencies that the Bank had expanded into areas such as research and evaluation in Indonesia, where other agencies had already spent considerable resources and were better able to monitor the results given their representation in the field. ANNEX B Page 58 of 58 - 81 - We are unable to respond to the query regarding the dissemination of the PPARs and PCRs in staff meetings, retreats or seminars, as most of the staff working on Indonesia changed following the reorganization and we are unaware of these activities in the past. ANNEX C Page 1 of 12 - 83 - ANNUAL REPORT ON OPERATIONS EVALUATION (FY88) MONITORING AND EVALUATION IN PROJECTS 1. This Annex reviews the present status of M&E activities in major sectors. It is based on contributions from sector departments in PPR. A. Structural/Sector Adjustment 2. By their nature adjustment operations contain in their design a set of agreed policy changes, institutional reforms and other actions, which are translated into conditions that provide benchmarks for monitoring their implementation, activate tranche releases and trigger follow-up operations. The process is also supported by country economic and sector work which, among other objectives, seeks to evaluate indicators of macroeconomic change and, to the extent possible, relate these to previous actions as well as to proposals for further policy change. 3. In February the Board discussed an Interim Report on Adjustment Lending (R88-15) prepared by PPR, which was the third such report on adjustment lending. That report indicated the more recent tendency to front-load conditionality in adjustment operations, so that more actions are taken before Board presentation of such operations. Front-loading had been a key recommendation made by the OED Report, "Structural ADjustment Lending - A First Review of Experience" (Report No. 6409, September 24, 1986) with the purpose of expediting implementation and simplifying monitoring. Examples where front-loading has been particularly evident include Indonesia Trade Adjustment Loan, Tunisia Industrial and Trade Policy Adjustment Loan and Turkey Second Financial Sector Adjustment Loan. Tranching has also been used much more effectively than in the earlier days of adjustment operations in order to enhance implementation, with more stress now being placed on specific actions which can be easily monitored. 4. In response to Board concerns, particularly over the evaluation of the results of adjustment, a much fuller report on adjustment lending is under preparation in PPR, to be discussed at a Board Seminar early in FY89. This report, though stressing the difficulties of assessing impact of policy reform in the presence of exogenous change, attempts nonetheless to relate the macroeconomic ANNEX C Page 2 of 12 - 84 - outcome to the reforms undertaken. Much further work is necessary on this aspect of adjustment, however, before definitive conclusions can be expected. 5. To enhance the evaluation of adjustment operations a concerted effort has been initiated to intensify the production of Completion Reports on adjustment as well as on other operations so that OED can subject them to auditing without delay. A task force has been set up for this purpose and this has made proposals that would speed up production of PCRs and reduce the accumulated backlog. 6. Attending to the social cost of adjustment has been another area of improvement that will better serve monitoring and evaluation of adjustment operations. This is an aspect of these operations which receives special attention in the PPR report mentioned above. Addressing social concerns during adjustment has been handicapped by the poverty of information about income distribation, services reaching disadvantaged groups, poverty dimensicins, health and demographic measurements, etc. Research is in progress and several reports have already been produced in the Living Standards Management Study series. These have focused on measuring a variety of social dimensions in developing count.ies for use in the design of adjustment operations. This approach i&as been particularly sharpened in respect to Africa under a joint UNDP-World Bank-African Development Bank project, known as the Social Dimensions of Adjustment Stud- (SDA), which aims at strengthening the measurement of social dimensions of structural adjustment and enhancing the capaciti of participating governments to carry out policy studies and design socially-improved adjustment programs. By the end of FY88 requests had been received from 24 African governments to participate in the SDA, and implementation is planned at the rate of 8 a year. In time this effort should yield results that would greatly improve the design of such operations, produce benchmarks against which progress can be monitored and in retrospect properly evaluated. 7. In parallel with this progress the processing procedures for adjustment operations have been tightened at least in three specific respects. Firstly, new guidelines were issued to staff in November 1987 which clarify the various stages of processing these operations within the Bank, emphasizing areas to be covered and specifying attachments to be shown. Secondly, a formal system for considering tranche releases has been established in the Office of the Senior Vice President, Operations. This carefully examines conditionality observance before a detailed account is given to the Board subsequent to tranche releaso, indicating the grounds on which the release was made. A third area of improvement is the injunction to the Regions, ANNEX C Page 3 of 12 - 85 - which is contained in the new Guidelines, that the Initiating Memorandum on adjustment operations should indicate all conditions attached to previous adjustment operations, whether SALs or SECALs, showing a record of their observance or lack thereof. This information would further help improve design and give a retrospective account of previous conditionality. B. Energy 8. All of the 20 energy projects approved in FY88 included some monitoring and evaluation components, usually in the form of financial, project progress and completion reports. 9. In addition, seven projects (Algeria Transmission, Jamaica Rehabilitation, Malaysia Thermal, Niger Energy, Somalia Rehabilitation, Thailand Transmission and Zimbabwe Rehabilitation) included components for the upgrading of management information systems to improve operations and also to monitor project effectiveness. 10. The Jamaica project was particularly well designed in this respect, including specific target dates to monitor project implementation and performance indicitors to follow the improvements in utility operations, as a basis for measuring (and rewarding) the effectiveness of management. Mindful of the usefulness of performance indicators as a tool for utility management, IENED has prepared a PPR Technical Paper "Proposals for Performance Monitoring of Electric Utilities*. 11. Although the FY88 energy projects in general had satisfactory monitoring and evaluation components, there is scope for finding ways to increase the motivation of governments, borrowers, consultants and project personnel to implement projects on a timely basis within available resources, because the critical financial condition of most utilities in the developing world demands increased efficiency. C. Education 12. All of the 19 education sector projects approved in FY88 included monitoring and evaluation components, making this the third consecutive year of complete coverage. Sixty-three percent of the projects called for explicit evaluation research and studies to provide baseline data and appropriate comparators for assessing the implementation and impact of the project. Four projects included ANNEX C Page 4 of 12 - 86 - specific monitoring indicators (costs and quantitative indicators of participation). 13. Sixty-eight percent of the FY88 projects included investments designed to strengthen the monitoring and evaluation capacity of Borrower agencies through training and fellowships, technical assistance and purchase of computer hardware and software. Of these, four projects are notable for including components designed to improve the quality and efficiency of national examination systems (Jordan, Bhutan, Nigeria, Ethiopia). Fifty-eight of the projects included support for the development and installation of management information systems, typically as part of a program to improve sector management and administration. 14. The Education and Employment Division in PPR provided support for monitoring and evaluation through advice to project staff and publication of policy-oriented evaluation studies. Among these are studies that compare cost-effectiveness of alternative education inputs, assess linkages between labor markets and secondary vocational and technical education, examine school organization effects on student achievement, compare the effectiveness and efficiency of public and private schools, investigate rates of return to manpower training programs, and analyze teachcro salaries in comparison with non-teacher salaries. The DivV1.u participated in several international forums on monitorinL &ad evaluation, including two dealing specifically with improving education indicators for evaluation. 15. In preparation for staff seminars in FY89, a draft of "Building Educational Evaluation Capacity in Developing Countries: Lessons from the U.S. Experience, Strategies for the World Bank" has been completed and is under review. D. Population, Health and Nutrition 16. PHN FY88 projects recognize the importance of monitoring and evaluation both for strengthening the health sector in individual countries and for ensuring that potential project benefits are realized. Much of the built-in monitoring and evaluation of PHN projects utilizes and shares the systems and activities incorporated into regular project components, such as institutional development. The component-related activities are described first, and then the activities for monitoring and evaluating the actual projects are reviewed. ANNEX C Page 5 of 12 - 87 - 17. Monitoring and Evaluation as Project Components. All eight of the FY88 PHN projects include components for institutional development and strengthening of management, and the frequent inclusion of monitoring and evaluation activities as a significant part of these components reflects their importance to the planning, implementation, and management of effective programs and services in the PHN sectors generally. One project in particular (Sri Lanka) is notable for the attention given to monitoring and evaluation in its health management component, which clearly indicates the relationship between project inputs, managerial actions, expected outputs, indicators of these outputs, and their evaluation. 18. As part of institutional capacity building, the development of management and health information systems (MIS and HIS, respectively) is a feature of several projects (e.g., Guinea, India, Sri Lanka, Burundi, Kenya, Brazil). These information systems have various pLrposes: financial and operational planning and management; assessment and monitoring of country health and vital statistics, including child health and nutritional surveillance; and monitoring the performance and effectiveness of health services and interventions. Special studies are also included in some projects, such as baseline and annual household surveys and special studies to evaluate programs (as opposed to project) achievements (India); evaluation of potential investments in the health sector (Guinea); selected aspects of MIS and HIS systems (Uganda); and operational research programs. These components also include the strengthening of institutional and staff capacities in planning, monitoring and evaluation through the provision of technical assistance and training (Uganda, Guinea, Burundi, Brazil, Kenya). 19. Monitoring and evaluation is not limited to institutional development components of projects. For example, the health education component of the Ethiopia project includes the administration of periodic questionnaires to health staff to whom activities are directed, plus annual, population-based KAP (knowledge, attitudes, practices) surveys. The purpose of these is to monitor and evaluate project interventions in order to identify needs and opportunities for improvement. 20. Built-in Project Monitoring and Evaluation Activities. The monitoring and evaluation activities aimed at assessing project progress and impact incorporate, but are not limited to, the data and activities described above as project operations. The MIS and HIS systems developed as part of institutional strengthening serve to provide feedback on various aspects of project progress, management and effectiveness. Also, regular surveys such as those already ANNEX C Page 6 of 12 - 88 - mentioned provide an opportunity to asses the project's contribution to national programs or its impact on health status or on health service utilization. 21. All of the FY88 projects include regular progress reports and review mechanisms which compare planned and actual inputs and outputs. The annual preparation and revision of 12-month implementation plans or strategies are a feature of most projects (Brazil, Kenya, Sri Lanka, Ethiopia, Burundi, Guinea). These, sometimes accompanied by systematic annual project evaluations or special surveys, are used both to monitor progress and to serve as a basis for the following year's implementation plan. In addition to the (usually quarterly) progress reports and annual reviews, projects generally have larger, often independent, mid-term and final performance reviews. 22. It is difficult to directly measure and attribute with certainty the impact of any PHN project, although baseline and subsequent surveys, as well as data collected routinely in HIS and HIS systems, serve to document changes over time in health status and in health service needs, costs, utilization and, in some cases, effectiveness. Surveys and information systems are often already included as part of project components because of their value in improving components. However, a number of activities are to be undertaken for the more direct purpose of project evaluation. Examples include: a comprehensive evaluation of the performance of the primary health care system in selected areas of Sri Lanka; epidemiological studies to test the effectiveness of interventions (Brazil); sociological surveys of the reactions of project beneficiaries to p oject activities (Ethiopia); KAP studies aimed at assessing project impact in the IEC/FP (information, education, communication/family planning) component (Sri Lanka, Ethiopia); annual evaluation of cost recovery activities (Guinea); baseline and final surveys to determine changes in specified indicators (India); and independent evaluation of all program subcomponents (Kenya, Sri 1.anka, Ethiopia). In addition to these component-oriented evaluation activities, project management itself is an area of particular attention in evaluation and review efforts. E. Transportation 23. Project monitoring and evaluation in transportatinn projects relies mostly on management information systems developed and used by modal agencies. There is, however, a wide range in the sophistication and quality of the information systems. Revenue-earning entities such as railways and ports for the most part possess a basic information ANNEX C Page 7 of 12 - 89 - system related to their financial and operational performance. This information is routinely collected although it may not be used very effectively for planning, monitoring an evaluation purposes. The existing system in many cases do not meet the requirements of modern financial and operational management; available information is poorly processed and evaluation for decision-making mainly due to lack of qualified staff and appropriate data processing and management technology. In highways, the uain focus of monitoring and evaluation ie on traffic surveys, the size and composition of vehicle fleet, road condition evaluations, and management of maintenance equipment. While significant progress has been made under Bank-financed projects in implementing systematic traffic surveys, more work needs to be done to monitor. the condition of road networks and the utilization of equipment fleets. A recent Bank Policy study entitled Road Deterioration in Developing Countries, recommends the establishment of an adequate road management system or a phased monitorable program for establishing one as a pre-condition for Bank highway lending. In FY89, PPR will prepare operational guidelines for establishing highway/pavement management systems in support of this recommendation. An earlier departmental discussion paper (TRP 5) provided guidance on improving equipment management in developing countries. 24. Transport projects approved during FY88 utilize a variety of instruments to monitor and evaluate project performance. Action plans with a time ochedule of monitorable actions and responsibilities and annual reviews are included in all projects with an agenda of policy and institutional reforms (e.g., Algeria Railways II; Sudan Railways V; Nigeria Highway Sector; Togo Transport Rehabilitation; Indonesia Rural Roads II). A review of the experience with contract plans in railway lending is planned during FY89. Nearly ell highway projects include project performance indicators based on existing pavement, maintenance, or equipment management systems (notable examples are Mexico Highway Maintenance and Rehabilitation; Thailand Highway Sector; Madagascar Highways VII), or include provisions for setting up such management systems (e.g., China Sichuan and Shaaxi Provincial Road projects). Among the port operations the Mexico Ports IV project include3 a set of key operational targets to monitor and evaluate project performance. 25. The availability of microcomputer and database software in developing countries has provided a strong stimulus to monitoring and evaluation activities in transportation agencies in developing countries. Bank projects have played a catalytic role in introducing appropriate management information technology to these agencies and ANNEX C Page 8 of 12 - 90 - financial and technical support for information systems and related computer hardware and software is envisaged in future Bank lending in all transport modes. F. Urban Development 26. All FY88 urban projects include some form of monitoring and evaluation instrument. The majority will rely on a single, centralized, inter-ministerial, non-permanent committee to handle the day-to-day monitoring and evaluation of operations. Many times these committees or units were formed during a previous Bank-supported project and will be strengthened with the new loan (e.g. the Indonesia Jabotabek Urban Development Project). Where such a committee does not exist, one will be created. In the Guatemala Municipal Development project, a permanent government office--the Office of the Vice-Mayor- -rather than a committee, will incorporate the monitoring and evaluation responsibilities into its existing operations. 27. The tasks of the monitoring committees generally include monitoring the implementation of the project action plan and justifying proposed changes, ensuring the timeliness of inputs and the availability of resources, coordinating the physical components and supervising the financial--and sometimes technical--status of sub- agencies, collaborating with external agencies in the provision of technical assistance, and submitting periodic proposals for future projects. 28. Broader project objectives are also monitored in some cases. The Ghana Priority Works Project includes provisions for undertaking studies of the manpower requirements of the works to be executed. This will permit progress in achieving the project's employment- generation goals to be monitored. G. Water and Sanitation 29. Bankwide, 6.3 percent of nearly 1,840 projects are supervised by the Water Supply and Sanitation (WSS) Sector. These are distributed almost evenly worldwide, with 28 to 30 projects in each region. All five projects approved by the Board in FY88 have built-in provisions for evaluating and monitoring progress, including expected institutional performance indices. Monitoring indicators have been made part of periodical reporting requirements and are to be kept updated (usually twice a year) by the borrower for review by project supervision teams. ANNEX C Page 9 of 12 - 91 - 30. Proposed sector-type lending operations and many non-sector projects (Cote d'Ivoire, Argentina and Mexico sector operations, for example) have included technical assistance components to strengthen their reporting systems. The recently approved Istanbul WSS Project contains specific provisions for financial performance reporting to identify "significant variations from planned expenditures and financial performance objectives." Likewise, the more complex Colombia WSS Sector Project requires not only periodic reporting on performance of sector utilities using specific monitoring indicators but also performance indicators of the financing agency BCH/FFDU that swould be reviewed and updated annually." 31. The Infrastructure and Urban Development (INU) Department, as part of the FY87 Annual Sector Review, has reported on the 'Achievement of Sectoral Expectations", in which data from 70 WSS projects begun between 1966 and 1981 are analyzed and compared to actual results at project completion. Specific indicators include expected reductions in unaccounted-for water, increases in the number of connections, water volume (produced and sold), tariff and revenue projections, etc. Systematic discrepancies among the forecasts for key performance indicators and the actual results have occurred, mostly in the direction of unduly optimistic expectations from both the borrowers and the Bank. 32. To help Bank staff identify suitable and "down-to-earth" values for monitoring and evaluating borrowers performance indicators, the FY88 Water and Sanitation Annual Sector Review provides a framework or reference on the probability of achieving different rates of change in specific project parameters for evaluation purposes. These parameters include volume of production, volume of water sold, number of new connections, reduction in the percent of produced water unaccounted for in sales, operation and maintenance costs, real effective tariffs, and real sales revenue. H. Agriculture 33. As pointed out in the FY87 Report (page 42, para. 4)) the Monitoring and Evaluation unit of AGR was dismantled as part of the reorganization and all M and E staff were relocated to operations. Each Region, therefore, has become fully responsible for tracking its own M and E activities and for guiding and supporting its staff on M and E matters. 34. The only function of the former AGRME unit which AGR has continted to maintain is the agriculture database. From this it is possible to determine that H and E was included as a project activity ANNEX C Page 10 of 12 - 92 - in 34 of the 56 AGRD projects in FY88. The Bank and IDA lending specifically for M and E components in projects was US$19 M. However, this is probably an underestimate as M and E is occasionally subsumed under 'studies' in project cost tables. Attached is the relevant information from the databas e. N V4 Attachment a Pe Agriculture and Rural Development Lending: FY88 Projects Involving Monitoring Fnd Evaluation Specified M/E Included Cost of M/E Bank/IDA As a Project Component Lending Region Country Name Activity US$ M (*) US$M Africa Burkina Faso Ag. Research X .0 17.9 Burundi Agric. Muyinga X .0 10.0 Cameroon Cocoa Rehab. X .5 103.0 Chad Livestock X .6 18.6 Cote d'Ivoire Rubber V X .7 11.9 Ghana Cocoa Rehab X .0 40.0 Guinea Agric. Seeds X .0 9.0 Kenya Agric. Research X .0 19.6 Madagascar Forest Mgt. X 1.3 7.0 Malawi Agriculture Credit x .1 5.9 * Mali Office du Niger II X .0 39.8 Senegal Irrigation IV X .2 33.6 Sudan S. Kordofan Agr. X 3.8 19.7 Togo Cotton Sector Development X 1.6 14.2 Uganda Sweat Ag. Rehab X .6 10.0 Total Africa 9.4 360.2 Asia Bangladesh Sm. Sc. FDC & Irrig. II X .3 81.5 China Rural Credit III X .0 170.0 Rural Sector Adj. Loan X .0 300.0 Coastal Development X .0 100.0 India National Dairy II X 1.5 360.0 Indonesia Forestry Inst. & Conservation X .0 34.0 Irrigation Subsector X .0 234.0 Malaysia Felcra II (Agric.) X .0 66.0 Philippines Irrig. Operations Support Proj. X .0 45.0 Total Asia 1.8 1390.5 o 44 0 ON Specified x M/E Included Cost of M/E Bank/IDA As a Project Component Lending <104 Region Country Name Activity US$ M (*) US$M EMENA Morocco Agr. Sector II X .0 225.0 Pakistan Irrigation Rehab. II X 3.5 79.5 Oilseeds Development X .0 51.4 Tunisia Forestry x .0 120.0 Yemen A.R. Agric. Dev. (Northern Reg.) X .1 17.6 Total EMENA 3.6 393.5 LAC Argentina Ag. Credit II-A X 1.2 106.5 * Brazil Irr. Sub-Sctr. I-A X .7 195.0 Agr. Crd. A X .0 300.0 Minas Ger's. Forestry-A X 1.5 48.5 Itaparica-A x .8 132.0 Total LAC 4.2 782.0 TOTAL 19.0 2926.2 (*) Costs for Monitoring and Evaluation are underestimates. M/E costs are often consolidatri In appraisal reports under studies or technical assistance. ANNEX D Page 1 of 10 - 95 - ANNUAL REPORT ON OPERATIONS EAATION FY8) CEO REPORTS ISSUED DURING FY88 A. PROJECT PERFORMANCE AUDIT REPORTS Report Rport Covering Rawo Agriculture NO. Datg to Board 1. Papua New Popondetta Smallholder Oil Palm Development 7001 11/09/87 SecM87-1288 Guinea (a) (Credit 1149-PN) Second Agricultural Credit (Credit 841-PM) Southern Highlands Rural Development (Loan 1888-PN) 2. Tunisia Sidi Soleo Multipurpose (Loan 1481-TUN) 7087 12/11/87 SecM87-1860 S. Colombia Cordoba II Agricultural Development (Loan 1168-CO) 7062 12/21./87 SecM8-2 4. Peru Irrigation Rehabilitation (Loan 1408-PE) 7247 06/20/88 SocM6-589 5. Sudan Third Mechanized Farming (Credit 04-SU) 7282 06/27/ Secm"-it 6. Philippines National Extension (Loan 1626-PH) 72M6 06/18/8 SecM88-721 7. Malaysia Notional Extension (Loan 1408-MA) 7287 06/18/68 SecM8-746 8. Pakistan (2) Punjab Extension and Agricultural Development, 7266 06/28/68 SecM98-784 Sind Agricultural Extension and Adaptive Research (Credits 818/922-PAK) 9. Chile Second Agricultural Credit (Loan 1902-CH) 7802 00/22/88 SecM88-788 10. Brazil (4) Experience in the Development of the Rural Sector - 7881 06/27/68 Seems-881 Second Agro-Industries Credit (Loan 1817-BR) Minas Cerals Rural Development (Loan 1862-BR) First Agricultural Extension (Loan 168-R) Bahia Rural Development-Paraguacu (Loan 15809-R) 11. Yugoslavia Third Agricultural Credit (Loan 1801-1U) 7884 00/80/88 SecM88-824 12. Sri Lanka (2) Second Mahawell Ganga Development (Credit 701-CE) 7886 06/80/88 SecMB8-820 Mahawell Technical Assistance (Credit 979-CE) 18. Egypt Agricultural Development (Credit 680-ECT) 7887 08/80/08 SecM8S-882 14. Egypt (2) Upper Egypt Second Drainage 7886 08/80/68 8e*eB8-8 (Loan 1286/Credit 887-ECT) Nile Delta Second Drainage (Loan 1489/1440-EGT/Credit 719-ECT) 15. Sri Lanka Tree Crop Rehabilitation (Tea) (Credit 616-CE) 7889 08/80/88 SecMB8-85 ANNEX D Page 2 of I Report Report Covering Memo Agriculture (Cont.d) NO. Date to Board 16. Sudan Second Agricultural Rehabilitation (Credit 1889-SU) 7842 08/80/88 SecM88-764 17. Tanzania Export Rehabilitation (Credit 1188-TA) 7847 06/80/88 SecM88-80 18. Thailand Agricultural Credit (Loan 1818-TH) 7854 06/80/68 SecM88-828 Human Resources 1. Portugal First Education (Loan 1569-PO) 6926 09/10/9? SecM87-1008 2. Indonesia Second Agricultural Training (Loan 1692-IND) 7051 12/19/87 SecM87-1408 8. Mexico Technical Training (CONALEP I) (Loan 2042-ME) 7186 08/01/88 SecM88-247 4. Somalia Third Education (Credit 788-SO) 7187 08/01/88 SecM88-227 S. Colombia Integrated Nutrition Improvement (Loan 1487-CO) 7180 04/12/88 SecM88-82 6. Ethiopia Fifth Education (Credit 1141-ET) 7209 04/28/88 SecM88-40 7. Burkina Faso Second Education (Credit 956-UR) 7251 05/24/8 SecM88-598 S. Korea Sector Program on Higher Technical Education 7252 05/24/8 Seck"-595 (Loan 1800-K0) 9. Thailand Fifth (Multipurpose) Education (Credit 818-TH) 7284 06/20/68 SeeM88-728 10. Thailand Population (Credit 767-TN) 7886 06/80/88 SecM88-822 Transportation Highways 1. Gamb!e Highway Malatenance (Credit 897-Ol) 6895 07/22/87 SecM87-911 1. Mexico First Highway Sector (Loan 1871-ME) 896 07/20/87 SecM87-910 8. Tunisla Third Highway (Loan 1601-TI) 7182 04/11/8 SecM--408 4. Malaysia Third Highway (!.oan 1870-MA) 7808 06/22/88 SecMB8-805 5. Yemen PDR Third Highway and Roads Flood Reconstruction 7811 08/24/8 SecM8-740 (2) (Credits 1144/1295) 6. Yemen AR Third Highway (Credit '94) 7812 08/24/08 SecM88-785 7. Philippines Third Highway (Loan 1858-PH) 7816 06/80/8 SecM88-87b ANNEX D Pase 8 of 10 - 97 - Report Report Covering Memo Transportation (Cont.d) NO. Date to Board Ports 1. Egypt Alexandria Port (Loan 1289-EOT) 7301 06/22/e8 SecM98-786 Railways 1. India Railway Modernization and Maintenance (Credit 844-IN) 7020 11/8/87 SecM87-1827 Urban 1. Mexico Lazar* Cardenas Conurbation Development (Loan 1654-ME) 7008 11/09/87 SecM7-1287 2. Thailand Bangkok Traffic Management (Loan 1688-TH) 7068 12/29/87 SecMt-19 8. Philippines Manila Urban Development and Second Urban Development 7092 01/19/88 SecM88-111 (2) (Loans 1272/1282-PH and Loan 1647-PH) 4. Jordan Cities and Villages Development Bank 7226 04/27/U8 SecM8-618 (Loan 1826-JO) Industry 1. Turkey Sumerbank Cotton Textile Rationalization 7118 02/04/88 SecM68-148 (Loan 1847-TU) 2. Togo (2) Ciaao Regional Clinker 7828 06/28/88 SecMs-821 Cote d'Ivoire (Loans 1296-WAF/1296-TO/1297-IVC/1298-W) Ghana Claso Restructuring (Credits 1826/1827-TO) 8. Tunisep SGOITEX Textile Rehabilitatlon (Loan 2012-UN) 7868 06/80/88 SecM88-862 Enery and Public Utilities 1. Bangladesh Bakhrabad Gas Development (Credit 1091-BD) 6976 10/14/87 SecM87-1168 2. Thailand Liquifled Petroleum Gas (Loan 2184-TH) 7201 04/18/8U SecMB8-426 Power 1. Yemen AR (2) Power Distribution (Credit 887-YAR) 7064 12/21/87 SecM88-20 Regiont Electrification (Credit 1102-YAR) ANNEX D Page 4 of 10 Report Report Covering Memo Eneray and Public Utilities (Cont.d) NO. Date to Board Water Supply A Waste Diosal 1. Lesotho Sevan Towns Water Supply (Credit 87-LSO) 7021 11/28/87 SecM87-1828 2. Brazil Southern States Water Supply and Sewerage 7088 01/14/68 SecM98-66 (Loan 1828-3R) 8. India Maharashtra Water Supply and Sewerage (Credit 899-IN) 7202 04/18/8 SecM88-486 DFCs 1. Panama (2) Corporacion Financiera Nacional (COFINA) 7062 12/22/87 SecM88-1 Development Banking (Loans 1641/195-PAN) SALe 1. Yugoalavia Structural Adjustment Loan (Loan 2826-YU) 6954 09/29/87 SccM97-1085 2. Turkey (2) Fourth and Fifth Structural Adjustment Loans 7205 04/81/88 SecM88-448 (Loans 2821/2441-TU) 8. Togo Structural Adjustment Credit (Credit 1865-TO) 7861 06/80/88 SecM68-821 Technical Assistance 1. Indonesia Fifth Technical Assistance (Credit 898-IND) 7156 08/10/88 SecM8B-841 2. Togo (2) First and Second Technical Assistance 7861 06/80/88 SecM88-821 (Credits 980/1270-TO) ANNE) 0 Page 5 of 10 - 99 - 0ED REPORTS ISSUED DURING FY86 8. PASS-THROUGH PROJECT COMPLETION REPORTS Report Report Covering Veo Aariculture No. Date to Board 1. Cameroon Western Highlands Rural Development (Credit 784-CM) 6962 10/12/87 SecM87-1128 2. Greece Evros Development (Loan 1457-0R) 6966 10/14/67 SecM87-1182 8 Romania Third Livestock (Poultry) (Loan 17644-PO) 6987 10/14/67 SecM87-1126 4. Panama Second Livestock Development (Loan 1897-PAN) 6974 10/21/87 SecM87-1158 5. Malaysia FELDA VI Land Settlement (Loan 1690-MA) 6992 10/80/87 SecM8T-1224 8. Barbados Technical Assistance (Agricultural Component) 7022 11/24/87 So*M87-1826 (Loan 2116-BAR) 7. Malaysia Rompin-Enaau Agricuiturai beveiopment (Loan iii-iA) s0o6 i0j/7/7 SWGM7-1;i 8. Korea 0gso Area Development (Stage I) (Loan 1608-KO) 7041 12/06/67 SecM87-1890 9. Rwanda Bugesera East, Glaka/Migongo Mixed Farming 7042 12/11/87 Secu87-j892 and Rural Development (Credit 688-RW) 10. Tanzania Pyrethram (Credit 1007-TA) 7044 12/16/87 SecM87-1898 11. Malaysia Krian-Sungel Manik Integrated Agricultural 7071 12/28/87 SecM88-18 Development (Loan 1882-MA) 12. Indonesia First Smallholder Rubber Development (Credit 984-IND) 7072 12/29/87 SecM88-17 18. Bangladesh Second Foodgrain Storage (Credit 787-8) 7119 02/17/88 SecM88-164 14. Central Livestock Development (Credit 894-CAR) 7147 08/02/88 SecM88-295 African Rep. 15. Senegal Debi-Lampear Irrigation (Credit 775-SE) 7149 08/02/88 SecM88-829 16. Sudan Southern Region Agriculture (Credit 904-SU) 7161 08/11/86 SecM88-294 17. Burea Grain Storage (Credit 1092-BA) 7217 04/28/86 SecMB8-484 18. Burma Forestry II Pegu Yoma (Credit 949-BA) 7218 04/28/88 SecM88-485 19. Zambia Coffee Production (Credit 868-ZA) 7219 04/28/88 SecM8-462 20. Paraguay Agricultural Credit (Loan 2804-PA) 7288 05/12/88 SecM818-88 ANNEX D Page 6 of 10 -100 - Report Report Covering Memo Agriculture Cont.d) No. Date to Board 21. Colombia Third Agricultural Credit (Loon 1787-CO) 7284 05/12/8 Sec:1168-519 22. Ouinea Rice Development (Credit 962-GUI) 7280 08/16/88 S*c:M88-56W 28. Leo People's Agricultural Rehabilitation and Development 7250 05/16/68 Secudse-718 Democratic (Credit 760-LA) Respubl Ic 24. Mexico Small-Scale Agelcultural Infrastructure (Loan 1648-ME) 7258 05/27/68 SOcK89-835 25. Greece Integrated Forestry (Loan 1718-GRE) 7801 06/28/188 Sec:188=750 26. Rwanda Cinchona (Credit 656-R4 7308 06/16/88 SeeMN8-775 27. Pakistan On-Fare Water Management (Credit 118-PAm') 780 06/24/88 SocUM-773 28. Yugoslavia Bossae Kraj ins Agriculture and Agro-Industries 7810 06/22/88 SoeMBS-782 (Loan 1621-YU) 29. Indonesia Twelfth Irrigation (Loan 1646-IND) 7818 06/24/68 SscM88-746 So. Niger Bid& Agricultural Development (Loan 1687-UNI) 7815 06/24/88 SecM88-774 a1. Yugolavia Metohija Fi ret Multipurpose (Loan 18W0YU) 7817 06/24/88 SocM88-757 82. Tanzania Rural Development Dank (Credit 987-TA) 7822 06/26/88 Sec:188-858 88. Ecuador Agricultural Credit (Loan 1469-EC) 7828 06/28/88 SscM88-940 84. Mail Sud Agricultural Development (Credit 689-MUI) 7824 06/28/88 Sec988-861 85. Niger Dosso Rural Development (Credit 967-NIR) 7888 06/24/88 Sscii88-874 Human Resources 1. El Salvador Industrial and Agricultural Training (Loan 1571-ES) 7088 12/11/87 SecM97-181 2. Egypt Second Population (Credit 850-EGT) 7160 08/14/88 SecM89-831 a. Nepal First Education (Credit 772-WI) 7172 08/80/88 SeeM88-840 4. Bolivia First Education and Vocational Training (Loon 1404-80) 7289 05/16/88 SOC118S-560 5. Rwands First Education (Credit 667-RW) 7240 005/10/88 S*c;M98-547 6. Senegal Third Education (Credit 908-SE) 7242 05/10/88 SecM88-648 7. Coto dlIvoilre Third Education (Loan 1777-IYC) 7266 05/27/88 SwcM88-685 8. Uruguay Vocational Training and Technological Development 7272 06/18/88 SeeMSe-720 (Loon 169441JR) Page 7 of 10 - 101 - Report Report Covering Memo Transportation No. Date to Board Hichways 1. Paraguay Sixth Highway (Loan 1786-PA) 6909 08/07/87 SecM87-924 2. Botswana Fifth Road (Loan 1941-0T) 6987 09/16/87 SecM87-1066 8. Tanzania Fifth Highway (Credit 876-TA) 6988 09/16/87 SecM87-1072 4. Ohana Third Highway (Emergency Maintenance) 6946 09/18/87 SecM87-1071 (Credit 1029-OH) 5. Morocco Third Highway (Loan 1980-MOR) 7015 11/19/87 SecM87-1000 6. Algeria Third Highway (Loan 1688-ALO) 7048 12/14/87 SecMB8-4 7. Togo Feeder Roads (Credit 810-TO) 7048 12/09/87 SecM88-8 8. Honduras Eighth Highway (Loan 1901-H0) 7107 02/02/88 SecM88-127 9. Belize Road Maintenance and Rehabilitation 7106 02/05/88 SecM88-126 (Loan 2278-BEL) 10. Congo Third Highway (Loan 2227-COB) 7188 04/08/88 SecM88-874 11. Guatemala Highway Maintenance (Loan 1848-QUA) 7264 05/20/88 SecM88-821 Ports 1. Mexico Port Development Preparation 7248 05/18/88 SecM88-568 (Loan 1964-ME) R!lwan 1. Senegal SEFICs Rail Transport (Loan 2025-SE) 7258 06/20/88 SecMB8-822 Urban 1. India Calcutta Urban Development (Credit 768-ID) 6970 09/80/87 SecM87-1181 2. Philippines Urban Engineering (Loan 2087-PH) 7009 11/16/67 SecM87-1282 8. Burkins Paso Urban Development (Credit 766-MUR) 7082 12/01/87 SecM87-188 4. Brazil Second Urban Transport (Loan 1889-8R) 7069 12/24/87 SecM88-54 S. Colombia Urban Development (Loan 1568-CO) 7210 04/20/88 SecM88-500 ANNEX D Page 8 of 10 - 102 - Report Report Covering Memo Industry No. Date to Board 1. Jordan Arab Potash (Loan 1617-JO) 7078 01/06/88 SecM88-42 2. Brazil CSN Steel Expansion - Stage III (Loan 1161-8R) 7282 06/05/88 SecM88-567 8. Burundi Nickel Exploration/Engineering (Credit 1154-8M) 7278 08/06/88 SecM88-684 4. Romania Roman Seamless Pipes (Loan 1651-R0) 7851 06/80/98 SecM88-889 5. Portugal Fertilizer Modernization (Loan 1649-PO) 7852 08/80/88 SecM88-876 6. Egypt Pulp and Paper (Loan 1849-EQT) 7859 06/80/88 SecM88-857 7. Indonesia Central Java Pulp and Paper Engineering 7860 06/80/88 SecM88-819 (Loan 2199-IND) Enery and Public Utilities Energy 1. Philippines Coal Exploration (Loan 2181-PH) 6960 09/29/87 SecM87-1107 2. Morocco Oil Shale Engineering (Loan 2114-MOR) 7141 02/24/88 SecM88-228 8. Pakistan Coal Engineering (Credit 1856-PAK) 7818 06/17/88 SecMA8-756 4. Romania Videle/Balaria Eqhanced Oil Recovery (Loan 2148-RO) 7850 06/80/88 SecM88-872 Power 1. Costa Rice Sixth Power (Loan 1718-CR) 7070 12/28/87 SecM88-88 2. Sri Lanka Eighth (Diesel) Power (Loan 2187-CE) 7181 04/04/88 SecM88-878 8. Brazil Companhis Eatadual de Energle Eletrica (CEEE) 7211 04/20/88 SecM88-501 Power Distribution (Loan 1824-BR) 4. Brazil South-Southeast Powor Distribution (Loan 1688-BR) 7212 04/20/88 SecM88-488 5. Guinea First Power (Credit 1085-0UI) 7286 05/06/88 S H88-648 6. Morocco Village Electrification (Loan 1695-MOR) 7826 06/28/88 SecM88-884 7. Tunisia Third Power (Loan 2008-TUN) 7828 06/28/88 SecM88-886 Telecommunications 1. Burma Second Telecommunications (Credit 956-BA) 7197 04/11/88 SecM88-419 2. Colombia Fifth Telecommunications (Loan 1825-CD) 7827 06/28/88 SecMB8-869 ANNEX D Pag. 9 of 10 - 108 - Report Report Covering Memo Energy and Public Utilities (Cont.d) No. Date to Board Water Supply and Waste Disposal 1. Guinea Conakry Water Supply and Sanitation 6922 08/81/87 SecMB?-992 (Credit 870-UI) 2. India Punjab Water Supply and Sewerage 6988 09/16/87 SecMB7-1042 (Credit 848-IN) 8. Bolivia Urban and Rural Communities Water Supply 6988 09/16/87 SecM87-1041 and Sewerage (Loan 1824-80) 4. Portugal Lisbon Region Water Supply (Loan 1641-PO) 7008 11/18/87 SecM87-1281 5. Brazil Greater Sao Paulo Sewage and Treatment 7016 11/18/87 SecM7-1826 (Losn 1526-BR) 6. Philippines Second Manila Water Supply (Loan 1615-PH) 7158 08/04/88 SecM8-296 7. Colombia (2) Second Bogota Watee Supply and Third Bogota 7281 05/02/88 SecMB8-649 Water Supply, Sewerage and Drainage (Loans 741/1697-CO) 8. Sri Lanka Second Water Supply and Sewerage 7286 05/06/88 SecM88-694 (Credit 1041-CE) 9. Cyprus Limassol Sewerage A Drainage (Loan 2411-CY) 7819 06/17/88 SecM68-754 DFCs 1. Burundi Banque National de Developpement (BNDE) 6956 09/29/87 SecM87-1100 (Credit 781-BU) 2. Burkina Faso Artisan, Small- and Medium-Scale Enterprise 6977 10/15/87 SecM87-1152 (Credit 759-BUR) 8. Honduras (2) First and Second Industrial Estates Credite 7029 12/01/87 SecM17-1869 (Loans 1869/207540) 4. Barbados Industrial Development and Export Promotion 7140 02/25/68 SecM8B-248 (Loan 1818-BAR) S. Madagascar BankinIny Indostrie (BNI) (Credit 977-MAG) 7179 04/04/88 SecM9-881 6. Kenya (2) Second and Third Industrial Development Bank 7246 05/18/88 SecM88-84 (Loans 1148/1488-KE) 7. Egypt (2) Third and Fourth Industrial Development Bank 7882 06/27/88 ScU88-856 (Loans 18G8/1804-ECT) ANNEX D Page 10 of 10 - 104 - Report Report Covering Memo Technical Assistance No. Date to Board 1. Guyana Technical Ausistance for Power Development 7046 12/09/87 SecM87-1889 (Loan 1906-OUA) 2. Peru SIDERPERU Technical Assistance (Loan S-19) 7079 01/05/88 SocM88-41 8. Somalia Technical Assistance (Credit 821-SO) 7274 06/06/8 SecM88-719 OED Studies 1. World Bank Experience with Rural Development 1905-1988 6888 10/16/87 R87-240 2. A Review of Bank Assisted Free-Standing Technical Assistance 7168 08/10/68 SecMB8-841 Credits in Indonesia 8. Evaluation of Structural Adjustment Lending in Turkey - 7205 04/18/88 SecM88-448 Program Performance Audit Report of the Fourth and Fifth Structural Adjustment Loans (Loans 2821-TU and 2441-TU) and Overview of SAL* I-V 4. Educational Development in Thailand: The Role of World Bank Leaijng 7806 06/24/68 SecMS8-804 5. Renewable Resource Management in Development - Lessons from 7845 06/24/88 SecM8-808 World Bank Experience in Agriculture and Forestry 6. Agricultural Marketing: World Bank's Experience 7868 06/28/88 SecM88-880 7. Conditionality in World Bank Lending: Its Relation to 7857 06/29/88 SecM88-879 Agricultural Pricing Policies 8. Cost Discrepancy Study La 04/16/88 Annual Report 1. Annual Report on Operations Evaluation (FY87) 6978 10/20/97 R87-289 2. Annual Review of Project Performance Results 198W 6976 10/19/87 R87-288 8. Directory of DED Reporta -- 10/81/87 SecM87-1260 La Internal study prepared in support of Annual Review of Project Performance Results 1987, and distributed to OED and operational staff. ANNEX E Page 1 of 2 - 105 - ANNUAL REPORT ON OPERATIONS EVALUATION (FY88) OPERATIONS EVALUATION REPORTS AND PAPERS REVIEWED BY JAC (07129187) Pakistan - Aga Khan Rural Support Programt Interim Evaluation (SecM87-14) (09/16/87) IFC Operations Evaluation Work Program for FY88 (JAC87-13) (10/21/87) Impact Evaluation of Cotton Development Programs in Burkina Paso, Cote d'Ivoire and Togo (SecM87-833) (10/21/87) Report of the JAC Subcommittee on Project Performance Audit Reports (JAC87-20) (10/28/87) Annual Report on Operations Evaluation (R87-239) (10/28/87) Annual Review of Project Performance Results 1986 (R87-238) (10/28/87) Approach Paper - The World Bank in Senegal (JAC87-21) (10/28/87) Approach Paper - Review of Bank/Tanzania Relationships (JAC87- 22) (10/28/87) World Bank Experience with Rural Development 1965-1986 (R87- 240) (01/20/88) An Evaluation of IPC's Experience in the Agricultural Processing and Storage Subsector (JAC87-25) (01120/88) Power Section in Annual Review of Project Performance Results 1986 (R87-238) (01/20/88) Approach Paper - An Evaluation of Bank lending to the Colombian Power Sector Between 1970 and 1987 (JAC88-1) (04/27/88) OED Status of Current Work and Proposed Workprogram and Staff Budget for FY89 (JAC88-7) (05/25/88) Use of Bilateral Trust Funds for OED Work (JAC88-16) (05/28/88) Approach Paper on Bank Support on Industrialization in Newly Industrializing Countries (JAC88-11) ANNEX E Page 2 of 2 - 106 - (05128/88 Evaluation of Structural Adjustment Lending in Turkey Program Performance Audit Report of the Fourth and Fifth Structural Adjustment Loans and Overviews of SALs I-V (SecM88-443) (07/07/88) IFC Operations Evaluation Workprogram for FY89 (JAC88-17) (07107/88) Approach Paper for Study of Procurement in Policy-Based Lending (JAC88-15) (07/27/88) A Review of Bank Assisted Free-Standing Technical Assistance Credits in Indonesia (SecM88-341) (07/27/88) Approach Paper - Bank Support for Small and Medium Industry in Selected Countries (JAC88-21) (07/27/88) Approach Paper - An Evaluation of Bank Free-Standing Technical Assistance in Support of Public Sector Management in Sub- Saharan Africa Between 1976 and 1987 (JAC88/22) ANNEX F Page 1 of 2 - 107 - ANNUAL REPORT ON OPERATIONS EVALUATION (FY88) ONGOING OED STUDIES Type of Study/Sector Status of Study Country Reviews Senegal * Main work completed Tanzania * Main work completed Policy-Based Lending/Technical Asa.istance SAL/SECAL Monitoring Initiated SAL/SECAL Procurement * Initiated Technical Assistance in Sub-Saharan Africa * Field Tork completed for West Africa Agriculture Irrigation Impact Evaluations Mexico Main work completed Morocco Main work completed Philippines Main work under way Thailand Main work under way Credit Impact Evaluations ** Chile Main work under way India Main work under way Mexico Initiated Effective Rural Development in North-East Brazil ** Initiated * Approach paper issued ** Approach paper being prepared ANNEX F Page 2 of 2 - 108 - Plantation Crop Review Initiated Transport Civil Works Contracting/Procurement * Main work under way Rural Roads Maintenance ** Indonesia Initiated Philippines Main work under way Human Resources Education Impact Evaluations * a/ Indonesia Main work completed Colombia Main work under way Malawi Main work under way Sustainability of Education Benefits * a/ Initiated Higher Agricultural Education ** Initiated African Human Resources Development ** Initiated Indonesia Education Sector Reviews ** Initiated Industry and DFCs Sustainability of DFCs * Main work completed Industrialization of NICs * Initiated Small and Medium Industry * Initiated Utilities and Energy Colombia Power Sector Review * Main work completed Brazil Infrastructure and Environment ** Initiated Operational Issues Project Supervision ** Initiated * Approach paper issued ** Approach paper being prepared al Covered in one approach paper ANNEX G Page 1 of 6 - 109 - ANNUAL REPORT ON OPERATIONS EVALUATION (FY88) OED'S DISSEMINATION PROGRAM 1/ 1. Primary responsibility for feedback within the Bank (ie. ensuring implementation of findings, lessons and recommendations emanating from OED's products into Bank operational strategies, policies, economic and sector work, and project/policy-based operations) rests with Operations and PPR. In fact, the effectiveness of the feedback process ultimately depends on the diligence with which the findings and recommendations of OED's evaluative work is sought after, utilized and reflected in the Bank's policy formulation and operational decisions by the staff of the Operations and PPR Complexes. OED has no control over the feedback process. On the other hand, OED has primary responsibility for reporting to the Board on the effectiveness of the feedback process, and for supporting the delivery of evaluation findings regarding Bank operations both within the institution and to the wider development community (dissemination). 2. A Task Force, with OED participation, has been established in December 1987 to review means to enhance the effectiveness of the dissemination/feedback process in the reorganized Bank. The emphasis in the deliberations of the Task Force has been on the management response to OED's recommendations, identifying in the new Bank the responsibility for follow-up and the monitoring of the feedback process; communicating to OED topical concerns of the Operations Complex; improving existing and devising new instruments for dissemination; and strengthening collaboration between OED and the PPR and Operations Complexes. The report of the Task Force is currently being reviewed by management. The final report will also be discussed with the JAC. 3. To get the right messages to the right audiences in the newly reorganized Bank, and in addition to what the rest of the Bank can and should do, OED has been examining its own role in dissemination, taking a number of initiatives and exploring new ones. These activities and initiatives can be grouped under three main headings: (a) interaction with Bank staff: (b) formulating "dissemination ready" messages; and (c) supporting delivery of the message. 1/ Updated from Operations Evaluation Department, Status of Current Work and Proposed Work Program and Staff Budget for FY89, R88-95 dated April 29, 1988 (Annex 7). ANNEX G Page 2 of 6 - 110 - Interaction 4. In order to iarm.ove the effectiveness of internal processes, OED is 'aking steps toward: - Seeking much more systematically than in the past the views of PPR and Operations Ltaff regarding subjects to be selected for study or focus most relevant to their policy, research training (EDI) or operational work. - Fuller discussion of Approach Papers for proposed studies with relevant staff in both PPR and Operations. - Mid-course debriefings of studies in progress, involving staff across the Bank as well as those more directly concerned. - Participation in follow up seminars within the Bank, sponsored by PPR and the Regions. - Review of the terms of reference of all major research and policy work planned by PPR and the Regions. - Participation of top OED managers in high level policy- oriented meetings held by PPR and the Regions. - Participation of OED staff in retreats organized by parallel uniti in the Bank. - Greater involvement as warranted for OED staff to participate in research and policy work and, conversely, for other Bank staff to participate in the work of OED (this in itself will enhance dissemination). - Contribution in terms of staff and case material to EDf seminars. - Inviting the DGO to participate in the Project Implementation and Supervision Review both at the Regional and Bankwide levels. (One potential spin-off might be further OED/PPR/ Operations Complex workshops, overall or on a regional or sectoral basis to deal with specific issues emerging from these reviews). ANNEX G Page 3 of 6 - 111 - Content and Presentation of OED Output 5. Various initiatives have been taken to improve the content and presentation of OED's output: (a) Concerning content: - Treating in greater depth and documenting operationally more relevant issues, undertaking a comparative analysis of findings and conclusions, elaborating on factors accounting for superior or poor performance and identifying the prcximate causes of success or failure, passing judgement on the prospects for sustainability of benefits, reforms, etc., and highlighting outstanding, recurring, subtle or elusive issues that deserve constant attention and review by Operations staff. - Prioritizing and targeting findings and recommendations to specific audiences, and giving a higher profile to findings of strategic importance through the use of audiolvideo material. (b) Concerning presentation: - Preparing shorter, more readable documents, and using visual aids (e.g., tabular formats, graphs, boxes, etc.) to summarize information and highlight important points. - Editing professionally studies to be read more widely. - In the increasing number of cluster audits, as well as in the Annual Review and the other studies, paying special attention to the preparation of a concise, yet fully informative, Evaluation Summary to facilitate dissemination. - Formulating conclusions into more sharply focussed and actionable recommendations. - Grouping recommendations in the Annual Review and studies in declining order of importance and by audience, to facilitate Management response, discussion and follow-ap. ANNEX G Page 4 of 6 - 112 - Improving Existing and Devising New Dissemination Instruments 6. In planning the work program for the Annual Review, OED is considering a revision of its content and format to enhance the relevance of the report to Bank operations, rendering it a more actionable document, and facilitating follow-up and monitoring. E.g., an attempt is being made to incorporate findings from other related OED work (special studies, impact evaluations, clusters), to analyze in greater depth and summarize sectoral issues, and to pull together outstanding actionable recommendations. In terms of format, thought is being given to restructuring the sector chapters to reflect the Bank's new organizational structure, put in better perspective issues of accountability, and facilitate discussion on selected themes cutting across sectors. 7. To serve the needs of the Bank staff, aside from the annually updated cumulative Directory of OED Reports, OED continues to store Evaluation Summaries of all OED reports in a computerized system from which they can be extracted quickly and relatively easily. Data retrieval capabilities are being are being enhanced through the development of sectoral databases (agriculture, education and transport are already in place). Further work will be needed to facilitate access to the wide range of lessons learned. 8. OED is considering the production of a separate document, e.g., OED Abstracts or OED Digest, with an appropriate format as is the practice in a number of bilateral agencies, to be issued quarterly, which would present the findings, lessons of experience and recommendations from all OED products, i.e. including the Annual Review, studies and audits, drawing primarily on the Evaluation Summaries. It could also announce upcoming studies and report on progress of ongoing ones. The availability of such a document should enhance accessibility to the findings of OED's products by Bank staff, member countries and agencies concerned. The Digest could also include summaries of the findings of reviews undertaken by other multilateral and bilateral agencies where, for instance, the donors' experience is discussed. 9. The creation of Regional Information Centers, and the attendant dismantling of the Reports Desk through which the distribution of OED documents was effected, the need to store records and documents over a long period of time for retrieval and building up an institutional memory, and the nature of most OED products which cut across regions, will necessitate the establishment of an Information Center within OED. The matter, as well as its space and resource implications, are presently under study. ANNEX G Page 5 of 6 - 113 - Widening the Scope of the International Dimension 10. Closer relations have been instituted with evaluation units in bilateral and multilateral agencies directly, as well as through participation in meetings of the DAC Expert Group on Aid Evaluation, the UN interagency group on aid evaluation and other agencies with a view to share experience in methodology and development issues and to explore prospects for cooperation in evaluation, dissemination and support of evaluation activities with and by developing countries. As an experiment in experience sharing, OED sponsored a seminar on rural development in the Bank's Paris Office recently for the benefit of the DAC's Expert Group on Aid Evaluation and with participation of other donor rural development experts, as well as representatives of the Aga Khan Foundation. Evaluation experience was exchanged and compared on strategy, policy framework, project design and implementation of rural development. Useful lessons were drawn and the conclusions of that seminar were reviewed and adapted by the Expert Group on Aid Evaluation which also prepared a follow-up report. 11. Sharing experience with developing countries is also very important. While the most efficient means of dissemination is through Operational staff, more direct means are also in demand. The principal channel is EDI, to which OED contributes evaluation material and staff support on a limited basis for seminars at headquarters and more selectively in developing countries, focussing on evaluation and on operational issues, such as adjustment lending. A more proactive approach, in which OED would suggest relevant topics, interested audiences and bilateral partners, is being considered. Actual OED inputs in such endeavors will have to be limited given resource constraints. OED has also experimented recently with the most direct approach in which OED staff, in the course of evaluation missions, conduct workshops on evaluation experience in sectors they are familiar with under the auspices of a central evaluation cell or line ministry. Publication 12. The Annual Review has been published since 1977 but OED is increasingly publishing a greater number of studies. An OED Publications Committee was established early in 1987 to propose criteria for publication of OED material and advise the DGO on the publication of selected studies each year, subject to agreement of the JAC and/or the Board. Seven OED reports were published during FY88. In addition to two Annual Reviews they included the following studiess ANNEX G Page 6 of 6 - 114 - * Sri Lanka and the World Bank A Review of a Relationship * The Aga Khan Rural Support Program in Pakistan An Interim Evaluation * Rural Development World Bank Experience, 1965-1986 * Cotton Development Programs in Burkina Faso, Cote d'Ivoire and Togo * The Jengka Triangle Project in Malaysia Impact Evaluation Report It has been OED's practice to translate, usually into French and Spanish, audit reports. This practice has now been extended to special studies and the Annual Review. As a result, OED's translation resource requirements have doubled. Suggestions as to whether or not translation into other languages (e.g., Portuguese, Chinese) might be warranted, should be pursued bearing the costs in mind. 13. Publication, however, calls for high quality products, a sophisticated review process, and more editorial inputs, which are no longer available from the External Affairs staff. Hence, to manage these processes more efficiently, OED has made arrangements with professional foreign language editors and translators to assist in the management and quality control of translation and publication. ANNEX H Page 1 of 5 - 115 - ANNUAL REPORT ON OPERATIONS EVALUATION (FY88) IFC INVESTMENT ASSESSMENT REPORTS AND PROJECT COMPLETION REPORTS FILED IN THE OFFICE OF THE DIRECTOR-GENERAL, OPERATIONS EVALUATION Summary of Reports Issued Fiscal Year Number Issued FY79 22 FY80 13 FY81 18 FY82 9 FY83 17 FY84 9 FY85 10 FY86 11 FY87 16 FY88 27 ANNEX H Page 2 of 5 - 116 - IFC INVESTMENT ASSESSMENT REPORTS AND PROJECT COMPLETION REPORTS FILED IN THE OFFICE OF THE DIRECTOR-GENERAL, OPERATIONS EVALUATION Country Project PCR Date FY79 Dominican Rep. Cementos Nacionales, S.A. 11/00/78 Argentina Dalmine Siderca S.A.I. y C. 12/01/78 Jordan Jordan Ceramic Industries Company Limited 12/05/78 Tunisia Industries Chimiques du Fluor S.A. 12/06/78 Guatemali Exploraciones y Explotaciones Mineras Izabal 12/10/78 Korea Hae Un Dae Development Company, Ltd. (Chosun Beach Hotel) 12/26/78 Cyprus Cyprus Pipe Industries Limited 12/00/78 Indonesia P.T. Semen Cibinong 01/02/79 Argentina Alpargatas S.A.I.C./Tejidos Argentinos S.A. 01/31/79 Korea Korea Securities Finance Corporation 03/00/79 Brazil Companhia Siderurgica da Guanabara (COSIGUA) 03/07/79 Philippines Philagro Edible Oils, Inc. 03/27/79 Madagascar Societe Textile de Majunga (SOTEMA) 03/28/79 Malavi David Whitehead & Sons (Malawi) ltd. 03/28/79 Turkey Aslan Ve Eskihisar Muttehit Cimento Ve Su Kireci Fabrikalari A.S. (Aslan) 03/29/79 Korea Korea Zinc Company 04/12/79 Colombia Compania Colombiana de Clinker, S.A. 04/13/79 Philippines Philippine Polyamide Industrial Corporation 05/08/79 Thailand United Sugar Terminal Ltd. 05/15/79 Brazil FMB S.A. Productos Metalurgicos 05/25/79 Ivory Coast Etablissements R. Gonfreville (ERG) S.A. 06/01/79 Morocco Societe des Ciments de Marrakech (ASMAR) 06/08/79 FY80 Brazil Capuava Carbonos Industrialis (CCI) 07/02/79 Mexico Mexinox, S.A. 07/13/79 Brazil TECANOR - Textiles Catarinense do Nordeste S.A. 07/31/79 Kenya Rivatex 08/07/79 Zambia Century Packages Limited 09/05/79 Brazil Oxiteno Nordeste S.A. 10/31/79 Trinadad/ Canning and Company Limited 11/27/79 Tobago Ecuador Sociedad Agricola e Industrial San Carlos S.A. 11/31/79 Kenya Hotel Properties Limited (KHP) 12/16/79 Brazil Fabrica de Tecidos Tatuape S.A. (TATUAPE) 12/31/79 Guatemala Cementos PROGRESO, S.A. 03/14/80 Bolivia Molino Andino S.R.L. 04/30/80 Philippines Acoje Mining Company, Inc. 06/30/80 ANNEX H Page 3 of 5 - 117 - Country Project PCR Date FY81 India Escorts Limited 08/29180 India Mahindra Ugine Steel Company Limited 09/15/80 Mexico Mexinox, S.A. 09/25/80 Philippines Cebu Shipyard & Engineering Works Inc. (CSEW) 10/00/80 Costa Rica Scott Paper Company de Costa Rica, S.A. 10/27/80 Guyana Industrial Domestic and Electrical Appliances 12/31/80 Lesotho Lesotho Quality Aggregate Industries (LQAI) 01/15/81 Korea Chonju Paper Manufacturing Co. Ltd. 02/06/81 Argentina J. Minetti, S.A. 02/12/81 Mexico Papeles Ponderosa, S.A. 03/06/81 Brazil Cimetal Siderurgia 03/20/81 Nepal Soaltee Hotel Limited 04/02/81 Mexico Tereftalatos Mexicanos, S.A. (TEMEX) 04/16/81 Uruguay Acodike Supergas S.A. 04/00/81 Yemen A.R. Yemen Dairy and Juice Industries Ltd. 05/00/81 Turkey Trakya Cam Sanayii A.S. 05/22/81 Panama Vidrios Panamenos, S.A. 06/05/81 Thailand Bangkok Glass Industry Co., Ltd. 06/19/81 FY82 Jordan Jordan Lime and Brick Industries 08/10/81 Morocco Asment de Temara (ASMENT) 11/18/81 Brazil Mineracao Rio do Norte S.A. (MRN) S.A. 12/11/81 Trinidad/ Carib Glassworks Limited 12/11/81 Tobago Sri Lanka Mikechris Industries Limited 01/07/82 Thailand Siam City Cement Co., Ltd. 02/17/82 Korea Korea Investment Finance Corporation 04/19/82 Brazil Villares Industrias de Base, S.A. 05/19/82 Ivory Coast Societe Ivoirienne d'Engrais 06/30/82 FY83 Rwanda Societe Rwandaise pour la Production et la Commercialisation du The S.A.R.L. 09/27/82 Swaziland The Royal Swaziland Sugar Corporation 10/08/82 Malawi Dwangwa Sugar Corporation Limited 10/29/82 Sierra Leone Sierra Cement Manufacturing Company Ltd. 11/16/82 Cameroon Societe Camerounaise de 1'Aluminium Pechiney/ Ugine (ALUCAM) 12/17/82 Ivory Coast Moulins du Sud-Ouest S.A. 12/17/82 Argentina J. Minetti, S.A. Puesto Viejo 12/23/82 Cameroon Societe Camerounaise de Minoteries 01/24/83 Philippines General Milling Corporation 02/16/83 Argentine IPAKO S.A. POLISUR Project 03/28/83 Senegal Novotel Dakar 04/13/83 ANNEX H Page 4 of 5 - 118 - Country Project PCR Date FY83 (Cont'd) Paraguay Empresa Hotelera de Encarnacion S.A. 05/26/83 Turkey Ege Mosan Motorlu Araclar Sanayi ve Ticaret A.S. 06/07/83 Malawi Ethanol Company Limited 06/16/83 Brazil Bering do Nordeste S.A. 06/24/83 Brazil Textil Catarinense do Nordeste 06/24/83 Mexico Conductores Monterrey, S.A. 06/27/83 FY84 Yugoslavia Radoje Dakic 09/15/83 Yugoslavia SOKO Compressor Project 09/19/83 Mexico Cementos Veracruz, S.A. 09/20/83 Kenya Panafrican Paper Mills (EA) Limited 10/17/83 Ecuador La Cemento Nacional C.A. 10/31/83 Morocco Societe Miniere du Bou-Gaffer 12/02/83 Jamaica West Indies Glass Company Ltd. 03/22/84 Peru Compania de Minas Buenaventura, S.A. 04/06/84 Brazil Empresa de Desenvolvimento de Recursos Minerais, "CODEMIN" 05/25/84 FY85 Philippines Philippine Associated Smelting and Refining Corporation 07/30/84 Gambia Kombo Beach Hotel Ltd. 10/01/84 Thailand Bangkok Glass Industry Co. Ltd. 02/15/85 Tanzania Metal Products Limited 03/28/85 Zambia Kafue Textiles of Zambia Limited 03/28/85 Zambia Zambia Consolidated Copper Mines Ltd. 03/28/85 Tanzania Highland Soap and Allied Products 04/25/85 Malawi David Whitehead and Sons (Malawi) 04/25/85 Mexico Hotel Camino Real Ixtapa, S.A. 05/16/85 Senegal Industries Chimiques du Senegal, S.A. 06/18/85 FY86 Zimbabwe Wankie Colliery Company Limited 07/02/85 Mexico Vidrio Plano de Mexico, S.A. 07/25/85 Turkey Kirklareli Cam Sanayii A.S. 09/18/85 Kenya Tetra Pak Converters Limited 01/06/86 Cameroon Societe Sucriere du Cameroun (SOSUCAM) 01/15/86 Burundi Verreries du Burundi, S.A.R.L. 04/14/86 Turkey Mensucat Sanayi e Ticaret, A.S. 05/15/86 Morocco Cimenterie Nouvelle de Casablanca 05/27/86 Egypt Alexandria National Iron and Steel 05/28/86 Madagascar Pecheries de Nossi-Be S.A. 06/06/86 Argentina Alpesca S.A. 06/13/86 ANNEX H Page 5 of 5 - 119 - Country Project PCR Date FY87 Yugoslavia The Dr. Simo Milosevic Institute for Physical Medicine and Rehabilitation at Igalo 11/11/86 Kenya Bamburi Cement Co. Ltd. 01/08/87 Yemen National Company for Vegetable Oil and Ghee Industries 03/23/87 Cyprus Dome Hotel Limited 04/14/87 Argentina Alpagartas 04/16/87 Malaysia Pacific Hardwoods 06/02/87 India Modi Cement 06/02/87 Egypt Delta Sugar 06/12/87 Turkey Pinar 06/12/87 Egypt Suez Cement 06/23/87 Yugoslavia Iskra 06/23/87 India Nagarjuna Steel 06/23/87 India Nagarjuna Signodes 06/23/87 India Nagarjuna Coated Tubes 06/23/87 Pakistan Habib Arkady 06/23/87 Kenya Panafrican Paper Mills 06/26/U7 FY88 Panama Bladex 07/06/87 Portugal Banco Portugues de Investimento 07/22/87 Yugoslavia Ruen 11/12/87 Morocco Frumat 01/19/88 Thailand Siam City Cement 02/25/88 Colombia Enka de Colombia 02/26/88 Kenya Leather Industries of Kenya Ltd. 04/15/88 Bangladesh Industrial Promotion & Development Co. 04/20/88 Turkey Cam Elyaf Sanayii, A.S. 04/26/88 Tunisia Societe Miniere de Spath Fluor et de Barytine (FLUOBAR) 05/00/88 Zaire Grands Hotels Du Zaire 05/11/88 Yugoslavia INA-Naftaplin, Molve II 05/12/88 Indonesia P.T. Papan Sejahtera 05/12/88 Brazil Cimento Caue 06/06/88 India India Rayon & Industries Ltd. (IRIL) 06/06/88 Brazil CIHINAS 06/07/88 India Deepak Fertilizer & Petrochemicals Ltd. (DFPL) 06/10/88 Congo Congolaise Industrielle des Bois (CIB) 06/15/88 Pakistan Mari Gas Company Limited (MGL) 06/17/88 Madagascar La Cotonniere d'Antsirabe (COTONA) S.A. 06/17/88 Yemen Yemen Hunt Oil Company 06120/88 Barbados Town & Conercial Properties (Barbados) Ltd. (TCP) 06/21/88 Turkey Eska Turizm ve Ticaret A.S. 06/23/88 Turkey Silkar Turizm Yatirim ve Isletmeleri A.S. 06/23/88 Philippines All Asia Capital & Leasing Limited (AACL) 06/30/88 Mexico Proteison 06/30/88 Costa Rica Scott Paper 06/30/88