Report No. 3102 FILE col" Annual Report on Operations Evaluation August 25, 1980 Operations Evaluation Department FOR OFFICIAL USE ONLY Document of the World Bank This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. FOR OFFICIAL USE ONLY THE WORLD BANK Washington, D.C. 20433 U.S.A. Office of Director-General Operations Evaluation August 25, 1980 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Annual Report on Operations Evaluation Attached, for information, is this year's "Annual Report on Operations Evaluation." The report reviews the current status of the process by which the Bank, in its continuing search for ways to improve the effectiveness of its development assistance, evaluates the results of its operational activities and communicates the fruits of this experience to its members and its staff. The report also contains a summary of the current status of the recommendations in OED major policy reviews and evaluation studies. Attachment This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. FOR OFFICIAL USE ONLY ANNUAL REPORT ON OPERATIONS EVALUATION Table of Contents Page No. I. Introduction 1 II. The Project Performance Audit Reporting System 1 Volume 1 Environmental Aspects 3 Cost 4 Borrower's Role 4 III. Monitoring and Evaluation in Projects 4 IV. Post-evaluation in Member Governments 7 V. The Operations Evaluation Department 10 VI. Other Evaluation Activities 12 IFC 12 Research 13 Economic Development Institute 14 Country Economic and Sector Work 15 VII. Follow-up 16 Annex A: The Current Status of Recommendations of OED Major Policy Reviews and Evaluation Studies 17 Annex B: Philippines - Post-evaluation System 37 Annex C: OED Reports Issued During FY80 47 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. ANNUAL REPORT ON OPERATIONS EVALUATION Abbreviations used JAC Joint Audit Committee OED Operations Evaluation Department PPAR Project Performance Audit Report PCR Project Completion Report CPS Central Projects Staff OMS Operational Manual Statement IFC International Finance Corporation EDI Economic Development Institute M&E Monitoring and evaluation RORSU Rural Operations Review and Support Unit DEDRB Urban and Regional Economics Division of the Development Economics Department ANNUAL REPORT ON OPERATIONS EVALUATION I. Introduction 1. The purpose of this report is to review recent develop- ments in the Bank's operations evaluation activities and the status of still outstanding recommendations of past OED reports. The report assumes that readers are familiar with "Operations Evalua- tion: World Bank Standards and Procedures," Second edition, August 1979. Details of the earlier evolution of operations evaluation in the Bank are available in past Annual Reports: R76-245, October 12, 1976; R77-256, September 28, 1977; R78-206, August 28, 1978; and R79-225, August 30, 1979. 2. This year's report makes clear again that the operations evaluation function is now well established in the Bank. Indeed, its principal features--comprehensiveness in coverage, full disclo- sure in reporting, a prescribed role for operational staff and persistent effort to enlarge the borrower-s role are, in combina- tion, unique. The function, which offers continuing commentary on the effectiveness of the Bank's development assistance, is apparently being followed with interest in those member governments and development assistance agencies concerned with establishing or upgrading their own systematic evaluation of development plans and projects. II. The Project Performance Audit Reporting System Volume 3. In FY80, the Operations Evaluation Department (OED) issued reviews of 109 completed projects. This number is substantially below the number expected to have been issued during FY80. The shortfall is due to the receipt by OED of fewer PCRs than expected: 90 as against 145. The following table shows the number of complet- ed projects in respect of which PCRs were received by OED and those for which PPARs were issued in FY78-80 compared with projections; the estimates for FY81 are also given. - 2 - PCRs/PPARs Estimated and Actuals (FY78-80) PCRs Projects Audited b/ Fiscal Estimate a/ Received Original Year of PCRs By OED Estimate Actual 1978 87 114 130 124 1979 147 130 155 L/ 91 1980 145 90 145 S! 109 1981 162 - 160 - a/ As shown in the Bank's Administrative Budgets. bI As shown in OED's FY78-80 Work Program and Budget documents. cI Subsequently revised to 135 and 125 projects for FY79 and FY80 respectively, due to reduced receipts of PCRs. 4. The PCRs received by OED in FY79 were somewhat less than the original anticipation; they were, however, bunched in the second half of the year resulting in a carry-over of a number of PPARs to FY80. In the first six months of FY80, the receipt of PCRs showed a further drop, only 38 having been received in the first six months, the total for the year being only 90 against the original estimate of 145. -The PCRs received in FY80 have thus been 30% below the previous year's level, and almost 38% less than the number original- ly programmed. The main reasons would appear to be competing demands for staff resources, especially since a special effort was made in the year to build up the project pipeline for the next year, certain projects taking longer to complete than anticipated, and delays in the receipt of information from borrowers. 5. The smaller-than-anticipated number of PCRs issued in FY80 does not alter the prospect that, given the substantial increase in Bank operations over the last few years, OED will need to audit an increasing number of completed projects. The 200 completed projects reviewed in the last two years is about double that of four or five years ago, and is expected to double again in the next few years. The FY81 Administrative Budget anticipates that completion reports are expected to be issued for more than 200 projects in both FY84 and FY85 (R80-129, May 16, 1980, p.31). - 3 - 6. Several steps have been taken during the year to strength- en OED's ability to cope with the expected increase in the volume of completed projects coming up for review, while sustaining special studies at about the level now reached. The first has been to increase the number of OED's authorized professional and research assistant staff effective FY81 by four staffyears, or 16%, over FY80. The second is to improve the effectiveness of OED-s recruit- ment and upgrade the experience level of staff in OED through reassignment in and out, in which substantial progress is now being made. The third step is to contain the unit cost of PPARs in the face of the changing mix of projects now coming up for review by increasing the proportion of abbreviated audits. Explicit criteria to be used by OED staff in determining which completed projects should be subject to the abbreviated audit process have been formu- lated and the proportion of abbreviated audits has been estimated at 40% for FY81. 1? The FY80 goal of 50% mentioned in last year's Annual Report was not reached given that more than one-half the projects audited were in the agriculture and education sectors, which by their nature largely receive full audits. The actual percentage of abbreviated audits for the year was 26% and of inter- mediate audits, 27%. Experience with abbreviated audits will be monitored closely and reviewed with the Joint Audit Committee (JAC); a first discussion has already taken place as part of the JAC's most recent Subcommittee review of selected PPARs. (See below) 7. An internal Subcommittee of the JAC, comprising six Alternate Executive Directors, has reviewed a number of PPARs each year since 1977; in respect of FY80, the Subcommittee selected and discussed six PPARs and related documents in depth with relevant OED and operational staff. While making some suggestions for further improvement, the general conclusion of the Subcommittee in its report for FY80 was "...that OED has managed its audit function efficiently and economically, that the approach taken by the PPARs is appropriate and that the PPARs are of consistently high quality." (para. 7). The report of the Subcommittee was distributed to Executive Directors as an attachment to the minutes of the JAC meeting at which it was considered (JAC/M80-4, June 6, 1980). Environmental Aspects 8. The Office of Environmental Affairs of the CPS Projects Advisory Staff DOEA) examines the environmental aspects of projects where relevant. Steps have been taken for certain projects which are due for audit to be reviewed by OEA with a view to obtaining 1/ FY81 Work Program and Budget for OED, R80-75, April 8, 1980. OEA's assessment of the environmental impact of such projects and its comments on OED's analysis and conclusions. Two PPARs distri- buted to the Board in FY80 were subjected to such a review and OEA-s comments were taken into account in the final version of the PPARs. Eight further projects have been identified as meriting such review and, in due course, the relevant PPARs will take into account the assessments of OEA. Cost 9. The cost of preparing PCRs currently averages 10.4 staff- weeks and, of PPARs, 3.8 staffweeks. The cost to the Bank of the project performance audit process thus now approximates 14 staff- weeks per project. Borrower-s Role 10. Few PCRs are as yet completed by borrowers--only eight PPARs issued in FY80 included PCRs prepared by borrowers--but many borrowers now contribute substantially to their preparation. Comments on PPARs continue to be made by most borrowers, either to OED missions or by correspondence (83% of the PPARs issued in FY80 reflected such comments). OED country visits were made to 50% of the projects for which PPARs were issued in FY80. Borrowers' written comments continue to be reproduced in full in PPARs. 11. In the years ahead much of the responsibility for pre- paring PCRs will pass to the borrowers. But this prospect is not yet widely perceived either by the Bank's operational staff or by borrowers as an opportunity to support the establishment of the broader national evaluation function that many Part II governments say they are interested in. (See Section IV below.) III. Monitoring and Evaluation in Projects 12. Monitoring and evaluation (M&E) elements built into projects have been discussed in earlier Annual Reports. OED has prepared two reports on experience to date, one in October 1977 (SecM77-745) and the other in November 1979 (SecM79-788). The first report dealt with M&E in six projects in the agriculture sector and the second, with experience in six education projects. A third such review will be initiated shortly. While the two reviews found significant deficiencies in the operation of M&E in some projects-- poor or no results because of lack of trained staff, unfocussed data collection, inadequate data analysis, and over-ambitious M&E objectives and design--they also observed progress, particularly as regards borrower awareness of the importance of M&E and borrower response to their early experience. - 5 - 13. Provision for M&E continues to be made under many Bank projects, especially in rural and urban development, education and family planning. In urban projects, of the ten projects approved in FY80 seven provide for M&E in the form of the continuation of an existing system or the creation of a new one. In 1975, IBRD and the International Development Research Centre (IDRC) in Canada agreed to jointly sponsor a five-year evaluation of some of the first IBRD- funded urban shelter programs. The study, which originally included projects in Zambia, Senegal and El Salvador, was expanded in 1977 to include the Tondo Foreshore Project in the Philippines. By that time, a total of approximately 15 project sites, designed to affect approximately half a million people and covering both squatter upgrading and sites and services, were included in the IBRD/IDRC study. The Urban and Regional Economics Division (DEDRB) of the Development Economics Department has been responsible for the design and supervision of the evaluation and for the provision of technical assistance to the national research teams established in each country. 14. The research methodology was developed jointly by the participating countries and DEDRB, and has been reviewed annually in conferences of researchers and program management. The evaluation covers four broad areas: project efficiency, effectiveness in achieving the original objectives and in making the projects acces- sible to the low-income target population, longitudinal impact studies, and an analysis of the effects on housing policy and the urban housing market. The five-year program will end with a final conference in late 1980, the publication of a final report on each country program, and a series of comparative studies and economic analyses of issues such as affordability, estimates of the value of housing investment, the economics of self-help housing and the composition of household income among the urban poor. 15. Although the original agreement ends in 1980, the evalua- tion will continue in Senegal and the Philippines for at least one more year. DEDRB is also assisting in the design and implementation of similar evaluations in cooperation with the Urban Projects Department in Indonesia, India (Madras), Kenya and Colombia. Manuals are being prepared on the design and implementation of monitoring and evaluation systems for urban shelter programs, and a book is being written on the experience of IBRD and other organiza- tions in evaluation in this sector. 16. Of the ten education projects approved in FY80, five included financing for evaluation and two provided for technical assistance; tracer systems are already in place for evaluating the results of the three remaining projects. - 6 - 17. The Bank has encouraged the dissemination of experience with M&E most actively in the agriculture and rural development sector through a series of Regional workshops. The most recent of these workshops was held in Kuala Lumpur in December 1979 to con- sider M&E in rural development projects in the East Asia and Pacific region.l Thirty-six officers from seven member countries attended this workshop, which was organized by the Rural Operations Review and Support Unit (RORSU) of the CPS Agriculture and Rural Develop- ment Department, in collaboration with the East Asia Region's Projects Department. They discussed 21 case studies submitted by the participants in advance. The principal conclusions drawn from this workshop were that: few project managers were highly committed to M&E, mainly because they have not been involved in its design, but also because of doubts concerning the quality of information presented by the monitoring unit; the monitoring systems are some- times too elaborate to be relevant to the project management's needs; and ineffective use is often made of monitoring information by project managers because they lack the understanding and/or the authority to do so. 18. It was noted at the Kuala Lumpur workshop that while reference to M&E in appraisal reports was made in 70% of the 64 projects in agriculture and rural development supported by the Bank in the East Asia and Pacific region over a five-year period, only 17 of these projects made financial provision for M&E. The workshop concluded that without such finance, the effective operation of units is administratively difficult, if not impossible. A report on the workshop is expected to be distributed shortly. 19. 1A similar workshop was organized by the Bank's Latin American Region in San Jose, Costa Rica in October 1979. The focus of this workshop was on sharing M&E experience in agricultural credit and livestock projects as well as integrated rural develop- ment projects. Participants came from nine Latin American countries and comprised 33 project staff members, representing 16 different institutions and associated with 24 active Bank-financed projects. The participants concluded that it was in their countries' interest to establish M&E to improve project performance and related programs and strategies. As in Kuala Lumpur, the participants also emphasized the need for adequately trained personnel in M&E units and for less elaborate M&E arrangements. I/ The first workshop was held in Nairobi in April 1979; a summary of its discussions and findings are contained in the World Bank Staff Working Paper "Managing Information for Rural Develop- ment: Lessons from Eastern Africa" (No. 379 of March 1979). - 7 - 20. The evidence suggests that M&E performance to date has been falling far short of the intentions of both borrowers and the Bank. Borrowers have not generally been providing adequate staff for M&E units nor ensuring that such units are given a status appropriate to the functions so that the collection, analysis and dissemination of information can be effective; and the Bank--whose catalytic role was highlighted at the San Jose workshop--has not been providing systematically for financing M&E in projects nor helping sufficiently to establish appropriate M&E design. In response to this experience, steps have recently been taken to strengthen Bank staff resources in this area in the Regions and in CPS. Further RORSU workshops are planned for FY81, and also more direct assistance to borrowers and Bank staff on appropriate objec- tives and design of M&E for specific projects. IV. Post-evaluation in Member Governments 21. In addition to the initiatives referred to in the pre- ceding section, the Bank has continued to encourage post-evaluation in borrowing countries through the Office of the Director-General, Operations Evaluation. A third regional seminar on operations evaluation was held in Kuala Lumpur on May 29-31, 1980 (SecM80-506). Participants from 13 Asian member countries discussed the Bank's experience alongside their own, and agreed on the desirability of establishing systematic feedback from experience with major projects to enhance the effectiveness of their national development manage- ment, with assistance where possible from the World Bank. These views are very similar to those expressed by participants in the two earlier regional seminars on operations evaluation--for French- speaking African member governments in February 1978 in Abidjan (SecM78-207, March 16, 1978) and for English-speaking African member governments in February 1979 in Nairobi (SecM79-160, March 20, 1979). The Director-General and the Director, OED also separately visited various member countries during the year, in the course of which discussions were held on evaluation in these governments and on their involvement in the evaluation of their Bank-financed projects. 22. Staff of member governments continue to visit OED. In FY80, twelve officials from key countries visited, for periods ranging from one to two weeks, on programs which exposed them to the project cycle as experienced in the Regions, CPS and OED. 23. EDI now includes monitoring and evaluation as an integral part of most of its project courses (see last year's Annual Report), whether taught in Washington by EDI staff or in member countries by staff of regional or national training institutions with which EDI is actively collaborating. - 8 - 24. However, these various initiatives cannot be said to have been fruitful to date. There remains a conspicuous gap between the expression of interest by member government officials in systematic analysis of project effectiveness and meaningful steps by their governments to this end. Even effective project implementation monitoring, which is substantially removed from post-cumpletion evaluation, is uncommon among members, as the Bank's special imple- mentation reviews for Bank-financed projects in several member countries indicate. As far as is known, only one Part II member government--the Philippines-- has taken steps to initiate a national project evaluation system to which future Bank PPARs will be able to connect. The recent Philippine arrangements are reproduced with permission in Annex B of this report, in view of their possible interest to other member governments. 25. In light of the above, thought had once been given to reviewing briefly in this and succeeding annual reports where the Bank's major borrowers stand in institutionalizing some kind of national project evaluation function. This idea has been dropped, in the short run because of inadequate information about country practices and intent, but principally because of the substantial diversity of views about what are reasonable country goals in this regard. There are senior borrower government and Bank staff who believe that solving pressing implementation problems must take priority and that, until this is done, even limited attention to evaluation cannot be justified in administrations chronically short of qualified staff. The fact is that the subject is not yet a matter of systematic Bank operational dialogue with member govern- ments, not even with those which are now or will soon be engaged with Bank staff in preparing numbers of project completion reports. 26. When those who express reluctance about completion report- ing are questioned about their apparent lack of concern about the results of development expenditures which had been subject to considerable ex-ante analysis and review, most, whether in member governments or in the Bank, respond that they are just as concerned about effectiveness as about the commitment and expenditure of funds; they are only reluctant to impose on over-burdened govern- ments the cost of reporting as they perceive it to be now prescribed by the Bank. In their view, few direct benefits are seen to accrue to these governments from their undertaking such reporting since all the needed adjustments to projects during implementation and the informal identification of lessons to be drawn from experience with particular projects have already taken place; the completion report- ing standards of the Bank are thus to be avoided and not replicated locally. - 9 - 27. To the extent that concerns about unduly elaborate report- ing now inhibit borrowers, it is incumbent on operational staff to encourage borrowers to be selective on issues and flexible in their reporting on experience with completed projects, as the Bank's own PCR guidelines encourage Bank staff to be. Erroneously rigid perceptions of what the Bank's Executive Directors, management, or the Regions "require" in such reporting could well frustrate the objective of enlarging borrower involvement in the Bank's evaluation process. 28. The most obvious candidates for greater involvement in the PCR/PPAR process are those Part II member governments which either have or will soon have sizeable numbers of completed projects coming up for review in any given year:V 18 of these countries already have five or more PCRs scheduled during the next 18 months. Within this group, the perceived need for assessing project experience and the administrative capabilities for doing so vary widely. In a few countries, there has been substantial progress in developing sector and/or program-oriented monitoring and evaluation mechanisms; but in many, the present level of effort is either rudimentary or non- existent. The preparation and review of forthcoming PCRs/PPARs in a manner designed to support and assist in the development of a domestic project evaluation process is currently being contemplated in only one of these countries (para. 24). However, what is known about the others suggests that there is both interest in and the potential for comparable arrangements in several. This is why the brief report on the Kuala Lumpur regional seminar on operations evaluation (SecM80-506, June 23, 1980) observed: "The immediate operational conclusion for the Bank would appear to be that PCR missions henceforth (i) seek out and involve country evaluation staff, including seminar participants wherever appropriate, in all future reviews of completed Bank-financed projects, and (ii) report in the PCR the nature of the bor- rower's involvement in it." 1/ Algeria, Argentina, Bangladesh, Brazil, Colombia, Egypt, Hondu- ras, India, Indonesia, Ivory Coast, Kenya, Korea, Malaysia, Mexico, Morocco, Nigeria, Pakistan, Peru, Philippines, Romania, Senegal, Sudan, Thailand, Turkey and Yugoslavia. - 10 - 29. The time seems ripe for more deliberate effort to link the Bank's PCR/PPAR process with appropriate central as well as project and sectoral agencies in these countries. This linking could usefully begin to be included in the agenda of matters subject to systematic operational planning and review within the Bank. V. The Operations Evaluation Department 30. Last year-s Annual Review of Project Performance Audit Results (R79-229) was issued on August 27, 1979, and the last Annual Report on Operations Evaluation (R79-225) on August 30, 1979. The Annual Review, with only the names of countries and projects re- moved, was issued for the general public in November 1979 following the precedent set in respect of the Reviews for 1977 and 1978. This year's Annual Review will be issued shortly. 31. The findings of the 514 project performance audits on which reports were issued by OED through December 31, 1979 have been classified and entered into OED's computerized memory and retrieval system, which has now been in place for more than two years. The first computerized Concordance to Project Performance Audit Reports was issued on August 31, 1978 (SecM78-699). An updated version of the Concordance covering project audits and other reports of OED up to December 31, 1978 was issued on May 15, 1979 (SecM79-356). A study is underway in OED, with the cooperation of the Computing Activities Department, to expand and improve upon the system with a view to making the considerable information in it more readily available for use by Bank staff. A further revised and updated version of the Concordance is expected to be published shortly. 32. During FY80, OED issued seven special studies, among which were four Impact Evaluation Reports in the "second look" series: on an irrigation project in Mexico (SecM79-526), a smallholder agricul- tural project in Kenya (SecM80-370), an irrigation project in Sudan (SecM80-539), and two education projects in Colombia (SecM80-540). In addition, a study on the Supervision of Bank Projects (SecM80- 129), a study on Delays in Project Implementation (SecM80-280), and the second review of Built-in Project Monitoring and Evaluation Systems (SecM79-788) dealing with six projects in the education sector were published. All the foregoing studies have been reviewed by the JAC and the studies on Supervision and Delays in Implementa- tion by the full Board. An Impact Evaluation Report on a Papua New Guinea Smallholder Development project was issued on July 14, 1980 (SecM80-556) and the Sector Operations Review: The Industries and DFCs Program in Turkey, was issued on July 21, 1980 (SecM80-573) and reviewed by the JAC on July 30, 1980. - 11 - 33. OED's current program includes reviews of experience with loan covenants, training in Bank-financed projects, a second review of agricultural operations in a country, experience with institution- building, a third review of built-in project monitoring and evalua- tion, and a number of impact evaluation studies in the "second look" series. In addition, an interim report on procurement is expected to be issued in the near future. 34. Several items in this program are currently being reviewed in the light of related work underway elsewhere in the Bank. There have been occasions in the past when specific OED studies have coincided with similar work undertaken in other parts of the Bank. A number of years ago, OED plans for a review of population projects were cancelled when they turned out to coincide with the Bank's decision to establish an External Review Panel: significant OED work had not yet begun, and the costs of parallel exercises were not considered to be warranted. More recently, there has been a similar parallel initiative--in education. In this case, signifi- cant OED work was already underway when the External Panel first convened and both reviews proceeded to completion. (Report of the External Advisory Panel on Education, SecM78-817 of November 6, 1978; Review of Bank Operations in the Education Sector, SecM79-3 of December 29, 1978.) The two studies were complementary and contri- buted to the education sector policy paper then being prepared; but the experience suggested that, in future, benefits to the Bank could be even greater with more deliberate advance planning of potentially overlapping studies. OED accordingly intends, henceforth, to consult with management on work plans sufficiently early to ensure complementarity of effort, in the expectation that, whenever a subject suggested for OED's work program is taken up for internal Bank study and not included in the OED work program in order to avoid overlap, the internal staff report is made available to interested Executive Directors in an appropriate manner. Such coordination does not, of course, mean that OED will never undertake studies in areas being covered by other parts of the Bank. The Executive Directors and/or the management may feel it desirable to complement particular staff studies with an independent OED view based on past OED reports, special case studies and dialogue with borrowers. 35. OED's authorized staff now consists of 26 professionals, 5 research assistants and 18 other staff. In last year's Annual Report, reference was made to staffing problems because the re- assignment of staff between OED and the operating departments in the Bank was not taking place in timely fashion. The experience of the past year has shown a considerable improvement in this respect, - 12 - reflecting in part the operation of the Bank's Reassignment Panels and the implementation of the Career Planning and Management Poli- cies published by the Personnel Management Department in July 1979. Two OED staff members transferred to other parts of the Bank in FY80; others are expected to do so in the next few months. Of the seven higher level vacancies in the Department on July 1, 1980, of which three were new positions authorized for FY81, firm arrange- ments have been made for the recruitment of five staff by reassign- ment from regional projects departments. Suitable candidates for the remaining two positions are being sought. Responsibility for OED staff within the Personnel Management Department was reallocated in July 1980 from the section dealing with support staff to the section dealing with operational staff, to facilitate the matching of OED and operating staff reassignments and vacancies. VI. Other Evaluation Activities 36. When the Executive Directors considered the first Stan- dards and Procedures document in 1976, they said that operations evaluation should be enlarged to include IFC projects and Bank research. Annual reports on operations evaluation since have described the status of evaluations of these activities, and also of the others noted below, even though these evaluations do not yet include substantive involvement by OED. IFC 37. IFC continued to prepare and send to OED, as agreed, completion reports on all projects. In FY80, IFC met its objective of producing 12 completion reports. The projects covered were in food processing, textiles, plastic wrapping, cement, chemicals and petro-chemicals, and iron and steel industries, located in ten countries. Reflecting the experience gained in earlier PCRs, these reports review systematically project appraisal objectives, imple- mentation experience, and financial and economic prospects and draw lessons from the projects which are generally applicable to the Corporation's work. As a body of PCRs is built up over time, they will serve as prime source material for IFC Sectoral Evaluation Reports, which will be submitted to the Director-General, Operations Evaluation, for OED review and subsequent submission to the Board. 38. IFC's first Sectoral Evaluation Report, which is being completed with OED support, is expected to be issued shortly. This report deals with all IFC investments in hotels made to date and focusses intensively on four projects to illustrate the principal issues and problems which confront IFC in dealing with this sector. With OED's assistance, terms of reference are being completed for IFC's next Sectoral Evaluation Report, which will cover the textile industry. - 13 - 39. As part of its study of industry and DFC operations in Turkey, OED analyzed the impact of IFC-financed projects in the country, including the economic and financial returns on a number of these projects. The conclusions appear in the Sector Operations Review: The Industries and DFCs Program in Turkey, which was distributed to the Executive Directors on July 21, 1980 (SecM80-573). Research 40. As noted in last year's report, OED's response thus far to the Board's direction to include research in the Bank's evaluation system has been to report on the Research Committee's evaluation process. Since six subject-oriented external panels! as well as an overview panel2/ have evaluated the Bank's research program in the last two years, no further OED initiative is currently planned. 41. In addition to endorsing the quality of Bank research, the overview panel made a number of recommendations. Among these, the most important, from an operations evaluation point of view, was that which emphasized the need for Bank researchers to make greater use of the Bank's rich operational experience in various subject areas. This is one of the many matters receiving the attention of the research Steering Groups that were formed at the recommenda- tion of the overview panel. These bodies, of which four have already been established (for Industry, International Economy, Agriculture and Rural Development and Urban and Regional Economics), are designed to bring research and operational staff together and to facilitate a joint determination of the Bank-s research priorities. OED is currently exploring with the managers of Bank research and with the Steering Groups the potential for more effective complemen- tarity than now exists between the Bank's research and its opera- tions evaluation reporting. 1/ Report of the Research Advisory Panels on Income Distribution and Employment (SecM78-493, June 6, 1978); and on Agriculture and Rural Development; Commodities; Energy, Water and Telecom- munications; Industrial Development and Trade; and Transport (SecM79-461, June 1979). 2/ Report of the General Research Advisory Panel (R79-221, August 1979). - 14 - 42. OED continues to be represented on the Research Com- mittee's internal evaluation panels for completed research projects, the findings of which are reported to Executive Directors in the Annual Reports on the Bank's Research Program. As agreed with Executive Directors in their 1976 meeting, a copy of each evaluation panel report on these completed research projects is filed with the Director-General for reference by any interested Executive Director. Four such reports were filed for CY79. Economic Development Institute 43. EDI continues to request participants in many of its Washington (and increasingly in overseas) courses to complete weekly evaluations to check (1) the relevance and usefulness of individual sessions and (2) the quality of the speaker's presentations. In some courses, interim questionnaires are used as a more extensive form of evaluation. At the end of each course, the weekly and/or interim evaluations are summarized and participants are also asked to express their overall judgments of the course. 44. EDI is also exploring other ways of extending these course evaluations, such as the recent experimental mail survey of the participants who completed an agro-industrial/industrial projects course in Arusha, Tanzania in 1978. 45. In addition, EDI will continue to evaluate the effective- ness of other aspects of courses and their presentation, including the process by which participants are selected. 46. As was stated last year, EDI has recently been paying particular attention to the evaluation of its courses given in developing countries in partnership with other agencies. These cooperative programs are intended to serve two objectives: (1) a large increase in the number of LDC officials skilled in project planning, analysis, and execution; and (2) the development of local training programs and institutions to the point where with local staff, facilities and equipment, and financing, they will be able to operate without EDI assistance. 47. Two studies of the effectiveness of this program have been completed--one in Bangladesh and the other in Sudan. A third study--in Indonesia--is underway. EDI hopes to do two more of these reviews during the next fiscal year. Even in promising situations it is now clear that the development of stable institutional arrange- ments for conducting EDI-style courses will take much longer than the three years originally projected. Experience so far suggests that attention and support has to be given not only to course content, geographical coverage, participant selection, and teaching techniques, but also to the cohesiveness of local institutions, - 15 - the stability of their teaching and administrative personnel, the facilities and equipment available, and the financial commitment which they are willing to make. It is also apparent that successive evaluations, perhaps at three- to five-year intervals, will be needed to make reliable judgments about progress. A summary of the study of the Bangladesh program was included in EDI's Annual Report for FY80 (R80-89, Annex H) and summaries of other such studies are expected to form part of future EDI Annual Reports. 48. A first seminar on monitoring and evaluation for national economic management, scheduled for June 1980, has been postponed because the earlier timing conflicted with important annual budget- ary planning activities in a number of key countries. This seminar will provide occasion for a searching exchange of relevant expe- rience among senior government officials and World Bank personnel, and will thus complement and reinforce the objectives of the Bank's regional seminars on operations evaluation (see para. 21 above). 49. EDI has decided that greater emphasis should be given to the management of projects in EDI courses, and has scheduled for FY81 a course on managing rural development and agricultural pro- jects and a senior seminar on the design and implementation of rural development projects. Both will contain large components on moni- toring and evaluation. EDI has also scheduled a short course on the implementation and evaluation of education projects, which will focus on how to carry out evaluations and how to use the results in decision-making. In two urban projects courses in Africa, one in French (June-July 1980) and the other in English (October-November 1980), the monitoring and evaluation of low-cost shelter projects will receive considerable attention. Country Economic and Sector Work 50. Last year's report referred to a major internal review of country economic and sector work carried out during the last three months of CY 1978 by Development Policy and Central Projects Staff. That review confirmed that the Bank's country economic and sector work was highly valued by both the Bank's borrowers and by the major bilateral and multilateral aid agencies, a view that is occasionally reconfirmed by Executive Directors during Board discussions. At the same time, it was noted that while a significant shift in economic reporting had occurred towards addressing poverty issues, an expand- ed effort was justified to better underpin Bank lending for "new style" projects. - 16 - 51. Lending for structural adjustment has given heightened importance, as well as a new focus, to country economic and sector work. It has become necessary for the Bank in its economic report- ing to move beyond general analysis and recommendations for policy change, to assist countries to elaborate specific action-oriented measures of reform. This calls not only for a high level of econo- mic expertise, but also for experienced staff with a practical knowledge of economic policy making. Sound programs for structural adjustment will depend on both more in-depth sector work and more detailed macro-economic analysis. OED's forthcoming reviews of experience with structural adjustment lending will provide occasion for further staff and borrower comments on this work. VII. Follow-up 52. The process by which the conclusions and recommendations contained in OED reports are disseminated to Bank operating staff includes individual PPARs--which incorporate PCRs--being reviewed by relevant senior CPS and Regional staff; all PPARs for each calendar year being reviewed on a sectoral basis at meetings arranged by CPS and attended by representatives of the Regions and CPS staff; and special studies being reviewed as the subjects dictate. While the process is now well established, the quality of its implementation could still be improved. The responses of Bank management to explicit OED recommendations are tracked in this section of these annual reports. 53. The record shows that the Bank's operations and practices have been responsive to the lessons of operational experience highlighted in OED reports. The continuing response to OED's report on The Role and Use of Consultants in World Bank (SecM77-865, December 9, 1977) is illustrative of Bank concern with the issues. An initial response took the form of a statement to the Executive Directors by the Vice President, Projects Staff in May 1978, which was subsequently circulated. Since then there have been further developments reported in the Annual Reports for 1978 and 1979 and in Annex A of this Report. A seminar on the subject for Executive Directors will be held on September 10, 1980. 54. The main recommendations of OED special studies and management-s reactions to them were first reviewed in the 1978 Annual Report on Operations Evaluation. The practice that has evolved since then has been to report at least once on each study, and thereafter only if appropriate. Annex A of this Report provides information again on the status of the recommendations contained in the study on consultants referred to above, and on the principal recommendations of the major OED reports published in FY80. - 17 - ANNEX A INDEX Part Report Page No. I Built-In Project Monitoring and Evaluation: A Second Review 1 II The Supervision of Bank Projects 4 III Delays in Project Implementation 10 IV. Fifth Annual Review of Project Performance Audit Results 13 V. The Role and Use of Consultants in World Bank Projects 15 - 18 - ANNEX A THE CURRENT STATUS OF RECOMMENDATIONS OF OED POLICY REVIEWS AND EVALUATION STUDIES Introduction Attached as Annexes A-I to A-IV are summaries of the recommendations made in the major OED reports on aspects of the Bank's operations issued in FY80; the reports are dealt with in chronological order, with the main observa- tions of the Fifth Annual Review at the end. Annex A-V contains information on the status of the recommendations of the Report on the Role and Use of Consultants in World Bank Projects issued on December 9, 1977 which updates Annex A of the Reports on Operations Evalua- tion for FY78 and FY79. The sequence of the recommendations set forth in the attachments is not necessarily as in the Reports themselves. - 19 - ANNEX A Page 1 I. BUILT-IN PROJECT MONITORING AND EVALUATION: A SECOND REVIEW (SecM79-788, November 5, 1979) This Second Review was based on an examination of monitoring and evaluation (M&E) components in six education projects approved over the period 1972-1976 and located in Kenya, Liberia (two projects), Sierra Leone, and Tanzania (two projects). The Review made three recommendations for the better design and implementation of M&E components, and one recommendation concerning resources for M&E. Cautioning that conclusions must remain partial at this early stage of experience with M&E components, the Second Review concluded that "the inclusion of M&E components in the education projects reviewed has been justified and that the Bank's effort has been useful not only in supporting M&E efforts but also in encouraging the rapidly developing importance of M&E and borrower interest in a wide range of M&E activities." The Second Review's specific recommendations are set out below. The Bank fully endorses all four recommendations, with the qualifications noted under the second and third, where account is taken of the Bank's experience in the education sector as a whole and specifically in the period extending beyond the date (1976) of the approval of the last of the projects covered by the second review. (i) Project Preparation Close Bank/borrower consultation and elaboration of project M&E components prior to project approval are needed on the purposes, objectives, approach and methods of M&E as well as on the organizational arrangements and the resources to be made available. (ii) Data Gathering and Analysis Emphasis on the wider adoption of regular and systematic data collection and analysis as a basis for both monitoring and evaluation is urgently needed. Projects should increasingly address the needs in terms of equipment and buildings which such improved data gathering and processing would generate. The above recommendation needs to be broadened in scope. Projects should increasingly address the -20 - ANNEX A Page 2 whole issue of building national analytic capacities and the needs that such a process generates in terms of personnel training, institutional setup, equip- ment, buildings and financial resources. (iii) Tracer Studies Tracer studies in the countries concerned have been both practical and useful. However, care should be taken in designing M&E components to (a) avoid too high a frequency of surveys, (b) keep studies to manageable proportions in relation to the logistical manpower and financial resources available, (c) tap existing staff resources in the field such as district supervisors, and (d) involve as early as possible the staff of the relevant schools in the design and administration of survey instruments intended for the graduating classes of these schools. Most education projects include nowadays built- in monitoring and formative evaluation and many include tracer studies. The experiences during 1977-1979 show that both the Bank and the borrower often underestimated the manpower and work needed to design, carry out and supervise those important activities. - 21 - ANNEX A Page 3 (iv) Resources for M&E Greater attention needs to be devoted to staff development in educational evaluation and related disciplines such as sociology, planning statistics, and research techniques. To this end, projects should finance, where necessary, and on a larger scale than hitherto: (a) the strengthening of the borrowers' universities or institutes of education to enable them to train evaluators, planners and educational researchers and statisticians; and (b) fellowships for the training of personnel in these disciplines. - 22 - ANNEX A Page 4 II. OPERATIONAL POLICY REVIEW: THE SUPERVISION OF BANK PROJECTS (SecM80-129, February 22, 1980) The above report and the management response to its recommendations (R80-142, May 29, 1980) were discussed at a meeting of the Executive Directors on July 15, 1980. A summary of each of the principal recommendations of the report is set forth below. Generally, it has been decided that the next Project Implementation Review will have, as its special topic, the effectiveness of supervision including some of the issues raised by the report. The manage- ment-s comments on the main recommendations specifically are not repeated or commented upon in extenso; only indications are given of the action or further study proposed to be undertaken. Where individual recommendations have not been accepted, this is also indicated. Summary of Principal Recommendations I. Increased Borrower Involvement (i) Supervision Planning Greater borrower involvement in the planning of supervision timing and intervals is needed and should be done early in the project cycle, at appraisal or during loan negotiations and should include an assessment of specialist assistance required during the project start-up period. (ii) Use of Borrower Agencies There is potential for placing greater reliance on borrower agencies in undertaking some supervision func- tions. Bank supervision should be increasingly tailored to each borrower-s management capability and resources; where the borrower has a strong organization, Bank efforts should concentrate essentially on supervising the super- visors. Why several of the Bank's more experienced borrowers involved the Bank in the heaviest supervision cost per project and had a high proportion of the more slowly disbursing projects would appear to deserve special attention. - 23 - ANNEX A Page 5 The Bank intends to examine further the reasons for the high supervision co-efficients of some of the Bank's more experienced borrowers. (iii) Borrower Reports and Their Use Further steps need to be taken to make borrower reports and their use more effective. There is evidence that reporting intervals are sometimes too short. A logical corollary of reporting requirements being dis- cussed during preparation and appraisal would be to have them set out in the appraisal report. The Bank proposes to take appropriate administrative action on the monitoring of reporting by borrowers. (iv) Other Approaches To Borrower Involvement Consideration should be given to extending the practice of mid-term reviews of project progress with borrowers with respect to innovative and complex projects. The continuing expansion of built-in monitoring and evaluation units in projects is encouraged. II. The Conduct of Supervision Missions (v) Timing and Duration of Missions Adequate supervision in the early stage of imple- mentation is necessary and the study raises questions about the present adequacy thereof. In addition, the study suggests the desirability of longer duration missions especially on projects of the more complex type. Specialist technical missions in between main review missions is also suggested. The Bank proposes to examine the issues raised in the above recommendation during the next Project Implemen- tation Review. However, it notes that generalizations about mission frequency and duration are difficult given, for example, the different types of missions and country preferences. - 24 - ANNEX A Page 6 (vi) Mission Composition More conscious selection of supervision staff seems indicated for the needs of particular projects given borrower's views that Bank missions are not always appro- priately staffed. In addition, supervision training and guidance for new staff seems indicated. The Bank will review the guidance being given to new staff, and the need for more intensive staff training focussed explicitly on the supervision function. (vii) Mission Coordination Occasional lack of coordination between operating divisions within the Bank was found to have led to super- vision inefficiencies. Appropriate administrative action will be taken. (viii) Notice to Borrowers Adequate notice of supervision missions should be given to borrowers. Appropriate administrative action will be taken. (ix) Resident and Regional Missions and Resident Projects Staff There is a need to review the present role in super- vision of resident and regional missions and resident projects staff, including their possible role in follow-up in between missions and in reviewing documents. The role of resident missions is being reviewed in the broader context of the study being undertaken by OPD on the role of field offices and Bank decentralization. III. Supervision Reporting and Follow-up (x) Reporting to the Bank Improvements in the coverage and continuity of issues in supervision mission reporting could be achieved (a) by requiring managers to sign off/on reports before their - 25 - ANNEX A Page 7 distribution and (b) if less discretion on topics reported upon were allowed and more structured reporting required so that certain core issues would receive systematic and continuous attention. Furthermore, greater use could be made of devices such as critical path analysis to monitor implementation. On a related matter, it was noted that some project files are in a poor state and borrowers' progress reports frequently could not be traced. The Bank intends to take appropriate administrative action regarding the apparent deficiencies in project files. (xi) Reporting to Borrowers Aide-memoires prepared in the field should normal- ly be used for reporting to borrowers the main findings of supervision missions. Aide-memoires are being increasingly used and encour- agement will be given to their wider application. (xii) Reports as Sources of Lessons Staff should be encouraged to look out for supervi- sion experiences of a general value with a view to their being disseminated throughout the Bank. There is a need for better communication within the Regions and across divisions, as well as Bank-wide, on such experiences. While there is a fairly elaborate system in place to distill the experience from project implementation and supervision, the Bank plans to explore other avenues, including a greater effort by CPS to highlight individual cases where Bank assistance during supervision was par- ticularly effective. Moreover, Project Implementation Reviews have been modified to focus more on policy issues. - 26 - ANNEX A Page 8 IV. Other Assistance and Approaches to Supervision Work (xiii) Country Implementation Reviews Country implementation reviews to deal with common policy and coordination issues in countries with a sizable borrowing program, or those which are having problems, have much to commend them. A broader evaluation of Bank experience with country implementation reviews, the use of which has been extended to many countries, is to be made by the Bank this year as part of the Project Implementation Review. (xiv) Increased Role for Programs Staff There should be increased participation in supervi- sion missions by Programs staff. (xv) Manual on Procedures More specific written guidance on procurement and disbursement procedures should be available to both borrowers and Bank staff and handbooks thereon would be valuable. The Bank hopes to make a start in the near future on the systematic collection of information on local procurement practices and procedures with the view to cataloguing and analysing them and making them readily available to Bank staff. - 27 - ANNEX A Page 9 V. Implications for Operational Priorities and Costs (xvi) Supervision Priority Project appraisal and loan presentation activities appear often to receive priority over supervision of existing projects, although it is not clear if staff allocation reflecting this has had a serious effect upon the supervision function. However, it is recommended that future internal implementation reviews examine cases where conflicting claims on staff have led to a change in supervision plans to assess their consequences on the adequacy of project supervision. These questions will be examined during the regular Project Implementation Reviews, although the evidence accumulated to date through such reviews reportedly does not support the feeling OED found to be widespread among staff that supervision is given second priority. (xvii) Project Preparation and Design Improvements in project design and the development of realistic implementation schedule would do much to make supervision more cost effective. One approach to innova- tive and complex projects is the use of pilot operations as an alternative to relying on the supervision process to reshape the design of projects in the course of implemen- tation. Several measures have been taken by the Bank in recent years to improve project preparation. Pilot projects are being undertaken on an increasing scale. - 28 - ANNEX A Page 10 III. OPERATIONAL POLICY REVIEW - DELAYS IN PROJECT IMPLEMENTATION (SecM80-280, April 14, 1980) The following is a summary of the principal recommendations of the above report and the management response to them. No formal response was made at the time of the Board discussion on July 15, 1980; however, the Vice President, Projects Staff earlier made an oral statement on the report to the Joint Audit Committee on July 2, 1980 (JAC/M80-5/1). The Vice President, Projects Staff indicated that, with the qualifications noted below, the Bank was in general agreement with the recommendations contained in the report. Summary of Principal Recommendations I. Scheduling (i) Implementation Planning Much greater realism in scheduling project implemen- tation is needed and the use of network analysis and the critical path method (CPM) are recommended as instruments for identifying activities in the implementation schedule, and highlighting critical actions. It is recommended that the use of network analysis and the CPM be extended at least to all multiple activity and more complex projects. Revisions to the critical path and the plan of operations should be made in consultation with the borrower in the event of major slippages from schedule. The Bank has always been conscious that unrealistic target setting is counterproductive and has discouraged it. A reasonable degree of targetting, based on judgment of local conditions, however, does provide an incentive for better performance. The Bank agrees that greater use should be made of network analysis and the CPM, although these are complex and not suitable for all borrowers and projects. They are most appropriate for large civil works and elaborate multi-component projects. - 29 - ANNEX A Page 11 (ii) Sensitivity Analysis Completion time be used and presented in the sensiti- vity analysis of project costs and benefits in all loans and credits made by the Bank. The Bank agrees to this recommendation and CPS will issue an instruction about it. II. Location and Causes of Delays (iii) Project Preparation and Design Greater borrower involvement in the preparation and design of projects, on which the Bank is now placing greater emphasis, should help to ensure closer relation- ship between project objectives and the sector environ- ment. The recent Bank instructions on the need for a closer understanding and greater congruence between project and sector objectives at the design stage were commended. The local capability of borrowers whenever a series of projects is envisaged should be reviewed and the need for assistance agreed upon to ensure the foregoing. The Bank strongly supports the recommendation calling for more borrower participation in the preparation and design of projects and is placing much emphasis on this area. The subject of project preparation and design was further discussed in the Bank response (R80-142, page 16) to the OED report on Supervision (SecM80-129). (iv) Project Start-Up and Execution (a) Intensive Bank supervision during the start-up period of projects should be given; (b) In selecting consultants and contractors for civil works, their previous experience in similar situations should be taken into account by the Bank and the borrowers; and (c) In regard to technical assistance, clear under- standing should be reached by the Bank with the borrower as to its purposes and approach during project preparation. -30 - ANNEX A Page 12 These recommendations are largely being implemented. However, including knowledge of and experience with similar local condition as a factor in the selection of consultants and contractors is a subjective judgment. To provide more guidance to the borrowers and staff about making such a judgment is thus a difficult task, but one the Bank proposes to undertake as soon as possible. (v) Country Factors The Regions should undertake special issues-oriented implementation reviews to identify reasons for the inci- dence of seriously delayed projects in seven countries mentioned in the report. Programs staff should be fully involved in the project implementation reviews in these countries and also in the design of projects. Some issues-oriented implementation reviews have taken place and more are contemplated. Programs staff are generally associated with such reviews. III. Adjustment and Training (vi) The fact that projects are changed during implementation led to the general suggestions that: (a) the Bank should occasionally accept more modest objectives in line with local conditions in the design of operations and see if a similar design in a small number of cases would not produce almost as good results but less strain on local conditions; and (b) full-scale review of project progress at the specified time or phase in its implementation should be provided for at the appraisal report. The Bank disagrees with the proposal for a full-scale review as a general rule as suggested in (b) above for reasons set out in the Bank response (R80-142) to the OED report on Supervision (SecM80-129). - 31 - ANNEX A Page 13 IV. FIFTH ANNUAL REVIEW OF PROJECT PERFORMANCE AUDIT RESULTS (R79-229, August 27, 1979) The number of projects covered by PPARs which formed the basis of this Review was 98, representing approximately US$1.6 billion in Bank loans and credits and some US$3.7 billion of total investments. These reports form part of OED's continuing review of the Bank's response to PPAR findings; the Reviews so far distributed have covered a total of 384 projects. Among the principal issues arising from the projects covered, this Review again noted the need for understanding local cultures as they affect organizational or economic behavior. In particular, emphasis was placed on two aspects of the sector environment within which projects function, namely, the institutional setting and the borrower's sector policy. In this connec- tion, the following points were made in the Review, with both of which the Bank agrees: (i) More effort by the Bank seems indicated to enlist full borrower understanding and support for the institutional components of projects. Systematic efforts at institutional improvement could be improved by the Bank and borrower jointly developing an institutional profile of the sector and a time- phased program of action. The institutional profile would review the technical manpower availability within the country and the sector and analyze the existing institutional framework in the sector. (ii) To the extent that the Bank makes series of loans in any sector, the policy environment becomes crucial to the attainment of longer term sector objectives. Sufficient sector analyses are thus essential to establish a basis for reaching prior understanding with the borrower on sector policies and keep them under frequent review thereafter. This applies to operations in the industrial and DFC sectors as it does in agriculture and education. The Review noted that several efforts are now being undertaken by the Bank to relate sectoral and - 32 - ANNEX A Page 14 project objectives more closely. An important development in recent years has been a substantial increase in the use of country sector memoranda. Together with the Project Brief System introduced in 1977, this may evolve into a suitable instrument for more effectively defining project objectives in a sector context. - 33 - ANNEX A Page 15 V. REPORT ON THE ROLE AND USE OF CONSULTANTS IN WORLD BANK PROJECTS (SecM77-865, December 9, 1977) A response to the findings of the OED report and to the recommenda- tions based thereon of an informal Subcommittee of the Joint Audit Committee (SecM78-244) on this report was made by the Vice President, Projects Staff, in a statement to the Executive Directors on May 16, 1978 (SecM78-464, May 25, 1978). The recommendations were largely accepted, and a program of determin- ing and implementing appropriate Bank policy has been underway since mid-1979. The principal recommendations are given below, together with a status report. A. Recommendations Where Action Has Been Taken (i) Clarification is needed of the Bank's assistance to borrowers in preparing short lists of consultants. An Operational Manual Circular detailing interim proce- dures for providing assistance in preparation of list of consultants was issued in October 1978; the procedures have worked smoothly. A computer-based information system (DACON) covering consulting firms will be operational late 1980. At that time, the procedures will be revised to utilize DACON, which will allow for a comprehensive scanning of capable firms and an equitable selection system. - 34 - ANNEX A Page 16 (ii) The evaluation of consultants' performance has not been regularly included in supervision reports, and steps should be taken to remedy this. A new confidential filing system holding information from both supervision reports and final evaluation reports is now in operation. To ensure that evaluation of consul- tants' performance is carried out systematically, an Opera- tional Manual Circular detailing evaluation procedures was issued in April 1980. (iii) The Bank needs to take steps to encourage local consultants and to increase their exposure to the international market. The Bank is expanding its knowledge of developing country consulting firms, and including suitably-qualified local firms on short lists being given to borrowers. Following a survey by the Bank of the local consulting industry in Indonesia at the end of 1978, the Bank has agreed to be the executing agency for a UNDP-financed project to promote and assist Indonesian consultants which will commence October 1980. Six other such surveys have now been completed covering Sudan, Kenya, Senegal, Sierra Leone, Upper Volta, and Cameroon. Four to six surveys are planned each year, with surveys scheduled for Bangladesh (July 1980) and Egypt (October 1980). Surveys have also been included in Bank-financed projects in Malaysia and Ecuador. - 35 - ANNEX A Page 17 (iv) The Bank should improve the definition of consultants' tasks, especially training. CPS expects to prepare a CPN covering these recommenda- tIons in 1981. (v) In seeking proposals from consultants, the "second-envelope" procedure (i.e., a price for the work) should become standard procedure. The Bank (CPSVP) has indicated that it will consider ways and means to enhance the relevance of price, and the way in which it is taken into account without having it override qualitative considerations. A position paper on the selection of consultants and covering the relevance of price has been prepared (June 1980) and will be discussed during a Board seminar on consulting services on Sept. 10, 1980. Revised guidelines on the selection of consultants will be prepared thereafter. B. Recommendation Where No Action Has Been Taken Alternatives to the Agricultural Projects Management Unit in Africa (APMU) should be explored. No action has been taken. - 36 - ANNEX A Page 18 C. Recommendations of a General Nature (i) Bank staff should be reminded that Bank funding of consul- tants' services includes local consultants. (ii) Borrowers should be encouraged to specify in consultants' contracts the work to be carried out in the borrowing country and that in the consultants' home country. (iii) Implementation schedules for consultants should be realistic and clearly specified and, where a training element is involved, adequate provision should be made for it in the schedule. (iv) Consultants' terms of reference should be unambiguous so that tasks are clearly understood and monitorable. (v) In suggesting weights to borrowers in evaluating consultant proposals, experience in the country concerned or in a similar environment should be distinguished from general experience. (vi) Weight should be given, in the evaluation process, to the qualifications of the individuals proposed for the work as well as the firm itself. (vii) Key consultant staff should attend contract negotiations. (viii) The entity which will use consultant services should be represented at loan negotiations. Most of the foregoing eight recommendations are covered by current policy and procedures. All the recommendations will be considered in the forthcoming revisions (early 1981) of Operational Manual Statements and Guidelines on consulting services. - 37 - ANNEX B Page 1 PHILIPPINES: POST-EVALUATION SYSTEM LETTER OF INSTRUCTION NO. 902 To: Minister of Economic Planning All Heads of Government Implementing Agencies, Including Government-Owned or Controlled Corporations or Authorities Subject: REQUIRING ALL GOVERNMENT IMPLEMENTING AGENCIES, INCLUDING GOVERNMENT-OWNED OR CONTROLLED CORPORATIONS AND AUTHORITIES, TO FULLY COOPERATE AND PARTICIPATE IN THE POST-EVALUATION OF MAJOR DEVELOPMENT PROJECTS WHEREAS, the country's development programs for the coming years call for substantial expansion of the public investment program; WHEREAS, the relative scarcity of investible resources stresses the need for continuing efforts to ensure that projects to be carried out are well-designed, appropriately reviewed and select- ed, and effectively executed; WHEREAS, the process of post-project evaluation consti- tutes one of the means by which improvements in project development can be effected; NOW, THEREFORE, I, FERDINAND E. MARCOS, President of the Philippines, by virtue of the powers vested in me by the Constitu- tion, do hereby order and authorize: The institutionalization of a post-evaluation system through which concerned government agencies, by conducting a review and assessment of project performance shortly after completion of implementation/construction and/or during the operational stage can draw lessons that may fruitfully be applied to current and/or future project preparation and implementation work. Under this system: A) All government implementing line agencies, including government-owned and controlled corporations and authorities, are required to prepare Project Comple- tion Reports (PCRs) in collaboration with the Nation- al Economic and Development Authority (NEDA), six months to one year after completion of implementation of major development projects. For this purpose, the NEDA shall determine those projects that are to be considered as "major" for each project category and advise the concerned agencies accordingly; - 38 - ANNEX B Page 2 B) Upon completion of the aforesaid PCRs, all government implementing line agencies, including government- owned or controlled corporations and authorities, shall transmit them to the NEDA for evaluation. Based on these PCRs and such other investigations that it may conduct independently, the NEDA shall prepare Project Performance Assessment Reports (PPARs) which shall be forwarded to the NEDA Board through the Director-General, for whatever action is deemed appropriate; and C) For major projects that have been in operation for at least 5 years, Mid-Operation Reports (MORs) shall be prepared by all government-owned or controlled corporations and authorities, in collaboration with the NEDA. The MOR for each major project shall be forwarded to the NEDA for evaluation, on the basis of which a Mid-Operation Assessment Report (MOAR) shall be prepared by the NEDA. The MOAR shall similarly be transmitted to the NEDA Board, through the Director- General, for whatever action is deemed appropriate. This Letter of Instruction shall take effect immediately. Done in the City of Manila, this 26th day of July, in the year of Our Lord, nineteen hundred and seventy-nine. /s/ FERDINAND E. MARCOS - 39 - ANNEX B Page 3 POST-EVALUATION OF PROJECTS IN THE PHILIPPINES This paper constitutes a preliminary proposal for the institutionalization of post-evaluation of projects in the Philip- pines. The proposal is premised on the acknowledgement of the value of post-project evaluation as an instrument for improving not only the project planning and execution capabilities of implementing agencies of Government but also the related evaluation and appraisal functions particularly of NEDA. Given this, it is argued that, considering the fairly sizeable volume of public investment in the past decade, together with the program for further expansion in the years to come, it is opportune at this time to begin to institution- alize a system for post-project evaluation in the Philippines. A. Definition and Objective of Post-Project Evaluation Post-project evaluation constitutes one of the last stages of the project cycle. It involves assessment of a project's performance shortly after completion of implementation/construc- tion and/or during the project-s operational stage. The activity essentially involves: 1. a determination of whether and the extent to which: (a) assumptions made during the pre-investment stage have been validated; and (b) projections and expectations with respect to project costs and benefits have been or show promise of being realized; and 2. an investigation into the factors that might explain such divergence as may have been ob- served in 1(a) and in 1(b) above. The whole point of the exercise is to draw lessons from experience that may fruitfully be applied to current and/or future project preparation and implementation activities of line agencies of government, as well as to NEDA's own project evaluation and appraisal functions. B. [Text omitted since it deals with evaluation by lending institu- tions which has been described in other documents.] - 40 - ANNEX B Page 4 C. The Need for a System of Post-Evaluation in the Philippines The past decade has witnessed a fairly rapid growth of public investment in the Philippines. Accordingly, quite a number of projects have now come on stream that are eligi- ble for post-project appraisal. It is herein submitted that the time is ripe for the Philippines to now begin to institutionalize a system for post-evaluation in the country. This derives from three factors: 1. The development programs for the coming years call for an even more substantial expansion of the public investment program if development aspirations are to be realized; 2. Investible resources will continue to be rela- tively scarce, underscoring the need to ensure that projects that are to be carried out are well-designed, appropriately reviewed and selected, and effectively executed. Lessons from post-evaluation would enhance this process; and 3. Post-evaluation activities are beginning to emerge within agencies. It would be desirable for these to be organized in some systematic fashion to avoid duplication and dispersal of individual efforts. D. The Proposed System 1. Reporting to the NEDA Board To begin with, it is proposed that, as in the case of the World Bank and the ADB, post- evaluation reports be ultimately submitted to the NEDA Board for information and appropriate action. This is because, as the highest policy- making body on economic and social development matters, it is the Board that can lay down whatever policies may be deemed appropriate in light of findings and recommendations emerging from post-project evaluation. Furthermore, - 41 - ANNEX B Page 5 considering its composition, presentation to the Board would facilitate the speedy transmission of evaluation results and recommendations to a large number of implementing agencies. 2. The NEDA as Overall Coordinator The NEDA is deemed to be the most appro- priate locus of coordination of the proposed post-evaluation system. This derives from two factors: (a) as the agency of government responsible for evaluating project proposals and recommend- ing their inclusion in both the National Budget and the borrowing programs from external lending institutions, it is incumbent upon NEDA to take such steps as may be necessary to constantly improve its project appraisal and funds programming procedures; only thereby can we approach resource optimization. The review of performance of approved projects would provide useful insights in this regard; and (b) as an agency that is, except for its monitoring function, removed from the implementation of projects, NEDA can take a more detached and a more objective view of project performance. 3. The System (a) Scope It would clearly not be feasible nor would it be economical, to subject all completed projects to post-evaluation. In the first place, many projects, taken individually, are so small that the time and the expense that would be devoted to post-evaluation would probably not be - 42 - ANNEX B Page 6 justified. Secondly, since post-evaluation involves a comparison of expectations with accomplishments, its conduct would be possible only for projects where such expectations are documented; in this context, this means projects for which feasibility studies have been undertaken. In light of the foregoing, post-evalua- tion is proposed to be confined to major programs and projects, somehow defined, for which feasibility studies were previously prepared. (b) Activities Essentially, two activities would be involved in the process: (i) preparation of project completion reports (PCR); and (ii) preparation of project performance audit reports (PPAR), where these are understood to have the same contents as in the case of IBRD. (c) Procedures (i) From six months to a year after completion of a project, a PCR is to be prepared by the implementing agency, in collaboration with NEDA. The PCRs are to be prepared on the basis of certain standard guidelines for each project category, which are to be issued by NEDA to implementing agencies. Such guidelines would ensure that all points that are relevant to post-evaluation are given adequate treatment. (In the case of IBRD-assisted pro- jects, the preparation of PCRs is a matter of standard procedure. In these cases, a separate PCR need not be prepared; it would suffice for NEDA to be copy-furnished.) - 43 - ANNEX B Page 7 (ii) Upon completion of the PCRs, these are transmitted to a unit of NEDA for review corresponding to the World Bank's OED, which would subject all PCRs to a limited review and some to a more thorough review. On the basis of these reviews, PPARs are drafted and circulated within NEDA and to the concerned implementing agencies. Upon receipts of comments, draft PPARs may or may not be revised and, in any case, finalized for transmittal to the NEDA Board through the Director- General. (iii) After consideration of the PPARs, the NEDA Board takes whatever action may be deemed appropriate. E. Mid-Operation Appraisal The foregoing discussion has focused on the evaluation of projects shortly after completion of implementation/construction. As such, at the time that the evaluation is undertaken, benefits expected from the project would, in general, not have begun to be generated. Accordingly, while on the cost of side, comparisons would be made between pre-construction estimates and those actually incurred, benefit comparisons would be confined to'pre-construction vis-a-vis post-construction estimates. It would, however, be of interest to find out at some point during the project-s lifetime whether and to what extent expected benefits are in fact being realized. In addition, one might also look into actual operating and maintenance costs relative to projections. Such an evaluation would provide a more substantive basis for arriving at conclusions on whether in fact the project should have been carried out at all. More importantly, valuable lessons might be drawn from the operating experience of evaluated projects; these information can subsequently be fed into not only the preparation, evaluation and implementation of other projects but also their operation. Thereby, improvements in the entire process can be effected. - 44 - ANNEX B Page 8 In light of the foregoing, it is herein further proposed that, for projects that have already been in operation for some time (say, a minimum of five years), a Mid-Operation Report (MOR) be prepared,1/ again by the implementing agencies in collaboration with NEDA. These are similarly transmitted to a unit of NEDA for review and preparation of Mid-Operation Audit Reports (MOAR). , for eventual transmittal to the NEDA Board through the Director- General. Clearly, the objectives, the substance and the system of mid-operation evaluation are essentially the same as those of post-construction/ implementation. The principal difference lies in the availability of additional factual information since benefits are already being generated and operating and maintenance costs are already being incurred. I/ For projects that have already been in operation for sometime but for which PCRs and PPARs have not been prepared, the MOR and the MOAR should include an evaluation of project construc- tion. - 45 - ANNEX B Attachment I Criteria for Determining Projects For Post-Evaluation 1. Availability and comprehensiveness of the feasibility study. The feasibility study should be in a form suitable for post- evaluation, i.e., it should include an in-depth presentation and evaluation of the economic, financial, technical, and operational aspects. In the case of credit relending projects, the availability of the loan agreement. 2. Completion of the project. The project's construction/imple- mentation should have been completed. 3. Cost outlay. For each project category, a cut-off investment figure was sent to determine which projects can be considered as "major" development projects (see Attachment II). - 46 - ANNEX B Attachment II Minimum Investment Cost for Projects to be Considered as Major Development Projects Project Category Minimum Cost (PM) Air navigations and Airports 9 Portworks 26 Waterworks 20 Flood Control 10 School Building 8 National Building, Hospital, Rural Health Units 2 Irrigation 38 Highways and Bridges 19 Power, Generation, Transmission, Electrification 125 Railways 13 Credit Relending 148 - 47- ANNEX C Page 1 OED REPORTS ISSUED DURING FY80 Project Performance Audit Reports Public Utilities Kenya First Nairobi Water Supply SecM79-556 7/18/79 (Loan 714-KE) Colombia Cali Water Supply and SecM79-632 8/22/79 Sewerage (Loan 682-CO) Malaysia Fifth Power (Loan 700-MA) SecM79-644 8/30/79 Papua New Guinea Upper Ramu Hydro- SecM79-721 10/05/79 electric (Loan 737-PNG) China Third Power (Loan 749-CHA) SecM79-723 10/09/79 Brazil Power Distribution and Subtran- SecM79-756 10/23/79 mission (Loan 887-BR) Brazil Salto Osorio Hydroelectric SecM79-757 10/23/79 (Loan 728-BR) Iran Second Telecommunications SecM79-775 10/26/79 (Loan 927-IRN) Colombia Power Interconnection SecM79-780 10/31/79 (Loan 575-CO) and Chivor Hydro- electric (Loan 681-CO) Bolivia ENDE Third Power SecM79-798 11/07/79 (Credit 433-BO) Indonesia First and Second Electricity SecM79-825 11/16/79 Distribution (Credits 165/334-IND) Iraq Telecommunications (Loan 788-IRQ) SecM79-870 12/07/79 Ireland Second and Third Power SecM79-887 12/17/79 (Loans 726/804-IRE) Tanzania Kidatu Hydroelectric (First SecM79-894 12/19/79 Stage (Loans 715/715-2-TA) Brazil Marimbondo Power (Loan 677-BR) SecM79-905 12/28/79 India Fifth TelecoTmnunications SecM79-908 12/28/79 (Credit 403-IN) Ethiopia Addis Ababa Water Supply and SecM80-147 3/03/80 Sewerage (Loan 818-ET) Guatemala First Telecommunications SecM80-200 3/21/80 (Loan 792-CU) Fiji First Teleconnanications SecM80-213 3/27/80 (Loan 833-FIJ) El Salvador Second Telecormnunications SecM80-264 4/08/80 (Loan 811-ES) Brazil Sao Paulo Water Supply and SecM80-265 4/08/80 Pollution Control (Loans 757/758-BR) - 48- ANNEX C Page 2 Public Utilities - continued Costa Rica Fourth Power (Loan 800-CR) SecM80-368 5/09/80 Ecuador Third Power (Credit 286-EC) SecM80-429 5/30/80 India Power Transmission II SecM80-443 6/03/80 (Credit 242-IN) El Salvador Sixth Power (Loan 889-ES) SecM80-526 6/30/80 Transportation Thailand Fourth Railway (Loan 898-TH) SecM79-576 7/27/79 Niger Niamey International Airport SecM79-722 10/09/79 (Credit 473-NIR) Nicaragua Second Corinto Port SecM79-806 11/09/79 (Loans 879/879-1-NI) Yemen Arab Republic First and SecM79-809 11/12/79 Second Highways (Credits 315/558-YAR) Malaysia First Railway (Loan 799-MA) SecM79-851 11/30/79 Venezuela Maiquetia International SecM79-872 12/07/79 Airport (Loan 796-VE) Tunisia First Highway (Loan 746-TUN) SecM79-899 12/26/79 Malaysia First Sabah Ports (Loan 774-MA)SecM80-7 1/04/80 Algeria First Highway (Loan 912-AL) SecM80-174 3/13/80 Haiti Third and Fourth Highways SecM80-428 5/30/80 (Credits 478/556-HA) Papua New Guinea First Port SecM80-465 5/21/80 (Credit 326-PNG) Korea Third Railroad (Loan 669/ SecM80-500 6/19/80 Credit 183-KO) Korea Technical Assistance - Transport, SecM80-517 6/26/80 First Highway and Second Highway (Credit S-4, Loans 769/956-Ko) Somalia Mogadishu Port and Port Exten- SecM80-525 6/30/80 sion (Credits 359/586-SO) Cameroon First and Second Railways SecM80-528 6/30/80 (Loans 1038/S-4-CM) DFCs and Industries Colombia First Small-Scale Industry SecM79-658 9/06/79 (Loan 1071-CO) Bangladesh First Small-Scale Industry SecM80-388 5/16/80 (Credit 353-BD) - 49 -ANNEX C Page 3 DFCs and Industries - continued Brazil CSN Steel Expansion (Stage II) SecM80-404 5/21/80 (Loan 797-BR) Turkey IGSAS Fertilizer SecM80-518 6/26/80 (Loans 845/845-1-TU) Agriculture Mali Drought Relief Fund (Credit 443-MLI) SecM79-610 8/12/79 Upper Volta Drought Relief Fund SecM79-635 8/24/79 (Credit 442-UV) Lesotho Thaba Bosiu Rural Dev. SecM79-642 8/28/79 (Credit 361-LSO) Upper Volta Rural Dev. Fund SecM79-650 9/04/79 (Credit 317-UV) India Andhra Pradesh, Tamil Nadu and SecM79-733 10/15/79 Maharashtra Agricultural Credits (Credits 226, 250 and 293-IN) India Punjab and Haryana Agricultural SecM79-735 10/17/79 Credits (Credits 203/249-IN) India Agricultural Refinance and SecM79-778 10/29/79 Development Corporation First Credit (ARDC I) (Credit 540-IN) Turkey Seyhan Irrigation (Stage II) SecM79-848 11/30/79 (Loan 587/Credit 143-TU) Israel Second Agricultural Credit SecM79-853 11/30/79 (Loan 972-IS) Senegal River Polders (Credit 350-SE) SecM79-910 12/28/79 Trinidad & Tobago Caroni Sugar SecM79-914 12/26/79 (Loan 888-TR) Iraq Grain Storage (Loan 925-IRQ) Seci179-915 12/28/79 Philippines Third Rural Credit (Loan SecM80-24 1/09/80 (Loan 1010-PH) Indonesia Seeds (Credit 246-IND) SecM80-5 1/04/80 (Credit 246-IND) Guatemala Livestock Development SecM80-6 1/04/80 (Loan 722-GU) Tanzania Smallholder Tea Development SecM80-9 1/04/80 (Credit 287-TA) Ecuador Third Livestock Development SecM80-10 1/04/80 (Credit 222-FC) - 50- ANNEX C Page 4 Agriculture - continued Tanzania Flue-Cured Tobacco SecM80-11 1/04/80 (Credit 217-TA) Sri Lanka Drainage and Land SecM80-12 1/04/80 Reclamation (Credit 168-CE) Upper Volta Cotton (Credit 225-UV) SecM80-13 1/04/80 Indonesia First Fisheries SecM80-27 1/10/80 (Credit 211-IND) Mauritania Drought Relief Fund SecM80-28 1/10/80 (Credit 444-MAU) Chad Drought Relief Fund SecM80-29 1/10/80 (Credit 445-CD) India Madhya Pradesh Agricultural SecM80-41 1/15/80 Credit (Credit 391-IN) Sri Lanka Lift Irrigation SecM80-57 1/21/80 (Credit 121-CE) Ghana Fisheries (Credit 163-GH) SecM80-64 1/23/80 Bangladesh Foodgrain Storage SecM80-124 2/20/80 (Credit 381-BD) Thailand Sirikit Dam and Chao Phya SecM80-164 3/10/80 (First) Irrigation Improvement (Loan 514-TH and Credit 379-TH) India Karnataka Agricultural Credit SecM80-287 4/15/80 (Credit 278-IN) Ethiopia Agricultural Minimum SecM80-427 5/30/80 Package I (Credit 416) Liberia Agricultural Development SecM80-455 6/14/80 and Technical Assistance (Credit 306-LBR) Malaysia Second Jengka Triangle Land SecM80-495 6/18/80 Settlement (Loan 672-MA) Ethiopia Amibara Irrigation Sec480-497 6/18/80 (Credit 413/ET) Korea Yong San Gang Irrigation SecM80-51n 6/26/80 (Stage 1) (Loan 795/Credit 283-K0) Philippines Upper Pampanga Irrigation SecM80-529 6/30/60 (Loan 637-PH) Ivory Coast First, Second and Third SecM80-530 6/30/80 Oil Palm (Loans 613/760/1036-IVC) Iran Dez Irrigation SecM80-531 6/30/80 (Loan 795/Credit 283-IRN) - 51 - ANNEX C Page 5 Education Tanzania Third Education SecM79-586 8/01/79 (Credit 232-TA) Chad First and Second Education SecM79-590 8/02/79 (Credit 126/251-CD) Singapore First Education SecM79-906 12/28/79 (Loan 802-SI) Brazil First Education (Loan 755-BR) SecM80-23 1/09/80 Guatemala First Education SecM80-83 2/01/80 (Loan 576-GU) Malawi First Education SecM80-260 4/07/80 (Credit 102-MAI) Ireland First Education SecM80-315 4/23/80 (Loan 785-IRE) Ivory Coast First Education SecM80-375 5/12/80 (Loan 667-IVC) Indonesia First Education SecM80-440 6/02/80 (Credit 219-IND) Lesotho First Education SecM80-444 6/03/80 (Credit 497-LSO) Indonesia Second Education SecM80-527 6/30/80 (Credit 288-IND) Special Studies and Other Reports Annual Report on Operations Evaluation R79-225 8/30/79 Fifth Annual Review of Project Performance R79-229 8/27/79 Audit Results Built-In Project Monitoring and Evaluation: SecM79-788 11/05/79 A Second Review Operational Policy Review - The Supervision SecM80-129 2/22/80 of Bank Projects Operational Policy Review - Delays in SecM80-280 4/14/80 Project Implementation Kenya - Impact Evaluation Report SecM80-370 5/12/80 First Smallholder Agricultural Credit Sudan - Impact Evaluation Report SecM80-539 6/30/80 Roseires Irrigation (Loan 284/Credit 2-SU) Colombia - Impact Evaluation Report SecM80-540 6/30/80 First and Second Education (Loans 552/679-CO)