Document of The World Bank FOR OFFICIAL USE ONLY Report No.: 22467 PERFORMANCE AUDIT REPORT BULGARIA TECHNICAL ASSISTANCE PROJECT FOR ECONOMIC REFORM (LOAN 3384-BU) June 27, 2001 Sector and Thematic Evaluation Group Operations Evaluation Department This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Currency Equivalents (annual averages) Currency Unit: Bulgarian Leva (Period average) Exchange rate as of March 7, 2001 US$ 1 2.10146 Leva 1 Leva US$ 0.47586 US$ Abbreviations and Acronyms BPC Bulgarian Post Company BTC Bulgarian Telecommunications Company CPT Committee for Posts and Telecommunications EBRD European Bank for Reconstruction and Development EIB European Investment Bank ICR Implementation Completion Report OED Operations Evaluation Department SAR Staff Appraisal Report TAL Technical Assistance Loan Fiscal Year Government of Bulgaria: January 1 to December 31 Director-General, Operations Evaluation Mr. Robert Picciotto Director, Operations Evaluation Department : Mr. Gregory K. Ingram Manager, Sector and Thematic Evaluation : Mr. Alain Barbu Task Manager : Mr. Nagy K. Hanna FOR OFFICIAL USE ONLY The World Bank Washington, D.C. 20433 U.S.A. Office of the Director-General Operations Evaluation June 27, 2001 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Performance Audit Report on Bulgaria Technical Assistance Project for Economic Reform (Loan 3384-BU) This Performance Audit Report (PAR) covers the Technical Assistance Project for Economic Reform Loan (TAL) for US$17 million, approved on June 27, 1991, and made effective on January 9, 1992. The original closing date of June 30, 1994, was extended five times; the project finally closed on June 30, 1999,with 72 percent disbursed (USS 4.9 million cancelled). The objective of the TAL was to strengthen government institutions and agencies central to reform by creating an effective and efficient enabling environment in support of private sector development, and to finance the needed technical help to the extent that financing cannot be obtained from other sources on concessional terms. The TAL covered priority areas of the transition: the privatization and restructuring of enterprises; reforms in the banking and energy sectors; and human resources and institutional development. OED rates the project's overall outcome satisfactory, its sustainability likely (as in the ICR), and its institutional development substantial. The project's objectives were comprehensive and demanding, but they were highly relevant to the overall economic reform agenda and the Bank's development assistance strategy. Judged by results, the TAL had a wide-ranging impact and facilitated implementation or prepared the ground for Bank assistance in many areas. But it did so at high cost, very unevenly, and with significant shortcomings in design and implementation. The audit rates the Bank performance satisfactory (as in the ICR), but with significant qualifications. Identification to appraisal was deficient because the Bank adopted a highly detailed pre-packaged design, with specified amounts to many activities, and a tight timetable. Partly because of its complexity, including 18 implementing agencies, and many extensions, the TAL was expensive to administer and the design and approach undermined ownership. The Bank did demonstrate flexibility during implementation, but without fundamentally re-thinking the design. Bank performance during implementation suffered from common problems of multisectoral free-standing TALs-instruments and objectives that call for sustained efforts, appropriate incentives, effective coordination, and multisectoral skills. Supervision was overstretched, tended to focus on issues of procurement and disbursement, and other administration, with limited dialogue on assistance strategies and quality. As measured by its commitment to use effectively the assistance resources, GOB performance is rated satisfactory (as in the ICR). A key factor in borrower performance is the continuity of staff in the central PIU and the long-term engagement of a seasoned advisor. Borrower performance might have been better with respect to maintaining continuity of personnelin the implementing agencies and being less resistant to using foreign TA. Frequent changes in government also undermined ownership. This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Experience with this project confirms a number of OED lessons: * Results-based management is critical to the development effectiveness of technical assistance. Despite achievements, a commonly held misperception among Bank staff is that this TAL was ineffective; perhaps reflecting the difficulty of measuring TAL results (building capacity; facilitating reform) and excessive preoccupation with measurable inputs (funds disbursed) and process issues (extensions and amendments) which resulted from over optimistic schedules at entry. * A blueprint approach to TAL is inappropriate in uncertain and fast-changing environments. It contributed to complexity, multiple amendments and extensions, higher transaction costs, lower ownership, cumbersome supervision, and perhaps drift from the fundamental objectives. A more promising approach would have focused the TAL on a few institutional and policy objectives, using these to define activities through regular reviews during implementation. * The Bank should create adequate incentives and provide the resources needed for quality implementation assistance in TALs. TA is inherently difficult, risky and experimental in the best of times. A bias for resource transfer, by the Bank and the MOF, may have devalued the developmental role of TA for capacity building and implementation of reforms. * More emphasis on timely monitoring and evaluation of the privatization process is necessary to continuously adapt and feed into ongoing privatization programs. Attachment Contents Principal Ratings ........................................................................................................................................ III Key Staff Responsible ................................................................................................................................ III Preface ........................................................................................................................................................... V 1. BACKGROUND...................................................................................................................................1 2. FINDINGS AND LESSONS.......................................................................................................... 1 3. OUTCOMES .........................................................................................................................................3 Private Sector Development, Privatization, and Restructuring.............. .............4 Bank Restructuring and Reform............................. ..... ..................5 Energy Sector ...........................................................5 Human Resources Development ...............................................5 Institutional Development ................................................... 6 4 . RATINGS .............................................................................................................................................. 7 Assessment of Outcome .................................................... 7 Sustainability ............................................................7 Institutional Development Impact ..................... ....... ......................... 7 5. BANK AND BORROWER PERFORMANCE .................................................................................8 Bank Performance ........................................................8 Borrower Performance ..................................................... 9 6. FINDINGS AND LESSONS..............................................................................................................10 7. FUTURE DIRECTIONS ................................................................................................................... 12 Annex A. Basic Data.................................................................................................................................... 15 Annex B. Comments from the Borrower ............................................................................................. 17 This report was prepared by Nagy K. Hanna, who audited the project in January 2001. William Hurlbut edited the report, and Soon-Won Pak provided administrative support.  Ill Principal Ratings ICR PAR Outcome Satisfactory Satisfactory Sustainability Likely Likely Institutional Development Substantial Substantial Borrower Performance Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Key Staff Responsible Task Manager Sector Chief Department Director Appraisal Tom Hoopengardner M. Wiehen Completion Kyle Peters Pradeep Mitra Andrew Vorkink  V Preface This Performance Audit Report (PAR) covers the Technical Assistance Project for Economic Reform Loan (TAL) for US$17 million, approved on June 27, 1991, and made effective on January 9, 1992. The original closing date of June 30, 1994, was extended five times; the project finally closed on June 30, 1999, with 72 percent disbursed (US$ 4.9 million cancelled). The TAL was one of the first two operations by the World Bank in Bulgaria and was designed to support the $250 million Structural Adjustment Loan (SAL) that was made effective on September 27, 1991. An Implementation Completion Report (ICR) for the TAL was submitted on June 26, 2000. The PAR was prepared by the Operations Evaluation Department (OED) based on the ICR, the Staff Appraisal Report, a review of Bank files, and extensive discussions with Bulgarian policymakers, implementing agencies, NGOs, and private sector entities, as well as Bank staff, consultants, and donors. The cooperation and assistance of all stakeholders and government officials is gratefully acknowledged, as is the support of the staff of the World Bank Country Office in Bulgaria. This PAR complements and builds upon a good ICR. It focuses on issues, lessons, and future directions, since the project presents rich lessons of relevance to country assistance strategy and future Bank involvement in policy reforms, knowledge transfer, and capacity building. The audit does not cover the added component concerning support to the privatization of the Bulgarian Telecommunications Company, since this activity was treated in-depth and formed an integral part of the audit of the Bulgarian Telecommunication Project. Following standard OED procedures, the draft PAR was sent to the borrower for comments before being finalized. Borrower comments have been received and are included in Annex B.  1 1. Background 1.1 When Bulgaria was separated from the Soviet system it was left with high external debt and limited institutional capacity. Severe external debt threatened the sustainability of the reform program that the Government of Bulgaria (GOB) had begun. By the end of 1990, total debt was 50 percent of GDP. Bulgaria's most immediate needs were developing managerial, institutional and technological capabilities to make the transition to a market economy. The priority areas of the transition were in the privatization and restructuring of enterprises, and in the financial and energy sectors. Social safety nets were included to ease the pain of reform. 1.2 The objective of the TAL was "to strengthen the government institutions/agencies, which have a central role to play in the reform effort, by creating an effective and efficient enabling environment in support of private sector development, and to finance the needed technical help to the extent that financing cannot be obtained from other sources on concessional terms." 1.3 The TAL was designed at a time when GOB needed support to carry out reforms in multiple sectors and activities. The TAL strategy was that it was better to be comprehensive and try to meet as many needs as possible to minimize the risk that the reform program would be thwarted by an uncovered need. It was also a time when donors had little coordination and their comparative advantages or division of labor had not been established. The TAL was a complex project, which attempted to provide assistance to as many institutions in as many areas as possible. The resulting design proved demanding to implement, and by the time the project was completed it had strayed from the original plan and had canceled 28 percent of the original loan despite five extensions. Yet, its coverage had by then increased to 18 agencies. 1.4 As in other transition countries, the government needed substantial infusions of expertise in developing the policy and institutional mechanisms to carry out an ambitious reform program. The TAL would provide privatization expertise through a wide variety of legal, managerial, accountancy, and other consultancies. The legal system had begun to be revamped in line with a market economy. In the energy sector, the government needed support for reform through energy studies to find ways to reduce the energy intensity of Bulgarian industry. 1.5 Bulgaria's welfare system also needed a complete overhaul because of the transformation from a centrally planned to a market economy. Social security and health systems needed substantial legal groundwork and managerial training in order to operate under a new economic system. The educational system needed comprehensive reforms in school curricula and funding. GOB also required a range of other institutional assistance to support the economic reform program, particularly in trade, debt management, statistics, and economic analysis. 2. Findings and Lessons 2.1 The GOB recognized that it lacked the broad range of experience and institutional capacity necessary to design and implement structural reforms and manage a private sector economy. The TAL was designed to help build these capacities in five major areas: 2 Original allocation (millions) Private Sector Development, Privatization and Enterprise Restructuring $12.6 Bank Restructuring and Reform $4.5 Human Resource Development $6.1 Energy Sector $1.4 Institutional Development $4.5 2.2 Estimated project costs were $33.5 million of which the IBRD was to finance $17 million, the UK Know How Fund $1.3 million, the European Commission (EC-PHARE) $13.2 million, and the GOB $2.0 million. The IBRD loan was to finance only foreign costs and included $9.4 million for consultancies and studies, $1.8 million for equipment, $1.2 million for training, and an unallocated $4.6 million. The cofinancing of TAL was soon out of date because EC-PHARE opted to fund large amounts of technical assistance through grants rather than cofinance the TAL. 2.3 At loan approval in June 1991, the Bank and the GOB envisioned that this broad and detailed program could be carried out within two and a half years. Effectiveness was delayed until January 1992, leaving the project barely two years for implementation, which was unrealistic. In addition to many extensions, the Loan Agreement required seven amendments, often to add another implementing agency, as documented in the ICR. There was no midterm review since the TAL was extended each time by only a single year, which meant that the project's remaining life was too short to warrant a review. 2.4 TAL had mixed ratings throughout implementation. With one exception, implementation of individual components was rated satisfactory, but development objectives, unsatisfactory. The TAL financed only one of the subcomponents of the Bank Restructuring component, although the GOB found more favorable funding for the other activities elsewhere. The Enterprise Restructuring Component did not accomplish much until late in the TAL. The ratings are paradoxical, perhaps because supervision missions underestimated or could not foresee the development impact of the TA, and paid too much attention to processes, inputs, and rates of disbursements, at the expense of results. The changes in the ratings towards the end of the operation may reflect the positive changes in government as well as the growing appreciation by the Bank of the contribution of this TAL to policy and institutional reforms. 2.5 Frequent changes in government created major difficulties for the project. Other Bank projects suffered equally, especially in 1995 and 1996 when the administration in power was less committed to many of the free market measures (supported by the TAL). During this period of political changes , TAL was in a wait-and-see mode, rather than an operational mode. 2.6 The GOB was opportunistic in using grant funds to carry out a number of activities previously identified under the TAL. It meant that TAL did not neatly implement what it was originally designed to, but that should have been expected and taken into account through a flexible project. For example, under the legal component, the MOF was reluctant to approve the legal consultants' contract because of the high cost. The scope of the Bank-financed contract was ultimately reduced; other sources of assistance financed the rest. But the re-focused legal component was very successful, in terms of development impact. 2.7 The Ministry of Finance established a central PIU that was very effective and gave good support to the 18 implementing agencies and their PIUs. But the MOF did not always give priority to this project given the small sums of resources involved, the high transaction costs to the MOF, and the Ministry's underestimation of the potential payoffs of good technical assistance 3 and advisory services. However, there was no obvious alternative site for the PIU which would have offered superior high-level support for country-wide technical assistance. An early proposal for establishing an inter-ministerial steering committee had to be abandoned, as no ministry or body could take the lead for integrated and substantive management of the TAL. In the end, the central PIU remained small, and was influential primarily in enforcing Bank requirements for procurement and financial management and in training others to follow these requirements. The substantive management and ownership of TA was left to the implementing agencies. To a great extent, that mirrored the task management of the TAL in the Bank, with the task manager, located in the country unit, focused primarily on issues of TA administration, while sectoral staff were called upon as needed for substantive engagement. 2.8 The TAL got off to a slow start with modest implementation during its early years, except for some low-cost, but high-impact activities. These were difficult times for all projects, but high payoffs were possible where implementing agencies did value the then scarce resources made available. Implementation accelerated in 1997 when a new government was elected with strong motivation to solve the economic and financial crisis. 2.9 The individual agency PIUs had a mixed performance, mainly reflecting the very different levels of understanding, experience, skills, attitudes, and commitments regarding the individual TA activities. Some agencies were reluctant to identify or carry out TA using TAL loan funds, with the hope that grants would be made available from other sources. The Ministry of Industry and Ministry of Trade were reluctant to use TA funds, perhaps because of previous bad experiences with grant-funded TAs. Some agencies did not adequately train administrative officials to draft terms of reference and letters of intent, evaluate proposals, and negotiate and monitor contracts. This shortage of skills was aggravated by high personnel turnover. This led to delays for these components. On the other hand, the Ministry of Energy had an effective PIU and implemented its TAL activity very well. The Bulgarian National Bank (BNB) project management performed well, although under a very scaled-down component. The CPT successfully oversaw implementation of its main study, "Policy and Regulations in the Domain of CPT." The Ministry of Labor and Social Welfare had high personnel turnover, but carried out two major studies. All other agencies performed reasonably well. No projects that were actually implemented were unsatisfactory. 3. Outcomes 3.1 In reviewing both achievement of objectives and implementation record, it is important to appreciate the broader context. First, the TAL was the first operation in Bulgaria, and perhaps the earliest in transitioh economies. Second, the initiatives under the TAL were launched in an environment that attempted a major transition from a centrally planned to an open, free market system--a time of urgency, uncertainty, and fast change. Third, Bulgaria had just begun to open to the world, but many were not convinced of the need to access global know-how, let alone to pay for it. Fourth, for the Bank, it was a time of limited understanding of transition economies and of accompanying institutional transformations. Optimism and lack of realism were widely shared. Finally, the Bank and the Bulgarians had little experience and understanding of each other, yet the Bank (and its Board) wanted the usual specificity in design and detailed control in implementation. 3.2 The TAL financed many components through many implementing agencies. It supported a vast range of objectives and activities that had significant development impacts. A detailed 4 checklist of objectives by component and their record of achievement are found in the ICR. Below is a selective coverage of the main components. Private Sector Development, Privatization, and Restructuring 3.3 Privatization of Larger Companies. The TAL supported the Bulgarian privatization program through funding and non-funding means. The TAL helped arrange, but did not fund, the Monitoring Unit under the Ministry of Finance (MOF). MOF monitored the larger companies, and tracked their profitability losses and general performance. The TAL provided substantial funding for consultancies to advise troubled companies that were "privatizable." Both the Monitoring Unit and the individual consultancies were valuable in the preparation of higher value companies for privatization. A consultancy to assist the GOB in the privatization of Bulgaria Telecom Company (BTC) was the final (additional) activity of TAL. Although the sale did not go through, the experience gained from this first large-infrastructure privatization experience was substantial, and the lessons learned from this component are captured under the PAR for the Telecommunications Project. 3.4 Legal. The legal component was very successful, and was valued highly by the client. It contributed significantly to legal reform, particularly in its support of drafting new legislation. Specifically, TAL funded a resident legal advisor (RLA) and other consultants who (a) reviewed and commented on the Land Law; (b) contributed to drafting of the Bankruptcy Code and the Composition Law; (c) contributed to the Law on Obligations and Contracts; (d) made outlines of laws to establish notaries; (e) provided a draft of the Public Procurement Law; (f) assisted in the preparation of Securities and Stock Exchange Law; and (g) commented on Privatization Laws. 3.5 TAL-financed consultancies were instrumental in drafting the Securities, Stock Exchanges, and Investment Companies Act. The consultant prepared the drafts of the legislation and trained local counterparts. The performance was rated highly by the client (GOB). TAL financed computer equipment, which not only supported the consultancy but also gave general support to the Legal Department of the Council of Ministers. 3.6 The legal inputs were valuable in all aspects, but the impact was greatest with respect to securities, procurement, notaries, and privatization, as shown by the rapid development of these functions. The combination of a long-term advisor to ensure continuity, recruitment of highly specialized legal expertise familiar with EU practices, assistance by an internationally-known law school with global access to expertise, and setting a consultative process through working group of stakeholders, proved very effective in initiating many new and good quality laws. During this process, know-how was transferred and further amendments were made, although some of the lawyers involved ended up joining private practice. 3.7 The challenge now is for implementation and practice. With practice, and the evolution of the market and new institutions, further improvements and adaptations of the legal frameworks are likely. Moreover, the guiding implementing procedures and the capabilities ofjudges and lawyers remain far behind. Public institutions are not yet capable of enforcing market-supporting laws, and much needs to be done to develop and retain staff who understand business needs and respect the importance of transparent public processes. This will also likely be a key to accession to the EU. 5 Bank Restructuring and Reform 3.8 The study on "Structural Issues of the Banking System" was financed by the TAL, while other activities envisioned under the component were financed by other donors. The study contributed to achieving a quantitative target of bank consolidations and other technical guidance, but did not succeed in the goal of guiding the restructuring process and laying the foundation for a well-functioning banking system. Energy Sector 3.9 The energy component was reduced as GOB located grants to finance the power sector studies on pricing, rehabilitation, and organization, and the expanded use of gas. However, TAL financed the nuclear safety study, and the GOB implemented a number of the safety recommendations. Human Resources Development 3.10 Education: TAL-financed work in the education sector cost a modest amount ($350,000), but had a major impact on the financing and quality enhancement of education. The project on "Establishment of National Standards for Secondary Education" was especially effective. Almost all recommendations from TAL consultancies were implemented. The Ministry of Education (MOE) expressed a high level of satisfaction with the expertise financed under TAL. In short, TAL helped the Bulgarian school system modernize in key areas and better allocate resources available to education. A total of seven subprojects were financed under TAL: * Alternative Sources for Financing Education-introduced legislative reforms for more flexible financing between budget line items, school fees, new revenue sources for vocational schools, among others * Establishment of Revolving Fund for Textbooks-helped develop private textbook publishers as well as supply textbooks to students * Improvement of the Material and Technical Basis in Education-drafted plans to upgrade school buildings * Identifying Alternative Sources of financing for Scientific Research-provided the basis of the National Research Program * Foreign Language Training and Upgrading in Bulgarian Schools * System of Inspection and Teachers Evaluation * Establishment of National Standards for Secondary Education-desperately needed to improve the pedagogical aspects, in particular the practical application of the basic disciplines. 3.11 Health: By contrast, there was very little activity in the health sector under TAL, despite the long list of needs originally solicited by the Bank from GOB. The main activities were three consultancies conducted for the Ministry of Health (MOH) for Health Sector Performance and Management, Food Safety and Nutrition, the development of the primary health component and the selection of 25 municipalities to provide a package of essential primary health services. 6 Institutional Development 3.12 Small amounts of TAL funds were allocated to build the implementation capacity of a number of government agencies. 3.13 Committee of Posts and Telecommunications (CPT). A consultancy reviewed the role of the CPT in the telecommunications sector in Bulgaria, and recommended changes in laws and regulations and proposed restructuring CPT. A financial and economic model was introduced to improve tariff calculations and assess the effects of policy changes. The consultants worked closely with Bulgarian counterparts, and workshops and training sessions enhanced local ownership of the proposals. 3.14 Ministry of Labor and Social Security. Only $400,000, this TA component was influential in preparing the policy setting, institutional framework, and laws necessary for the Bank and other donors to finance much larger follow up projects: Social Fund, and Child Welfare. The ideas for such projects grew out of this component. The GOB used TAL effectively to pre-finance start-up activities for the National Social Security Institute project, which, because of its jump-start, was a highly satisfactory performer. Assistance was timely and appreciated by the ministry; it helped build institutional relationships between the Bank and a ministry with wide-ranging social responsibilities. 3.15 Other Agencies. Several other agencies received small amounts, which nevertheless had an important impact on their operations. The National Statistical Institute (NSI) received computer equipment that improved data processing as well as statistical publishing. The Agency for Economic Analysis and Forecasting also received computer equipment to carry out its work of economic forecasting and developing national programs. Consultancies and hardware to the Ministry of Regional Development and Public Works (MRDPW) helped support the training of staff, interagency coordination (with the Council of Regional Policy, Trans-Border Infrastructure Systems, and local authorities), and administrative-territorial reform, although some of the consultants' contributions did not conform to the Bulgarian context. For the Foreign Investment Agency, TAL financed equipment, training tours, and the production of the first investment guide for Bulgaria. The guidebook helped foreign investors understand the business environment in Bulgaria. The Ministry of Industry (MOI) used only $200,000 allocation under TAL as substantial grant funding form other sources (from EC-PHARE) were available. 3.16 In summary, the objectives of the TAL were largely accomplished by the extended closing date of June 30, 1999. Some of the activities originally envisioned under TAL, although not carried out under it, were ultimately financed, mainly with grants from other sources, especially EC-PHARE. The availability of TAL funds also made it possible to finance several unexpected activities as opportunities arose, including the advisory services for BTC's attempted privatization and the provision of hardware to the NSSI. The performance of the TAL should be viewed as a part of and catalyst to a continuous effort ranging from supporting the SAL, to longer-term development of institutional capacity, to the various follow-up loans. 7 4. Ratings Assessment of Outcome 4.1 This audit rates project outcome satisfactory, since the TAL was highly relevant and did achieve major development results, even with significant shortcomings in design and implementation. The overall performance and outcome of TAL needs to be assessed in the context of its achievements and Bulgaria's uncertain environment at the time (para. 3.1). Contrary to original expectations, the TAL disbursed slowly, a substantial share of loan resources was cancelled, and the project required modification and restructuring to reflect revised government priorities over time. In many areas, however, the TAL had a strong positive impact well beyond the value of the loan resources, and funded activities for which there was no alternative source of funding. Judged by results, the TAL had a wide-ranging impact, and facilitated implementation or prepared the ground for Bank assistance in many areas, such as SAL I and FESAL I and II, Social Insurance Administration, and Social Protection Adjustment Loans. Moreover, the majority of specific and major objectives of the project were achieved and assessed to be sustainable. The inflexibility in project design was compensated for by opportunism during implementation. Sustainability 4.2 The Bank and the GOB set in motion an ongoing process of capacity building that supported a relatively successful structural reform program. Sustainability of the TAL initiatives as a whole is likely because most of the measures have effected a substantial reorientation of Bulgarian institutions and are mutually reinforcing. In a project heavy on consultancies, the transfer of skills is a key indicator of sustainability. The Bulgarians took quick advantage of the new legal foundation, suggesting that there was already considerable unutilized capacity, but which needed only appropriate stimulus. Although project implementation was slow, a solid legal foundation was built in a number of areas, both for changes already made, for example, in the Enterprise Privatization and Restructuring, social security reform, and education components, and for future progress as well. The privatization of the largest enterprises, such as BTC, has been slow but substantial and it continues. Institutional Development Impact 4.3 The TAL has broad institutional development impact on many central agencies (as discussed under outcomes), and OED rates this impact substantial. Among the main components, the privatization of large companies, the legal framework, health and education made significant institutional development. Although the institutional development component was fragmented into small amounts of TA funds across many agencies these modest amounts initiated studies and training activities that prepared the way for follow up projects with major institutional development focus. 4.4 The PIU setup was relatively effective in meeting the fiduciary responsibilities and procurement and disbursement requirements of the Bank, particularly at a time when knowledge of Bank procedures was limited and government changes were frequent. Project management capacity developed quickly, especially the procurement and budgeting functions. The location of the central PIU in the MOF helped in providing some coordination for a rather comprehensive and multi-sectoral coverage. But, this setup did not address or encourage building sustainable capacity and ownership for country-wide management of TA resources. 8 5. Bank and Borrower Performance Bank Performance 5.1 Typical of Bank operations in transition countries, the Bank essentially was trying to design a program in a country it knew little about, and assumed that a project that touched as many bases as possible was likely to offer the greatest support to the transition and was more likely to be relevant than a highly focused one. In fact, the TAL did meet Bulgaria's needs in many areas and enabled key reforms for transition to a private sector-based economy. 5.2 Given the context (para. 3.1), relevance and outcome, Bank performance for the overall TAL is rated satisfactory. Despite key shortcomings (below), the Bank demonstrated some flexibility during implementation, but without fundamentally re-thinking the design. For example, the Bank was supportive of GOB using grant funds to replace TAL funds, thus giving greater priority to the interests of the client. Also, the Bank understood that under a certain regime, implementation would not proceed and judged that it was better to keep the TAL open so that when opportunities to finance priority activities arose, resources would be available. The five extensions indicated that the TAL continued to be useful, but only at a much slower rate than envisioned at appraisal. The Bank's patience paid off and much was accomplished in the later extension periods. 5.3 But the Bank erred on several fronts. First, the design was overly complex as it anticipated as many needs as were possible in a single operation. There were up to 18 implementing agencies in the project, tasked with carrying out more than 100 activities. Activities were as disparate as identifying safety recommendations at a nuclear power plant, creating a revolving fund for school textbooks and carrying out a study on structural issues in the banking system. However, despite the lack of synergy, TAL activities, separately, had substantial impacts. But, partly because of its complexity and many extensions, the TAL was expensive to administer. The costs of preparation, appraisal, and supervision exceeded the original plan by over $500,000, for a total of $771,000, which is a higher cost than for the TAL average in other countries. 5.4 Second, the TAL adopted a pre-packaged, highly detailed design, with specified amounts to many activities, and tight timetable. Not only were there more than 100 project activities, but they were very specific with specific dollar amounts attached to them. This specificity constrained implementation, requiring frequent amendments and extensions and high transaction costs. For a new member country in great economic uncertainty, the timetable clearly was too tight. The Bank's rationale was to keep the Bulgarian authorities focused on the capacity-building activities necessary for the transition to a private economy. But that led to frustrations, unrealistic expectations, and excessive focus on disbursement targets. At times, this led to lower-quality studies. The banking study, for example, needed improvements, but was prematurely accepted without sufficient commitment and participation on the borrower's side. 5.5 Third, the design and approach undermined ownership on both the Bank and Bulgarian sides. Many early signs suggested that the GOB (MOF) did not want to borrow for technical assistance. This attitude was reinforced by the mixed record and high costs of foreign consulting, and the availability of grants from the EU and other donors. With the TAL's direct link to the SAL, however, GOB accepted the TAL even though it meant committing to borrowing for TA activities. The intangible and longer-term outcomes of knowledge transfer and capacity building expected from the TAL were not fully appreciated, particularly by the MOF, and particularly compared with the quick financial resource transfers from adjustment loans. 9 5.6 Finally, Bank performance during implementation suffered from common problems of multi-sectoral free-standing TALs-instruments and objectives that call for sustained efforts, appropriate incentives, effective coordination, and multi-sectoral skills. The Bank started out with a large TAL that covered the full reform agenda, but then lacked sustained and informed attention to match the demands set by the TAL. Staff often viewed it as a second-class activity, as the TAL did not show visible and quick wins or could be favorably compared with the size of resource transfer of a SAL of FESAL. The Bank failed to use the TAL effectively to engage the authorities in policy dialogue and to broaden ownership of the reform process. The task management of the overall TAL changed frequently, and was assigned to relatively junior staff, the ICR was prepared by a consultant, and the institutional memory appears to be lost to current staff. Task management tended to focus on issues of procurement and disbursement, and other TA administration, with little dialogue on TA strategies and quality. Managers of individual components or tasks had to do their supervision on the side, while preparing other projects where they had more visible roles. Best practice delivery of TA was not always used. In sum, implementation suffered from the Bank's cultural ambivalence about technical assistance. 5.7 In summary, the Bank created a complex and rigidly designed project that had to be adjusted in practice and ultimately led to a strong positive development impact. The Bank performance improved gradually as it became more knowledgeable of Bulgaria, its development partners, and the changing environment, albeit at higher transaction costs than necessary and with uneven ownership among implementing agencies. Borrower Performance 5.8 As measured by its commitment to effectively use the resources of the TAL to develop the capacity to carry out TAL-supported structural reforms; GOB performance is rated satisfactory. The government committed to a broad program of capacity building and accomplished most of its objectives under the loan. Performance was especially good in the legal and enterprise reforms, and in the education components. Procurement performance also was good. A key factor in borrower performance was the continuity of staff in the central PIU and the long-term engagement of a seasoned advisor familiar with Bank procedures. 5.9 Borrower performance might have been better, however, with respect to maintaining continuity of personnel in the implementing agencies and the willingness to use foreign TA. Of course it was reasonable for the GOB to prefer grant financing and to be selective in its use of TAL funds for the most high-priority cases, but opportunities for high priority reforms were at times delayed or missed, at higher costs. The central PIU was a bit rigid in dealing with some of the implementing agencies in terms of their exacting standards in accounting for expenditures, which some of the agencies found a bit intimidating. In the Bulgarian National Bank (BNB) component, the BNB, not committed to banking reform, made little contribution to the study, and financial sector consolidation began without the benefit of a systematic study. Rather than disseminating the study among banks, BNB allowed the consolidation process to become overly politicized. 5.10 The PIJ, on behalf of the government, maintained sound accounts of TAL. Government compliance with legal covenants under TAL was considered satisfactory. Two covenants were not met, namely the establishment of an inter-ministerial steering group for the legal framework for state enterprises, and of a steering group to oversee privatization. However, neither covenant fit well within the administrative context and both were later deemed unnecessary. 10 6. Findings and Lessons Lesson 1: Results-based management and evaluation are difficult but critical to the implementation and assessment of technical assistance operations. 6.1 Assessment of results of many components suggest counterintuitive outcomes for small amounts of effective TA. The Education component of the TAL illustrates the catalytic effect of modest but effective technical assistance. A very small amount of TA was well received, strongly owned, and effectively used by the MOE. The TAL initiated major institutional and policy reforms, leveraged other major sources of finance, and prepared the ground for larger investment projects by the Bank and other donors. The MOE was also exposed to new ideas and best practices in,financing, management, and the technology of setting standards for education. The TAL contributions to the isolation and restructuring studies were also critical to privatization and fiscal stabilization. The TA for the legal framework covered fewer laws, but these were among of the most fundamental. Meantime, reform can go awry without properly done studies, at substantial costs to the economy, as may have been the case with the banking restructuring study. 6.2 But there are lingering perceptions within the MOF and the Bank that this TAL was ineffective, and therefore the Bank should not attempt to have any TALs in the future. These perceptions reflect excessive preoccupation with inputs, processes, and disbursement measures, than development impact. On process measures the TAL was slow to disburse, costly, and frequently amended and extended. The Bank may have given signals that these measures count more than its catalytic and development impacts. Some implementing agencies felt under more pressure to increase disbursement ratios than to work out complex institutional change and secure the most value added from the TA resources. These perceptions are reinforced by the difficulties to capture the results of TA, particularly indicators of progress in capacity building and institutional change. Bank monitoring and evaluation methods also reflect these biases: preoccupation with measurable input (funds disbursed) and process indicators (studies completed), but a neglect of performance management and results on the ground. Lesson 2: Tradeoffs between broad-based and highly focused TALs should be explicitly considered. 6.3 Bulgaria's first TAL was designed at a time of high uncertainty, poor understanding of country priorities and institutional constraints, and an underdeveloped division of labor among donors. Hence, its scope was inevitably broad. A highly focused TAL at the start could have become quickly irrelevant as conditions and priorities changed and new opportunities for reforms or institution building emerged. But, like many other TALs, the broad scope of this TAL also reflected competing agendas, lack of selectivity, low level of partnership, proclivity for complexity, and poor appreciation of implementation constraints. While ensuring high relevance, the broad scope, combined with specificity, taxed implementation and risked overall effectiveness. 6.4 Some pointers for guiding these tradeoffs are: a) coherence and synergy among TA activities to enhance chances for success; b) close links between TA and the priorities of the country assistance strategy; c) continuity of specific TA to meet long-term objectives and ensure sustainability of institutional reforms; d) likely availability of Bank resources and expertise to ensure the quality of supervision in selected areas; d) level of local ownership and commitment to the objectives of various activities; e) level of local coordination and capacity to manage complexity; and f) interest and expertise of other donors. 11 Lesson 3: Blueprint (pre-packaged) approaches to TAL are inappropriate, particularly in uncertain and fast-changing environments. 6.5 Blueprint (pre-packaged, designed in detail, and rigidly applied) approaches to TAL design are often the product of top-down thinking, excessive concern for control, and little appreciation of the learning and adaptation capabilities that can be engendered by flexible designs. Such approaches are a poor fit to TA, to institutional change in general, and to transition economies in particular. Pre-packaged design cannot substitute for local ownership and commitment or for having high-quality staff to guide knowledge transfer. The design for this TAL specified more than 100 project activities, with dollar amounts and ambitious timetables. This contributed to complexity, multiple amendments and extensions, higher transaction costs, lower ownership, cumbersome supervision, and perhaps drifting from the fundamental objectives. A more promising approach to TAL design would focus on a few broad objectives, and to use these to define activities through regular reviews during implementation. That would be akin to programmatic lending, applied to TA. But even here, the temptation to broaden the scope and cover too many sectors should be resisted, to ensure depth and quality in TA management. Lesson 4: Ownership of technical assistance must be nurtured throughout design and implementation. 6.6 The Bank has found it difficult to build country ownership and commitment to TAL operations, yet these are the essential condition of good performance. This audit gives some insight into ways to build such ownership. First, demonstrated quality matters to ownership, and the rising costs of TALs must be matched by increasing effectiveness. Ownership is more forthcoming when high-quality TA services are delivered. The audit found strong appreciation for timely advisory and good TA services, and equally strong opposition and resentment of wasted resources on poor consultancies. Second, ownership should not have been confined to the MOF, or even to the implementing agencies. Communication with key stakeholders is needed to build commitment to the intent, rationale, design, and implementation strategy for the TAL. Stakeholders should have been allowed to build the details during implementation. Joint discussion of results throughout implementation with all agencies can renew a shared vision for the project. Third, ownership could be enhanced by ensuring flexibility during implementation: in component activities, in timetable, and other areas. Fourth, adequate training in TA management for implementing agencies could enhance both ownership and performance. Finally, ownership and commitment vary over time, with changes in staff and government. They must be assessed, monitored, and continuously re-built, particularly in times of fast change. Lesson 5: Bank should create incentives and provide the resources needed for quality implementation of TALs. 6.7 TA is inherently difficult, and a highly risky and experimental investment at the best of times. A bias for resource transfer, by the Bank and its central counterpart, the MOF, devalues the critical role of TA for capacity building and implementation of reforms. It also deflects the Bank from investing the critical mass of energy and resources required to improve supervision and raise TAL performance. Overall task management of TALs is typically overshadowed by its fast- disbursing counterparts: adjustment loans. Measures of performance and impact for TALs are neither properly measured (lesson 1) nor adequately rewarded. It is therefore not surprising that this TAL suffered from staff turnover and narrow supervision. Quality of supervision typically depended on each sub-task manager's availability to supervise other operations, and on whether a specific TA activity would lead to preparing a Bank-financed investment or adjustment operation. 12 Lesson 6: More emphasis on timely monitoring and evaluation of the privatization process is necessary to continuously adapt and feed into ongoing privatization programs. 6.8 Concerning the privatization component, the focus of the Bank (and other donor assistance) has tended to be on the transfer of assets from the public to private sector. Speed was viewed as essential to secure momentum, particularly after a slow start during 1990-97. Fiscal needs were a primary driver of the process. But other measures of development impact and quality of privatization have been neglected, and perhaps as a result, lessons and feedback to improve further privatization and address post-privatization problems have been weak and slow. 6.9 For example, the Ministry of Industry argued that the consultancy under TAL provided a general prescription for Central and Eastern Europe countries experiencing similar developments, but was not tailored to the situation in Bulgaria. Specifically, MOI stated that the belief that privatization would take place through the participation of foreign investors turned out to be unrealistic both in terms of Bulgarian acceptance and of the supply of willing foreign investors. More attention to results and lessons learned will become increasingly critical to sustain public support for reforms, inform the search for more effective approaches to privatization, and inform public debate on issues of corporate restructuring, foreign investment, and growth opportunities. Such agile learning is even more critical as Bulgaria moves further in privatizing its infrastructure, as indicated by recent experience to privatize the Bulgarian Telecommunication Company (see PAR No. 22466 ). 7. Future Directions 7.1 Stand-alone TA is a crucial resource in transition economies, and in preparing the ground for deeper reforms. The slow adoption and disbursement of the TAL is less a reflection of the magnitude and urgency of these needs than a reflection of the rigid design and uneven ownership of the TAL by both the Bank and GOB. Good-quality TA enhances the prospects that future programmatic and fast-disbursing lending would be tied to real, tailored, and country- owned reforms. It is also indispensable to a "knowledge" Bank-to adding value and depth to Bank-assisted investment and adjustment lending. Given the immense needs of transition economies for institutional transformation, it is unlikely that knowledge transfer could be done only through economic and sector work from the Bank's own administrative budget. Of course, there will be areas where most of the analytical work and implementation support for reforms would be done or supported by other agencies, and Bank lending would only provide a "light touch." But many areas are not likely to be in such advanced stages or amenable to quick resource transfer, without strategic and sustained technical assistance. 7.2 Technical assistance needs for public sector reforms and private sector development remain substantial. As Bulgaria's recent Country Economic Memorandum suggests, and this TAL demonstrates, knowledge transfer and capacity building required to meet the dual challenge of completing the transition and joining the EU are significant. This TAL gave primary emphasis to preparing laws and policy studies for a new market economy. It paid little attention to capacity building to implement these laws and reforms. The new phase of technical assistance should focus on making the new policy framework work. The legal and regulatory frameworks are far from completed, and the enacted ones are likely to be revised in light of experience and a fast- changing environment. It also crucial to complete the supporting regulations and overhaul the administrative procedures, and to build capacity to monitor and evaluate the laws and regulations 13 on an ongoing basis. Public administration also needs fundamental re-orientation: from a command economy state service to a market-oriented, client-focused civil service. 7.3 Capacity building needs in the private sector and civil society may be also considered. A first TAL could not cover either sector. Few donors are extending innovative programs for know- how transfer and training to small and medium enterprises, but much more is needed for the private sector to develop and become competitive. Privatization is not enough, as the current problems after privatization of the agro-processing industry in Bulgaria demonstrates. Moreover, the infrastructure sectors are critical to the competitiveness of the overall economy, and much of the restructuring and regulatory frameworks of these sectors are still in their early stage of development. The recent attempt to privatize the Bulgarian Telecommunications Company under this TAL suggests how difficult and uncertain this process is likely to be. Second-generation regulatory and unbundling issues are likely to follow. Finally, civil society is likely to play increasing roles in reform, governance, and delivery of social services. But, more TA resources are needed to share information and build capacity within the civil society to become an effective partner. 7.4 The Bank must think through its comparative advantage in technical assistance, develop effective partnerships for TA at the local level, support government capacity for TA coordination, and ensure that its TA programs are strategically selective and supportive of the overall development strategy. This is particularly critical for countries where substantial grant funds are available for TA. As a first step, a comprehensive review of technical assistance is needed to determine national priorities for knowledge transfer and capacity building in support of policy and institutional development and EU accession. The Bank should work closely with the Government, private sector, and civil society to develop a consensus on these needs and ownership for the priorities and modes of TA delivery. The Bank should also clarify its comparative advantage as a knowledge institution, and how it will share and leverage its knowledge, and how it will mobilize and adapt global knowledge to deliver tailored advisory and capacity building services for Bulgaria. In particular, the Bank must articulate a strategy of assistance that takes account of the substantial concessionary resources available from the EU and other major TA sources, and build the appropriate partnerships. 7.5 A more thoughtful Bank approach to technical assistance for Bulgaria may consider the following factors. First, several Bank studies suggest that the Bank's TA loans continue to be the lowest performing in the portfolio. Yet, the Bank and other donors persist with TA, despite the mixed record. Moreover, reflecting the common view of TA as a free good, the Ministry of Finance is reluctant to borrow for TA regardless of its benefits. This audit does not argue that the Bank ought to get out of TA, but to try harder to improve the design, implementation and focus of such assistance. "Hard money for soft advice" is a bad bargain, but good advice and good training can have high returns, justifying borrowing even on quite hard terms. 7.6 Second, technical cooperation resources from leading donors have been growing since 1997, and for 1996-98 are estimated at 16 percent of total assistance. Given the scale of resources and multiplicity of players, there are gains to be made from prioritizing and coordinating the fragmented (among sectors and levels of governments) and uneven (among regions) allocation. Third, the Bank's TAL has focused on the central government, and its investment-embedded TA has focused on project implementation, while donor's TA was more focused on local governments and firm-level assistance. Fourth, these resources are not homogenous, some donors are slow to respond to windows of opportunity for reform or privatization, some are better at local-level capacity building, and some are focused on particular sectors or issues. Partnerships between the Bank and different donors for timely and effective delivery of TA should be assessed 14 in terms of the different benefits, costs and risks to the client and the Bank. Finally, IFC and the Bank should work more closely on TA for private sector development. 7.7 Finally, TA is closely intertwined with the Bank's ability to influence reforms and investment priorities and to facilitate capacity building. For this reason, it would be undesirable the Bank to exclusively rely on TA provided by other donors to implement the Bank's assistance objectives. The challenge is to target the Bank's rather limited TA lending to strategic areas that can leverage overall country assistance and complement the substantial TA resources of other partners. The supervision process should be thought through, including the balance between learning and control, and the role of the country office and local consultants. Key elements of success may be to start with modest amounts, commit to few but broad priorities, remain flexible in design, assign experienced staff to provide quality supervision, enhance the role of country office to leverage local and partner's knowledge, secure incentives and budgets for quality implementation, and build local ownership for the often difficult, intangible, and long-term objectives of TA. 15 Annex A Annex A. Basic Data Bulgaria-Technical Assistance Project for Economic Reform Project (Loan 3384-BU) Key Project Data Appraisal Actual or Actual as % of Estimate current estimate Appraisal estimate Total project costs (US$ M) 33.5 12.6 37.6 Loan amount (US$ M) 17.0 12.3 72.4 Cancellations (US$ M) Date physical components completed: *Appraisal cost estimates are not comparable to actuals because they included large amounts of funding (US$13.2 million from EC-PHARE) that never materialized. Cumulative Estimated and Actual Disbursements (US$ million) FY92 FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 Appraisal estimate 5.3 13.3 17.0 17.0 17.0 17.0 .17.0 17.0 17.0 Actual 2.0 2.8 2.8 4.5 6.3 10.6 11.3 12.3 12.3 Actual as % of Estimate 38 21 16 26 37 62 66 72 72 Date of final disbursement 10/99 Project Dates Steps in project cycle Original Actual Identification/Preparation 11/90 Appraisal 2/91 Negotiations 5/91 2/10/91 Board presentation 6/25/91 6/27191 Signing 7/22/91 Effectiveness 8/30/91 1/9/92 Project Completion 12/31/93 6/30/98 Loan closing 6/30/94 6/30/99 Staff Inputs (staff weeks) Stage of project cycle Actual/Latest Estimate Number of Staff Weeks US$ Preparation through Appraisal 249.3 Negotiations through Board approval 23.3 Supervision 470.3 Completion 28.9 Total 771.7 16 Annex A Mission Data Duration of Performance ratingsb Stage of project cycle Date No. of staft mission Specializations Implement. Develop. Types of (month/year) in field (# of days) represented Status Objectives problemsc Through appraisal Appraisal through Board approval Supervision 6/91 6 14 E,F,SS N.A. N.A. Supervision 6/92 5 14 E,F,SS, En S S Supervision 6/93 4 15 E,F,SS, En S S Supervision 4/95 5 12 E,F,SS S S Supervision 6/95 4 12 E,F,SS U U Supervision 3/96 6 16 E,F,SS U U Supervision 10/96 4 11 E,F,SS U U Supervision 7/97 4 14 E,F,SS S U Supervision 6/98 4 11 E,F,SS S U Supervision 7/98 3 10 E,F,SS S U Supervision 12/98 3 9 E,F S S Supervision 6/99 6 1 E S S a. E = Economics, F=Financial, SS=ocial Sectors, En=Energy. b. HS = Highly Satisfactory; S = Satisfactory: U = Unsatisfactory; N.A.=Not applicable Other Project Data Borrower/Executing Agency: FOLLOW-ON OPERATIONS Operation Credit no. Amount Board date (US$ million) SAL I FY92 Debt and Debt Service Reduction Loan FY95 Rehabilitation Loan FY97 Financial and Enterprise Sector Adjustment Loan FY97 Social Insurance Administration FY97 Social Protection Adjustment Loan FY99 17 Annex B Annex B. Comments from the Borrower REPUBLIC OF BULGARIA MINISTRY OF FINANCE 102 Rakovsky St., 1040 Sofit Telex: 22727, Fax: 980 68 63 June 27, 2001 Mr. Alain Barbu Sector and Thematic Evaluation Group Operations Evaluation Department International Bank for Reconstruction and Development 1818 H Street, N.W. Washington, D.C. 20433 U.SA Dear Mr. Barbu, Re: BULGARIA - Technical Assistance Projectfbr Economic Reforn (Loan 3384-BU) Drafi Performance Audit Report We refer to your letter of June 11, 2001 on the Techical Assistance Project for Economic Reform Loan performance evaluation. We have reviewed the World Bank's Operations Evaluation Department draft Performance Audit Report and have no comnents on the latter. Thank you for the opinion and evaluation work done. Yours sincerely, Plamen Oresharski Deputy Minister of Finance Cc; Mr. Pieter Stek Executive Director World Bank