Document of The World Bank FOR OFFICIAL USE ONLY Report No.: 23292 PROJECT PERFORMANCE ASSESSMENT REPORT RUSSIAN FEDERATION HIGHWAY REHABILITATION AND MAINTENANCE PROJECT (Loan 3706-RU) November 29, 2001 Sector and Thematic Evaluation Group Operations Evaluation Department This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Currency Equivalents (as of May 11, 2000) Currency Unit = Ruble 100 Rubles = US$3.47 US$1 = 28.8 Rubles Weights and Measures m = meter (3.281 feet) km = kilometer (0.621 mile) Principal Abbreviations and Acronyms CPI Consumer Price Index DORINVEST Project Implementation Unit, under FHD DRSU Road Maintenance Units DSU Road Construction Units ERR Economic rate of return FHD Federal Highway Department RFHS Federal Highway Administration of the Russian Federation FSU Former Soviet Union HDM III Highway Design and Maintenance Model HRMP Highway Rehabilitation and Maintenance Project ICB International competitive bidding IRI International Roughness Index MOT Ministry of Transport MTR Mid-term review NPV Net present value SAL Structural Adjustment Loan Fiscal Year January 1 - December 31 Director-General, Operations Evaluation Mr. Robert Picciotto Director, Operations Evaluation Department Mr. Gregory K. Ingram Manager, Sector and Thematic Evaluation Mr. Alain Barbu Task Manager Mr. Hernan Levy FOR OFFICIAL USE ONLY The World Bank Washington, D.C. 20433 U.S.A. Office of the Director-General Operations Evaluation November 29, 2001 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Performance Assessment Report on the Russian Federation Highway Rehabilitation and Maintenance Project (Loan 3706-RU) Attached is the Performance Assessment Report prepared by the Operations Evaluation Department on the Russian Federation Highway Rehabilitation and Maintenance Project (Loan 3706-RU, approved in FY94). The $300 million loan closed on December 31, 1999, 18 months behind schedule. An undisbursed balance of $19.2 million was canceled. This first highway project for the Russian Federation had wide-ranging physical and institutional objectives that aimed to improve the condition of the highway system and its management. Such objectives included slowing down the deterioration of priority roads; launching reforms in the structure of road administration and the construction industry and introducing competitive bidding; starting development of a consulting industry for the sector; improving the efficiency of public expenditures in roads; improving road safety; helping reform the institutional framework of the transport sector and the transport ministry; helping preserve employment in road rehabilitation and maintenance works while shifting more employment to the private sector. The project components included road rehabilitation (1,500 kin) and routine maintenance and safety improvements (10,000 ki); equipment, materials and supplies; and technical assistance, training and studies. The project included four pilot, road rehabilitation works intended to gauge the degree of market competition and provide an early indication of unit costs. Since the economic transition that started in the beginning of the 1990s was expected to significantly increase the role of the road system, which originally had been conceived as a short- distance feeder to the railway, the project's proposed improvements were relevant for the Russian economy. However, problems with project design, including a poorly conceived routine maintenance component, diminished relevance. Project complexity, stemming from broad objectives coupled with significant political, economic, and institutional uncertainty prevailing throughout the implementation period, affected project implementation. The uncertainties played a major role in the degree of competition in the bidding process and led to bid prices substantially higher than expected. Issues of technical standards and construction quality also affected implementation. The project's main physical objective were only partially achieved. Some 760 km of roadway were rehabilitated, or about half the original target (a revised target of 860 km agreed at the mid-term review was not reflected in a project amendment). The shortfall was due to higher- than-expected bid prices and inflation-based price adjustments, problems with design, and rapid road deterioration under increasingly heavy traffic. To secure funding for the road rehabilitation works, the mid-term review eliminated the routine maintenance subcomponent This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. -2- (under a commitment, whose compliance could not be verified, that such works would be carried out under domestic finance outside the project). The economic return for the rehabilitation investment was 74 percent (the appraisal estimate was over 100 percent) but there is no clearcut evidence that the scaled down project, with its much higher than expected unit costs, remained the least cost alternative. The project's institutional objectives were clearly a "stretch" for a first highway project and few immediate results could be expected. The project introduced international competitive bidding and helped launch development of a competitive domestic construction and consulting industry. Progress in other institutional areas was limited to raising awareness of relevant policies - notably about cost recovery and financing - and launching introduction of management techniques such as a model for road design and maintenance. The audit rates outcome as moderately unsatisfactory and institutional development as modest. Sustainability is rated as non-evaluable because the government in 2000 abolished a dedicated Road Fund that ensured adequate funding for road maintenance, and it is too early to pass a well-founded judgment on future funding from the central budget. Bank performance is rated unsatisfactory because of the project's inadequate quality at entry and poor performance during the mid-term review (including failure to formally restructure the project). Borrower performance is rated unsatisfactory because, despite a mostly good performance of the implementing agency, it was weak in pursuing the institutional objectives and made no progress in developing an accounting and auditing system for the Road Fund, which would have improved its financial management and likely avoided its demise. These ratings are generally lower than those of the implementation completion report. Experience with this project confirms the following OED lessons: * A wide-scope project design for a first-time Bank operation in a sector makes sense only if the operation will launch initiatives that will be continued in follow-on projects. Cancellation in September 1999 (prior to effectiveness) of the follow-on HMRPII and lack of other subsequent operation raise serious doubts about future lending in the road sector. * The HRMP suggests that PIUs may be essential in transition countries, especially if key sectoral agencies are being, or are likely to be, restructured, and generally for a first project in a sector. The project also showed that PIUs can be designed to avoid being privileged units that generally undermine institutional morale. * The use of a phased approach that starts with pilot components is advisable, especially in transition economies subject to significant economic, political, and institutional uncertainties, but achieving maximum effectiveness requires an early assessment of the results of the pilots, and the possibility to redesign the program if experience warrants. * Road funds - conditions for survival. The Russian federal and regional road funds confirmed experience elsewhere that road funds need to meet essential conditions to ensure their rationale, public trust, and, ultimately, their survival. Two such conditions exemplified by the Russian experience were missing and eventually led to the demise of the funds: (i) revenues feeding the funds should be user charges, not general taxes ( the regional fund was funded by taxes on enterprises' turnover), and (ii) a credible system of accounting and auditing the road funds should be in place. Attachment 1 Contents Principal R atings ........................................................................................................................... iii K ey Staff Responsible...................................................................................................................m Preface.............................................................................................................................................v 1. Background .............................................................................................................................. 1 2. Objectives and Design ......................................................................................................... 2 Relevance of Objectives .................................... ........ 3 Design Issues ................................................... 4 3. Im plem entation ........................................................................................................................ 5 Fiduciary Aspects............... ................................. 7 Procurement .................................................. 7 Financial Management. ....................................... 8 Allegations of Corruption ................................. .......... 9 Stakeholders' Workshop ..................................... ....... 9 Construction Costs..... ......................... ............. 10 4. O utcom e .................................................................................................................................11 Physical Objectives....................... ................. 11 Economic Return................. . ...................... ...... 11 Institutional Development Objectives ...........1.......................I 5. R atings...................................................................................................................................14 Outcome, Institutional Development, and Sustainability.................................... 14 Bank and Borrower Performance...................................... 15 6. L essons................................................................................................................................. 16 Basic Data Sheet ........................................................................................................................... 19 Borrower Com m ents ................................................................................................................... 21 This report was prepared by Hernan Levy (Task Manager) who assessed the project in April/May 2000. William B. Hurlbut edited the report. Romayne Pereira provided administrative support.  111 Principal Ratings ICR PAR Outcome Satisfactory Moderately unsatisfactory Sustainability Likely Non-evaluable Institutional Development Modest Modest Bank Performance Satisfactory Unsatisfactory Borrower Performance Satisfactory Unsatisfactory Key Staff Responsible Task Manager Division Chief Country Director Appraisal C.Queiroz J.Brown R.Cheetham Completion A.Bonde E.Molnar M.Carter  V Preface This is a Performance Assessment Report (PAR) for the Russian Federation Highway Rehabilitation and Maintenance Project (Loan 3706-RU), for which the World Bank approved a loan of $300 million in February 1994. The loan closed on December 31, 1999, 18 months behind schedule. This report is based on reviews of the Implementation Completion Report (ICR), the Staff Appraisal Report (SAR), loan documents, project files, transcripts of Board proceedings, and other Bank documents, and on discussions with Bank staff. An OED mission visited Russia in April-May 2000, visited project-financed works and reviewed the project with government officials and other stakeholders. Their kind assistance is gratefully acknowledged. The OED mission partially overlapped with a mission preparing an Intensive Learning ICR for this project. During the period of overlap a stakeholders' workshop was carried out with participation of representatives from various government agencies, consultants, and contractors. The PAR goes substantially beyond the ICR in three areas: project design; institutional development impact, especially regarding highway financing and management; and, the project's fiduciary aspects. Following standard procedures, copies of the PAR were sent to Russian authorities for comments. Comments were received from a former director of the Project Implementation Unit, DORINVEST, and are attached as Annex B.  1 1. Background 1.1 The Highway Rehabilitation and Maintenance Project (HRMP) was prepared, appraised, and implemented under unusually difficult conditions: * The Russian Federation, established in 1991, was in the early stages of transition from a centrally planned economy toward a market-oriented economy and was subject to high economic and institutional instability. * The project was the first Bank operation in the transport sector of the Russian Federation, and therefore the Bank had limited knowledge and no operational experience in this sector. * Reciprocally, the road sector authorities were not conversant with Bank procedures. * It was the first Bank project in the Russian Federation, in any sector, that financed civil works, and with it the Bank introduced procurement through international competitive bidding.' 1.2 The political and economic transition of the Russian Federation was marked by weak governance in all spheres of public life, as described extensively by both the press and academic journals, especially from the mid-1990s onwards. During the period, old institutions and policies were being dismantled, but new ones had not yet been put in place. 1.3 On the economic front, as shown below,' the gross domestic product declined by over 30 percent during 1994-99, with only one year showing a marginal growth. Inflation was rampant, and, although it declined substantially after 1995, it still remained in double digits. Year 1992 1993 1994 1995 1996 1997 1998 1999 GDP Annual Growth (%) -14.5 -8.7 -12.6 -4.1 -3.4 0.9 -4.6 -2.0 Inflation - annual average (%) 1354.0 895.3 303.2 188.7 47.5 14.8 26.8 70.0 1.4 On the political and institutional front, civil unrest in 1993, while the HRMP was being prepared, was followed by a new government in April 1994. Significant political instability ensued. During the period from mid-1997 to mid-1999, when the Bank's first Structural Adjustment Loan (SAL) was implemented, there were four changes in government accompanied by several changes of prime minister and key ministers.' 1.5 The Bank's first involvement with Russia's transport sector was a mission in February 1992, which found that: 1. A Bank loan for $605 million for an Oil Rehabilitation project, approved a few months earlier than the HRMP, financed equipment, materials, and commodities, but no civil works. 2. For example, M.Levin, G. Satarov, 2000. Corruption and Institutions in Russia. European Journal ofPolitical Economy, 16, 113-132. The paper notes that in 1995-96, national and regional newspapers and magazines published more than 3,000 articles on corruption. The paper notes that, historically, corruption has begun when a country enters into a period of modernization and change. 3. Source: The Russian Federation, Structural Adjustment Loans (RU-PE-49203 and RU-PE-50491), ICR of July 15, 1999, and Rehabilitation Loan (3513-RU) ICR of February 17, 1997. 4. Levin and Satarov 2000. 2 * Roads had been designed as feeders for the railway to avoid duplication of infrastructure (i.e., they focused on short trips associated with distribution of goods in the urban areas). * About 90 percent of the network was designed with little structural capacity (only to support 6-ton axles, although the country's regulations permit 10-ton axles - by comparison, the EEC limit is 11.5 tons). * Roads were not designed for all-weather use, only 19 percent of the highways were open year-round. * There was a large maintenance and rehabilitation backlog. * The road network was in poor condition and had little geographic coverage. * The highway authority wished to commercialize construction and maintenance, and convert its force account units into private contractors, but lacked an appropriate regulatory framework to facilitate the change. 1.6 Despite the weak condition of the road system, and consistent with experience in other transition economies, a gradual shift of traffic from the rail to the road started soon after the establishment of the Russian Federation. Consultant forecasts cited in the SAR for the HRMP foresaw that road transport would approximately double the percentage of non-energy freight (i.e., excluding oil and coal), from 13 percent in 1993 to 22 to 41 percent by 2015, depending on the rate of economic growth. 1.7 An unexpected feature of Russia's highway system, at the time of project preparation and appraisal, was that only one year after establishment of the Russian Federation, the government had set up a road fund system (including separate federal and regional funds) to finance the road network. This decision appeared to have been the result of (i) a positive assessment, by key Russian highway authorities, of the "user-pays" concept' of highway financing in the United States and other countries, and (ii) the need to ensure a stable source of funds to maintain and improve the road network.' 1.8 The HRMP was one of several projects that the Bank started processing even before the Russian Federation officially joined the World Bank, and was one of seven projects that the Bank had committed to process in fiscal 1993 and fiscal 1994 in the context of the G-7 program of assistance to the Russian Federation. Projects prepared under that program were expected to contribute a large transfer of resources. 2. Objectives and Design 2.1 The HRMP had significant physical and institutional objectives and components (Box 1) that aimed not only to improve the condition of the highway system but also to improve its operation and maintenance. Given the increasing demands being placed on this system as a result 5. Users pay via road-related taxes and charges and the proceeds are deposited into an off-budget account. 6. The road funds, comprising one federal road fund and 86 regional funds, were legally established in October 1991 and became operational in April 1992 (SAR, p2.37). While similar in concept, there are significant differences between the U.S. Highway Trust Fund and the Russian road funds. A key difference is the source of financing, since in the U.S. road fund revenues are collected from road users through a gasoline tax, while in the case of Russian regional road funds, a significant source of revenue is a tax on the gross revenues of enterprises. 3 of the modal shift from rail to road, such improvements would be beneficial for the Russian economy. Box 1. Project Objectives and Components Objectives * Slow the rapid deterioration of high priority parts of the public road network * Help preserve employment in essential rehabilitation and maintenance works and shift more employment to the private sector * Initiate reform of the institutional structure of road administration and the contracting industry, and introduce competitive bidding * Begin development of a road design, supervision, and environmental assessment consulting industry * Improve the efficiency of public expenditure on roads by supporting improvements in management, planning, and selected technical areas * Improve road safety * Help develop the policy, legal, and institutional framework for the transport sector and restructure and improve the capacity of transport ministry Components Road rehabilitation - rehabilitate about 1,500 kilometers of roads, make maintenance and road safety improvements on about 10,000 kilometers of mostly federal roads Equipment, materials, and supplies - provide vehicles, equipment, materials, and supplies to aid better management of roads Technical assistance, training, and studies - develop Federal Highway Department staff capabilities, train local contractors, and help the Ministry of Transport to develop the policy, legal, and institutional framework for the transport sector and restructure and improve the capacity of the transport ministry Relevance of Objectives 2.2 While the objectives of the project were broadly relevant to the expressed needs of the Russian Federation and to Bank strategy for the country, they were also, in hindsight, too ambitious for a first project in the sector. Experience with other first-time borrowers, especially in transition economies, bears this out. Just the process involved with a project's engineering and economics, and especially the introduction of competitive bidding for large civil works, is likely to absorb most of the interest and energy of the borrower and implementing agency during project preparation and implementation. During the identification and appraisal of HRMP there was reason to believe that this would be the case in Russia, which was (and is) undergoing an intricate transition process. To accomplish the changes envisaged by the project objectives, a sequence of operations rather than a single project would have been more suitable than the "big bang" approach of a single, highly complex, project loan. 2.3 Of the specific objectives, one, that of preserving employment in essential rehabilitation and maintenance works, was ill-advised. Overall unemployment in Russia was only 3.0 percent in 1992 and 5.3 percent in 1993, when the project was being prepared.' Moreover, maintenance and 7. Although it is true that, as noted in the ICR, "unemployment did not increase as rapidly as expected during the initial phase of the transition process. At the end of 1994, the number of officially registered unemployed workers amounted to only 2.2 percent of the labor force, while.. .a broader definition would raise this number to about 7 percent." 4 rehabilitation of main roads is not an activity that yields a high number of jobs per unit of capital. Furthermore, the objective of preserving employment may well be inconsistent with the objective of improving the efficiency of public expenditure on roads. The modem technologies that the project was expected to introduce to increase efficiency and quality of construction are capital intensive and use less labor per unit of output than the more traditional road construction methods used in Russia. At a time when the organization was moving from force account to contracting out, which normally involves redundancies, a more relevant objective would have been to assist in this transition and downsizing through retraining and other means. Design Issues 2.4 The design had some severe weaknesses, which were noted and partially corrected during the midterm review, as well as some strengths. 2.5 The subcomponent that aimed to improve, through routine/light maintenance, about 10,000 kilometers of federal roads appears, in retrospect, to have been poorly conceived. While the linkage between project objectives and components is well explained in the detailed logical framework model included in the SAR, the appraisal did not thoroughly analyze the issues associated with this part of the project. Although not labeled as such, this subcomponent was designed as a "sector loan" since the individual investments were not identified prior to appraisal, although selection criteria were listed. This assessment finds two major flaws with this component. First, the program was too large. In practice, it would have been unmanageable, or only manageable at a very high transaction costs, for the Bank' and the borrower. According to the SAR, the program would have resulted in about 270 civil works contracts, leading to about 350 contracts for the whole project when all components are considered. This is three to five times the number of contracts in similar highway projects in other countries. Second, as a "sector- type" component, the SAR should have reviewed the overall Russian highway maintenance program and assessed how the 10,000 kilometers would fit in that program."o This was not done. In the end, this component was deleted during the project's midterm review. 2.6 The same subcomponent also included a pilot study of maintenance by contract for about 500 kilometers of regional roads in three reform-oriented oblasts, with a view to helping them determine the most appropriate system of road maintenance. This was a worthy activity, with an important goal, that could have provided important insights into the management of routine maintenance in Russia. Yet, it was canceled along with the rest of the subcomponent, missing an important opportunity. 2.7 Technical assistance was an important component of the project. A review of the project by the Bank's adviser on technical assistance found the design of this component, as presented in the 8. The Region argues that the objective of preserving employment in road rehabilitation and maintenance, is not inconsistent or ill-advised with the overall project objectives, and claims that " recent studies of development effects of transport investments show that employment and its feedback effects are the primary, substantial effects in a disequilibrium economy, which Russia clearly was." 9. Even if most contracts were small and subject to post-review only. 10. The Region argues that the primary motivation of this component was to carry out light rehabilitation by contract so as to develop the private sector. It also states that monitoring implementation of this component as suggested by the PAR would have been impossible. While recognizing the importance of developing the private sector, OED finds that this component was too large for that purpose and that if works were not monitorable, this would have posed a major issue with the accountability of the funds for such works. 5 SAR, to be a model of emerging best practice. The same review praised the design of twinning activities between the project and the United States Federal Highway Administration." 2.8 The road rehabilitation component included as a first stage four pilot projects. The pilots were designed to gauge the degree of market competition and provide an early indication of unit costs. This approach was conceptually sound, given the significant uncertainty surrounding the market for execution of civil works. It is not clear, however, whether the project would have benefited from a more formal approach to stocktaking from these pilots. For example, the project could have included, prior to the midterm review, a specific review to assess and revise project targets based on the results of the pilot projects." 2.9 At the same time, the pilots showed that there was substantial uncertainty surrounding the implementation of the road rehabilitation component. This raises the question of whether physical targets were too risky and whether a range rather than a specific target would have been a better way to describe this component in the SAR. 2.10 During appraisal, the Bank was concerned about the need for Russia to rationalize road user charges, to ensure that they provided the right "price" signals to truckers and passenger vehicles. One special concern was the low price of gasoline. Another concern related to the need to develop an accounting and auditing system for the federal road fund. These concerns were incorporated in the project's Action Plan, rather than as a legal covenant. For a project that already included a multiplicity of objectives, this was an appropriate formulation. 2.11 The feasibility studies of the project-financed investments, carried out by international consultants together with Russian experts, suffered from two major drawbacks. First, engineering data was based on "as-designed documentation" from the soviet era, which proved unreliable especially regarding drainage and service lines. Second, traffic forecasts appear to have been overly conservative, since the low growth of traffic (1.5 percent) in 1991 was projected to continue for the first two years of the project. This assumption seems to contradict the expectations of overall high growth of road traffic in Russia (para 1.6). The projected low traffic volumes was an important input leading to the design of pavement overlays that in some cases were too thin. 3. Implementation 3.1 Project implementation was significantly affected by the economic, political, and institutional uncertainty prevailing throughout the implementation period. Among other effects, these uncertainties played a major role in the degree of competition in the bidding process and influenced bid prices. 3.2 A number of other problems also affected this project during implementation, including issues of technical standards, construction quality (including the need to repair some of the works done), and high costs. Contractually, by project completion, there were two unresolved problems, one with a civil works contractor, and one with a consultant. By project completion the condition of the works done was generally satisfactory, " although some defects (rutting) appeared on three 11. Memo by N. Raphaeli to then Country Director Mr. R. Cheetham dated September 6, 1994. 12. While today a Learning and Innovation Loan (LIL) would be used for such a purpose, the lending instrument was not available at the time HRMP was appraised. 13. As noted in the December 18, 1998, supervision report. 6 highways because of a sharp increase in heavy transport loads that exceeded the structural capacity of the pavement. The responsible contractors were expected to repair these defects at their own expense. 3.3 The ICR lists the following key implementation problems, describes the factors underlying them, and analyzes them in detail: * The introduction of international competitive bidding for road construction, required government officials to think and plan for the first time in terms of cost-effectiveness and quality rather than fulfilling centrally set quotas. This was a radical change and bound to be difficult. * Political and economic uncertainty reduced the level of competition for the works, and led contractors to add a substantial risk margin to their bids.14 * High inflation and uncertain exchange rate policies further exacerbated the risks and complicated the design and application of an equitable price adjustment formula. * Application of modern highway design techniques for project investment (such as the Bank's Highway Design and Maintenance Model, which has been used successfully in many countries) was hindered by poor engineering data, unreliable traffic counts, overly conservative traffic projections and by the difficulties of calibrating the model not originally designed for cold weather to Russian climate and road conditions. The result was poor designs that had to be revised during implementation. * Critical changes during implementation of highway sector organization and senior personnel. 3.4 To manage the project, the Federal Highway Department established an implementation unit, DORINVEST, staffed by professionals from various FHD units and reporting to FHD's senior management. During the early years of implementation DORINVEST acted as a significant stabilizing and trust-building factor for the project. It had sufficient staff, an appropriate skill mix (engineers, accountants, lawyers), seasoned and dedicated professionals, and was supported by substantial expertise provided by international advisers. It was thus able to play a key role in ensuring satisfactory project implementation. Unfortunately, following institutional changes in the highway sector, starting in late 1998 until the end of the project, the staff of DORINVEST was sharply reduced and it lost many of its key personnel. In retrospect, the special circumstances of this project - a new Bank project in the sector for a borrower in the early stages of transition - fully justify the decision to establish a PIU. " 3.5 Highway Organization. There were many organizational changes in the highway sector during project implementation. The most significant was the creation of the Federal Highway Service as a successor to the Federal Highway Department in April 1997, the transfer under its 14. For example, the Final Executive Project Summary (FEPS) draft of April 1993 notes that during the past 12 months, the consumer price index increased 26 times, while the exchange rate changed only 3 times. Such extreme conditions were certain to induce contractors to include a substantial risk margin in their bids. 15. In PARs of highway projects in other countries, OED has criticized the establishment of PIUs because they often cut across internal agency boundaries, provide their staff with special compensation and other privileges, and are therefore a force against institutional development. These negative traits apparently were not present, or were greatly reduced, in the HRMP since DORINVEST received high praise for its work (except for its performance in the last two years of the project) during the OED evaluator's discussions both with government and Bank staff. 7 authority of the federal road fund, and the appointment of a senior official from the tax administration to manage the agency. Thereafter, the highway agency no longer reported to the Ministry of Transport, but to a deputy prime minister. These changes appeared to have had an adverse effect on project implementation mainly because key personnel who had been involved with the project since its beginning remained in the Ministry of Transportation. 3.6 Reduction in scope of civil works. The midterm review (June 1996), considering the substantial cost increases for the civil works, decided to reallocate all funds for the routine maintenance subcomponent to the rehabilitation works. At the same time, the midterm review, considering the funds so reallocated and the actual prices of the rehabilitation works under way at the time, concluded that the rehabilitation goals had to be scaled down from the original 1,500 kilometers to 860 kilometers. The midterm review reports that, in parallel, the government committed itself to carry out, under its own financing, the 10,000 kilometers scheduled for maintenance or improvement under the project. However, the midterm review provides no details regarding the process to identify, select, and evaluate investments, the program of work for the 10,000 kilometers, and the role of the Bank in that program. 3.7 The project files reveal no substantive discussion on the midterm review by Bank management, which is surprising since the review decided on (i) a major reduction in the scope of the project's main subcomponent; and (ii) the elimination of the routine maintenance subcomponent, that represented close to 30 percent of the estimated project cost. 3.8 Traffic-related equipment. The project included purchase of traffic counters and equipment (mobile and fixed weighbridges) to weigh trucks' axle loads. The borrower started but did not complete the procurement process, and the equipment was not procured. In view of the widespread notion that traffic estimates in Russia are highly unreliable (during the auditor's visit to one of the project roads, engineers stated that they had seen no traffic counts for that road), failure to acquire the counters may significantly hinder efforts to improve both highway planning and maintenance management. On the other hand, international experience with enforcing axle load controls is mixed at best, so effective use of the weighing equipment, other than for collecting axle-load information, would have been doubtful. Still, some Russian-manufactured equipment is being installed under local financing." Fiduciary Aspects" Procurement 3.9 As an investment project, HRMP loan funds were directed to specific and clearly identified works, goods, and services. Civil works accounted for some 92 percent " of the loan. Contracts for 90 percent (in dollar value) of these" works were awarded through international competitive bidding, and the remainder through national competitive bidding Project files reveal that processing of civil works contracts followed established Bank procedures and were subject to the 16. A report of July 1997 by Bank consultant Leo Rothenburg notes that weighing equipment is being installed in some oblasts, but only two in the Leningrad oblast are actually functioning. This equipment is of Russian design and manufacture. The consultant states that such equipment is sensitive to temperature and moisture and does not work effectively in all seasons. 17. OED financial management specialist Vinod Saghal assisted with the design and review of this section of the assessment. 18. Based on the Loan Document, civil works plus unallocated categories. 19. ICR, Annex 2. 8 Bank's regular review and approval process. In at least one case, there was substantial disagreement between the Bank and the borrower, and within the Bank. The case involved very high bid prices, since the lowest evaluated bid was more than 2.5 times the engineer's estimate. In the end, however, after much debate and review at the highest procurement levels, the Bank agreed with the borrower that the bids had been competitive and that the prices reflected actual market conditions. This case gave rise to allegations of corruption (see below). An audit prepared by PricewaterhouseCoopers (PWC) in April 1998,20 reviewed a sample of nine rehabilitation contracts and noted that competition "can be considered sufficient to high." The Bank's Small Works document was used for all civil works bidding and contracts. This document is simpler, especially regarding resolution of disputes, than the standard document. While not conclusive, this auditor found that contractors and government officials appeared to be satisfied with the convenience of this document. 3.10 Regarding consulting services, the PWC audit found that major consultants' contracts for engineering design and supervision were not in line with Bank's rules, and notes two specific issues (i) the short lists were too restrictive in number (four consultants), leading to few bids, and (ii) consultant contracts were awarded as packages rather than respecting the results of ranking for individual contracts. Bank guidelines recommend that a short list consist of four to six consultants. An exemption to the compliance with Bank procedures, found during the review of files for the preparation of this assessment, is a non-objection for the selection of the consulting firm for the Ministry of Transportation study, conveying the Bank's "non-objection" directly to the firm. Bank notification to the implementing agency came nine months after it had, erroneously, advised the consultant. 3.11 All DORINVEST personnel dealing with procurement were trained early in the project by international experts and received certification in procurement. Financial Management 3.12 The SAR appeared confident that DORINVEST was a capable agency. Yet, DORINVEST, unlike most other PIUs in Russian projects, did not have the authority to approve payments. Final review and approval was given by the top management of the implementing agency (the Federal Highway Department) to which DORINVEST belonged. As noted earlier, DORINVEST appeared to be well staffed in all disciplines, including accounting, until mid-1998, when the unit was gradually dismantled, and reduced to a skeleton by the time of project completion. Bank staff managing the project believed that DORINVEST remained very competent until mid-1999. 3.13 Project financial statements prepared by DORINVEST were audited by RoseExpertiza, which is considered a reputable Russian private auditing firm by the Bank's financial management team in Moscow and which worked for several Bank projects. 3.14 The Bank established a Financial Management Unit in its Moscow office in 1999. The FMU joined Bank supervision missions (which were combined with preparation of HRMP II) starting with the February 1999 mission. The unit's main interest at the time appears to have been the introduction by DORINVEST under HRMP II of the Bank's financial management model, LACI (Loan Administration Change Initiative). Feedback provided to the author of this assessment suggests that such a requirement created confusion about DORINVEST's diverse obligations under the two projects. 20. The PWC audit, engaged by the Bank's Europe and Central Asia Regional Office, covered the HRMP project, as well as the Second Oil Rehabilitation Project. 9 3.15 Cancellation of the routine maintenance subcomponent with a very large number of expected contracts eliminated a potential source of high fiduciary risk ($95 million cost). 3.16 Drawing on information from supervision, reports, the ICR states that annual audits of project expenditures were carried out in a timely manner. (The ICR further notes that annual audits were unqualified). The auditor, however, has been able to find only some of the project's audits and therefore cannot fully verify the ICR's claims. 3.17 Review ofproject expenditures. The project included 63 civil works contracts. Following an agreement during implementation, all these contracts, including those with costs below the threshold, were subject to prior review. (Information collected by the auditor for similar, large highway projects in India, China, and Brazil showed an average of 60-70 civil works contracts under prior review)." In these contracts, borrower's applications before disbursing loan funds were, in accordance with standard procedures, reviewed prior to disbursement to ensure that expenditures were authorized, eligible, and appropriately documented. The standard process to handle disbursement requests in Washington is a three-level intervention: initial review, verification, and approval. In addition, all applications under the HRMP (as well as for other projects) were prescreened by the Bank's Moscow office. 3.18 This audit's review of rejected disbursement applications, found that the HRMP had a 3.1 percent rejection rate (14 rejections out of 450 disbursement applications), which is less than half the Bankwide average of 8 percent. However, according to information from the Bank Controller, it appears to be high compared with other Bank projects in Russia. Allegations of Corruption 3.19 Documentation in the project files shows allegations of corruption, relating to the civil works contract mentioned above, that stemmed from high bid prices and an alleged conflict of interest between a contractor and the implementing agency. The case was referred to the Bank's Corruption and Fraud Investigations Unit which reviewed the allegations, and decided not to pursue the case. Stakeholders' Workshop 3.20 In accordance with the requirements of Intensive Learning ICRs, the ICR mission organized and carried out a stakeholders' workshop, which was attended by the OED assessment mission. The half-day workshop was attended by some 25 representatives from various government agencies, consultants, and contractors. The discussions were free and open, and valuable insights were gained about positive and negative aspects and lessons learned from the project. 3.21 The ICR provides a good synthesis of the workshop, and details the positive and negative comments by main category of participant: consultants, contractors, DORINVEST, and others (including the Federal Highway Service, other ministries, and the World Bank). Comments covered a wide range of project benefits and problems. Among the benefits, several participants mentioned the project's role in introducing competitive bidding, price adjustments, independent 21. Brazil-State Highway Management (1994/$133 million/63 contracts); India-National Highways 11 (1992/$120 million/32); China- Guangdong Provincial Transport (1992/$240 million/84), and Henan-Hebei Provincial Transport (1994/$380 million/77). 10 supervision, new technologies, better quality of works, financial audits, and training. Lack of interference from other ministries was highlighted as a plus. Among the negative comments, were overly tight deadlines for works, errors in engineering design, poor performance of contractors that came with a good reputation in other places, high contract prices and price adjustments, not enough competition from Russian contractors, Highway Design and Maintenance Model not properly adapted to Russian conditions, and inadequate analysis of design options and of views of the regional highway organizations at feasibility stage. 3.22 Overall, although several participants focused on the problems encountered during project implementation, there appeared to be consensus that the project had introduced important new knowledge and ideas. A representative of the PIU listed as main gains learning of the following: To communicate in a world language; to introduce or increase competition; to carry out supervision by consultants; to conduct yearly financial auditing; to understand economic concepts of road financing; to use new technologies (such as higher layer of bituminous concrete with additives, thermoplastic paint for road signs); to get new skills in contract management. Construction Costs 3.23 At the workshop, as well as during this auditor's interviews, some government officials claimed that the project had resulted in unit construction costs significantly higher than those obtained for road works executed with domestic financing outside the project. The ICR discusses this issue and notes that: * A recent study carried out by Russian consultants2 comparing unit prices for various work items under domestically financed and project-financed schemes was not conclusive, showing some unit prices higher under domestic financing and others higher under the project. * The large spread in prices for individual work items under both domestic financing and project financing illustrated by the study made comparison especially difficult. For example, there was a spread of more than six times between maximum and minimum values for sand layers under domestic financing and over five times under project financing for pavement milling. (The price comparison is further clouded by the large devaluation of the ruble in 1998)." * Technical design specifications under domestic financing were less demanding and contractors were not subject to independent supervision of works execution, making the end product lower in quality overall and therefore not directly comparable to project- financed works. 3.24 The assessment finds the ICR's points to be credible and substantially valid.24 22. Rosdornii, HRMP: Cost Analysis of Contract Works, Moscow 1999. 23. Evaluator's note. 24. Similar claims regarding construction costs have been made against international competitive bidding-generated prices in other countries during the initial introduction of ICB. Such claims usually suffered from the same sort of misunderstandings as those that occurred in the HRMP. For the assessment to have a fully independent view on this issue, a contract by contract review would have been required, including analysis of price adjustments as well as content of foreign inputs in the contracts. Such a review was not undertaken within the scope of this assessment in light of time and budget constraints. 11 4. Outcome Physical Objectives 4.1 The project's main physical objective, to slow deterioration of the road network, was partially achieved. About 760 kilometers of roadway was rehabilitated. This is about half the original target of 1,500 kilometers, although it is close to the revised target of 860 kilometers agreed at the midterm review. The shortfall was due to a combination of higher-than-expected bid prices and inflation-based price adjustments, to design weakness needing to carry out repairs for recently completed works, and to more thorough design and additional rehabilitation work in some road sections necessitated by rapid deterioration under increasingly heavy traffic. 4.2 The subcomponent for routine maintenance on 10,000 kilometers of road was also expected to contribute to the main physical objective. Although the elimination of this activity at the midterm was rational, it reduced the degree to which the objective was attained. The commitment at the midterm review by the Federal Highway Department to carry out these works with its own funds, noted in the ICR, cannot be verified because the roads were not identified, and no alternative way to monitor compliance was established. 4.3 Regarding the employment objective, the ICR notes that 6,600 employee-years were generated by the project, mostly in the private sector. The outcome shows that it was quite expensive to preserve employment in the rehabilitation of trunk roads: considering the capital expenditures under the project, the cost per generated employment was over $ 50,000. Many other activities, including the canceled routine maintenance works, would have generated many more jobs for the same money. Economic Return 4.4 The ICR estimates the actual return for the rehabilitation works at 74 percent. While substantially below the SAR forecast (over 100 percent), this is still a high return. In view of the very high return estimated in the SAR, it is not surprising that despite significant increases in investment costs, the return remains high. The key factor, besides investment costs, is traffic, since the rehabilitation's main benefit is to reduce the operating costs of vehicles. Traffic statistics are unreliable, and systematic traffic counts nonexistent. The decline in GDP of close to 30 percent between 1994 and 1999 suggests that transport demand nationwide decreased during this period. This would contradict the significant traffic increase (based on available 1993 and 1997 traffic data) on which the ICR's reestimate is based. Yet, project roads may have seen a significant increase in traffic due to growth in the informal sector omitted from GDP statistics; large increases in rail tariffs, combined with serious problems with quality of service, such as a high percentage of freight theft, that diverted a substantial amount of rail traffic to the roads; and location of many project roads, in and near Moscow - it appears that Moscow did experience dynamic growth during the 1990s. There are also specific factors for individual roads, such as opening of the Russia-Belarus border in 1997 that gave rise to large increase in truck traffic on the Ml road. Institutional Development Objectives 4.5 The project's five institutional development objectives were clearly a "stretch" for a first highway project in Russia, especially since several of them were divided into various 12 subactivities and goals. It was obvious that few immediate results could be obtained, but using the project to launch the discussion in those areas was an important step in itself. 4.6 The introduction of international competitive bidding through project-financed works was, despite the initial problems, a significant achievement. Its later adoption as a procurement method for domestically financed civil works in Russia will likely have a great impact on further improving the construction industry, lowering prices of road works and reducing the potential for corruption.25 The introduction of independent supervision and an "engineer" (who acts on behalf of the client in the management of the contract) were also significant. The engineer's role was more restrictive than in international practice, given the active role of the Federal Highway Department in contract decisions, but such control was probably inevitable given the need for the highway agency, as the employer, to develop trust in the system. 4.7 Competition brought with it opportunities for local contractors and consultants, and, coupled with advice from international advisers in DORINVEST and several joint-ventures, helped strengthen these two industries, through the introduction of new technologies and modem management. A high advance payment of 30 percent established for civil works contracts under this project helped local contractors equip themselves with modern equipment for use in project- financed works, and will allow these contractors to better compete for future works. 4.8 Overall, the cost of training, studies, and technical assistance was significantly less than anticipated in the SAR - $16.0 million was spent, slightly over half the SAR estimate of $28.3 million. Part of the shortfall stems from little use of external training courses in areas such as road asset management and safety, which could have been of much value to Russian highway professionals. Project-financed training was supplemented by Economic Development Institute- provided training.26 By all accounts, the training was good, and helped train managerial and senior-level professionals, but the view of Russian officials interviewed," and of the auditor, is that the program was too short, and too few courses were offered." The twinning of the Federal Highway Department with U.S. Federal Highway Administration appears to have been instrumental in helping train highway department personnel and improve road design practices.29 4.9 The project intended to improve the efficiency of public expenditure on roads through improvements in management, planning, and selected technical areas. The outcome for this objective, which included multiple activities, can best be summarized as raising awareness of the issues. A study on regional road maintenance was discussed with the Federal Highway Department, but no change was made as a result. A study on the pavement management system (PMS) was not carried out, but related discussions helped develop a database and software, a first step in implementing a PMS. The Bank's Highway Design and Maintenance Model (HDM) was used for assessing maintenance improvements under the project, and, with improved calibration and adaptation to Russian conditions, will be useful for Russian road planners. 25. Levin and Satarov (2000) note, "under the Soviet regime, the main cause of corruption was state control over the distribution of basic resources. At the first stages of reforms, officials radically diversified their forms of control. They took control over the assignment of credits, privileges, licenses, winners in privatization contests, choice of authorized banks, choice ofagents for implementation ofmajor social projects etc." [emphasis added]. 26. The Economic Development Institute of the World Bank changed its name to World Bank Institute in 1999. 27. In particular, those of the Vice-Rector of the Transport Technical University in Moscow, who was closely involved with the EDI program. 28. By comparison, the EDI transport training program for China launched soon after China joined the Bank, lasted close to 10 years and comprised a multitude of courses both for officials and for trainers. 29. The first director ofDORINVEST has noted an important contribution made by advisers from the Finnish Highway Administration (Annex B). 13 4.10 During the project period, a number of oblasts separated highway administration from execution of maintenance. This reform is an important step toward the modernization of highway management. The project may have encouraged such reforms, since a project condition about the engagement of local contractors required that they be independent legal entities not receiving government subsidies. 4.11 Road financing. The outcome of the efforts to improve road financing is more complex. Until late 1999, it appeared that some progress had been made. The conclusions of the Road Financing Study (Phase II), carried out under HRMP, had been included in the new draft Tax Code submitted by the government to the Duma, thus incorporating significant improvements to the current Russian road financing system (e.g., full cost recovery of attributable costs from heavy trucks, gradual elimination of the enterprise tax). On the other hand, the project made no progress in its aim to develop a proper accounting system for the federal road fund. The Bank was provided an audit covering only the part of the Road Funds that was use for road financing, was under the HRMP II to satisfy an effectiveness condition of the follow-on loan.o At the same time, the third Structural Adjustment Loan, approved in mid-1998, included a commitment for an audit of the road fund by an international auditing firm; this project closed ahead of schedule and the audit was not carried out. 4.12 In mid-2000, the government eliminated the federal as well as the regional road funds. Misuse of funds, whether real or perceived, appeared to be the main reason, according to President Vladimir Putin's strong words about the management of the fund." The Bank appeared to have supported if not lobbied for this decision, since lack of financial controls of these funds provided a strong potential for misinvestments and fraud. Furthermore, the regional road funds, with revenues stemming mainly from the enterprise tax, were a source of significant distortion in the management of fiscal resources, at a time when both the Bank and the IMF were encouraging budget consolidation. 4.13 In the auditor's view, the discussions during project implementation on both the structure of road user charges and the need to rationalize the sources of funding of the road funds are likely be useful for future government action on this matter. In the long run, it may be possible to re- create the federal road fund under strict financial controls. A road fund in Russia, managed on a commercial basis, would "bring roads into the market place, put them on a fee-for-service basis, and manage them like a business."" Such a commercial approach would require road users' involvement in the management of the fund, which would provide greater assurances that fund resources are well used. 4.14 Transport sector framework and Ministry of Transportation restructuring. This component failed. While the study that was to be the basis for this component was being carried out, the ministry found the work by consultants unsatisfactory and the component ultimately was canceled with no satisfactory output. It appears that from project preparation and appraisal to 30. Russia - HRMPII. Project Completion Note. May 25, 2000, prepared by the Bank's Europe and Central Asia Regional Office. 31. According to the Moscow Times of July 25, 2000, President Putin had said, "I am sure that if these $5 billion [a sum he earlier had mentioned as being the road fund's annual revenues] were really spent on building roads, we would now have only highways and autobahns in Russia." The same article noted that a spokeswoman for the highway agency defended the fund: "The fund underwent very strict audits by the Audit Chamber and the presidential administration.... Some of the checks showed the money could have been used more efficiently, but it was all used for its stated purposes." 32. Memorandum of December 8, 1996, from Ian Heggie, the Bank's Principal Roads Adviser, to Stanley Fisher, Senior Managing Director, IMF. 14 implementation, the ministry, which underwent major reform when the highway sector was taken out of its purview, radically changed its commitment to this component, and apparently no significant effort was made to come to an agreement with the consultant that would have steered the study in a direction more useful for the client. 5. Ratings Outcome, Institutional Development, and Sustainability 5.1 The OED assessment rating for outcome is determined by three dimensions: * Relevance: While helping improve the road system and its management was important for Russia, the project's relevance was diminished by the overextended objectives and other design issues noted in section two. * Efficiency: Although substantially below SAR estimates because of higher than expected investment costs, the ex-post economic rate of return was still high (para.44) but there is no clearcut evidence that the scaled down project, with its much higher than expected unit costs, was still the least cost alternative." Efficiency was also low, in the pursuit of institutional objectives, given the limited efficacy and impact of the project regarding those objectives. * Efficacy: Road rehabilitation was done for 760 kam, representing one half of the SAR target and 88 percent of the target agreed at the MTR. Regarding routine maintenance, it is impossible to verify whether this task, or part of it, was carried out by the Federal Highway Department with its own funds as agreed at the MTR (para 4.2). Tasks included in the draft Action Plan were only partially done. In conclusion, efficacy is considered modest. On balance, outcome is rated moderately unsatisfactory. 5.2 Institutional development impact: The introduction of competitive bidding and independent supervision in the road sector, initiated by the project and later extended to domestically financed works, and launching development of a domestic construction and consulting industry were major achievements. The organizational changes that separated planning from execution, the introduction of project economic analysis, and the improved financial auditing (noted in the workshop) were also valuable._Yet, the project failed to make any progress regarding establishment of a new framework for the transport sector and restructuring the transport ministry. Overall, the institutional objectives were only partially achieved. Therefore, institutional development impact is rated modest. 34 5.3 Sustainability: The recent dissolution of the road fund and the transfer of road maintenance funding to the budget introduces an element of uncertainty regarding the upkeep of 33 The Region argues that the HDM model provides the least cost designs on a system-wide basis. This statement is correct, but the issue is whether with the much higher than expected costs the designs remained the least cost. 34. The first Director of DORINVEST states that the rating of institutional development impact is unfairly low, given improvements brought about by the project in road planning and financing, professional development of contractors, introduction of competitive bidding and market-based pricing, and other improvements mentioned in his letter (Annex B). 15 the project-financed road improvements. The fact that the project-financed roads are all of major national and international importance, are near to and connect to Moscow, may play a positive role in the allocation of maintenance funds for these roads, but it is too early to pass a well- founded judgment. On the institutional side, the key achievements, those related to new procurement and contracting approaches, have been adopted by law and are in effect for budget- financed contracts. However, the ICR notes that "there are signs that the RFHS is not always strictly following the guidelines [regarding competitive bidding] it has issued". In view of these uncertainties, sustainability is rated as "non-evaluable". Bank and Borrower Performance 5.4 Bank performance: Launching a large, first project in the road sector for the Russian Federation, under tight preparation, appraisal, and implementation schedules and evolving political and institutional frameworks was a challenge. There were problems with project design, noted earlier in this assessment. The midterm review, by canceling the poorly conceived routine maintenance component, corrected the main problem. The SAR is not clear in its assessment of project costs. SAR discussion of other topics suggests that cost estimates were based on international prices rather than on the highly distorted local prices. While this approach was probably wise, this critical subject deserved a more detailed analysis at the appraisal stage. 5.5 Bank performance on the critical issue of road funding was not always well focused. While the project addressed cost recovery and aimed to improve accounting and auditing of the road funds, it failed to identify from the beginning the critical issue that enterprises' turnover tax was the main revenue source of the regional funds. In the final stages of implementation, Bank management encouraged dissolution of all road funds, rather than attempt to salvage via improved financial controls the correctly funded Federal Road Fund. 5.6 During project preparation and throughout implementation, Bank staff built credibility and developed excellent relations with the borrower and implementing agency. Supervision missions followed implementation closely with the implementing agency, and maintained contacts with key stakeholders, including central ministries, universities, other donors, and Duma representatives. A major weakness of the supervision effort was failure to comply with standard Bank procedures following the MTR decision to significantly reduce the project scope. The project should have been formally restructured and loan documents amended to reflect the revised project description." At the MTR, the Bank failed to seek appropriate assurances that the Federal Highway Department would carry out the routine maintenance component. The support provided by EDI to train senior officials was good - although probably too limited. Overall, Bank performance was mixed, and is rated unsatisfactory. 5.7 Borrower performance. The highway sector suffered continuous reorganization that complicated project execution and supervision. In part, this problem was offset by the good job DORINVEST did for the first two thirds of project implementation. This changed later, however, when the PIU was downsized, changed directors five times, and lost most of its qualified 35. According to OD13.05, it would have been up to the Regional Vice President, in agreement with the Legal Department, to decide whether the amendment was major and required approval by the Board on a non-objection basis. Project files do not record whether the project team alerted the country management about the change of scope agreed at the MTR, or whether any member of the country team actually read the supervision report and the Aide Memoire reporting on MTR agreements. The first CPPR (March 1997) after the MTR makes no reference to the substantial reduction in the project's physical targets and rates the project's development objectives as highly satisfactory, despite the reduction in targets. The Region notes that the mid-term review was discussed with the Country Director and his management team. 16 personnel. Thanks to the federal road fund, the anticipated risk by the SAR about provision of counterpart funds did not materialize. 5.8 The borrower was weak on the institutional side. The government lost ownership of the Ministry of Transportation component, leading to the failure of this component." The non- procurement of traffic counters will significantly hinder efforts to improve road planning. The Borrower made no progress in developing a proper accounting system for the Road Fund, which would have greatly improved the Fund's transparency and likely avoided its demise. The Federal Highway Department provided no evidence that it was carrying out the routine maintenance program it had committed to execute at the mid-term review. 5.9 During project execution there appear to have been continued internal tensions between the Ministry of Finance and Federal Highway Department, on two aspects: (i) loan repayment, with the ministry insisting, and highway department opposing, the use of road fund revenues for the purpose, and (ii) contract prices, with the ministry questioning variation orders and inflation adjustments agreed to by DORINVEST and the highway department. It appears that the FHD did not promptly informed the MOF about the important decisions taken at the mid-term review. On balance, while acknowledging the mostly good performance of the implementing agency, overall borrower performance is rated unsatisfactory. 6. Lessons 6.10 Project design and long-term Bank-borrower relationship. A wide-scope project design for a first Bank operation in a sector can only be understood on the assumption that the operation will launch initiatives that will be continued in follow-on projects. If that assumption does not hold, the Bank and the borrower (i) did not design the most effective project, and (ii) missed an opportunity to build on the policy and management progress made under the project. Cancellation (prior to effectiveness) of HMRPII and lack of other follow-on lending operation appear to confirm the latter point. 6.11 Effectiveness and convenience ofPIUs. While PIUs often cut across institutional boundaries and can become privileged units that undermine institutional morale, DORINVEST demonstrated that PIUs can be an effective and advisable way to manage a project when the unit is well integrated into the parent organization, does not have special privileges, and incorporates international advisers. It also demonstrated the benefits of a PIU for a first project in a sector. The HRMP also showed that PlUs may be essential in transition counties, especially if key sectoral agencies are being, or are likely to be, restructured. 6.12 Flexibility in project design under high uncertainty. Projects in transition economies, particularly in countries or sectors in the early stages of such transition, are bound to suffer from major uncertainties affecting project costs and scope, as well as ownership of institutional objectives and components. The use of a phased approach starting with pilot components is a good concept, but achieving maximum effectiveness requires an early assessment of the results of the pilots, and the possibility to redesign the project if experience warrants. Use of the newer loan models that allow learning, innovation, and phasing an intervention should in the future be given serious consideration at the outset of project design. 36. The Region argues that MOT did not lose ownership but preferred to carry out the review using its own expertise. This clearly not what was expected at appraisal. 17 6.13 Demonstration value of road works for probity and efficiency. Road works can be a credible vehicle to instill integrity and efficiency in use of public funds in transition economies through adoption of a combination of approaches and techniques including competitive bidding, feasibility studies, resource allocation optimization models (HDM), audits, and separating administration from execution of the works. To increase their demonstration value, these techniques and approaches should be disseminated as much as possible through training programs and through making key information available to policymakers, senior government professionals, and academia. 6.14 Road funds - conditions for survival. The Russian federal and regional road funds, while providing a useful service as road maintenance funding instruments, confirmed experience elsewhere that road funds need to meet essential conditions to ensure their rationale, public trust, and, ultimately, their survival. Two such conditions exemplified by the Russian experience were missing and eventually led to the demise of the funds: (i) revenues feeding the funds should be user charges, not general taxes (a significant portion of the regional funds consisted of taxes on enterprises' turnover, which is unrelated to road use), and (ii) a credible system of accounting and auditing the road funds should be in place. Such a system was lacking in both the federal and regional road funds.  19 Annex A Basic Data Sheet HIGHWAY REHABILITATION AND MAINTENANCE PROJECT (LOAN 3706-RU) Key Project Data (amounts in US$ million) Appraisal Actual or Estimate Current estimate Total project costs 340.00 347.90 Loan amount 300.00 280.79 Cancellation 19.21 Date physical components completed 12/31/97 12/31/99 Economic rate of return Over 100 % 74 % Cumulative Estimated and Actual Disbursements FY94 FY95 FY96 FY97 FY98 FY99 FY00 Appraisal estimate (US$ thousands) 3 37 106 184 254 294 300 Actual (US$ thousands) 0 8 78 155 254 262 283 Actual as % of estimate 0 32 74 85 100 89 94 Date of final disbursement: August 30, 2000 Project Dates Steps in project cycle Date Planned Date Actual Identification 7/23/92 7/23/92 Appraisal 11/26/92 5/23/93 Negotiations 2/22/93 10/12/93 Board Presentation 3/30/93 2/17/94 Signing 5/1993 6/22/94 Effectiveness 6/1993 10/21/94 Midterm review 6/30/96 6/19/96 Project Completion 12/31/97 12/31/99 Loan Closing 6/30/98 12/31/99 Staff Inputs (staff weeks) Stage of project cycle Actual Weeks US$ Identification/Preparation 166.7 492.8 Appraisal/Negotiation 80.5 247.3 Supervision 289.5 858.7 ICR 25.0 75.0 TOTAL 561.7 1673.8 20 Mission Data Stage of project cycle No. of Persons and Specialty (e.g., 2 Economists, I FMS Performance Rating etc.) Month/Year Count Specialty Implementation Development Progress Objective Identification/Preparation 2/92 n.a. HE, HE 7/92 5 HE, TE, CI, C 3/93 10 HE, TE, PS, C AppraisalNegotiation 6/93 13 HE,FA,TE,PS,CI,ES, LC 12/92 3 HE,FA,RA Supervision 3/94 6 HE,PS,TE,CI,RA,C S S 5/94 3 HE,TE,RA S S 7/94 8 HE,PS,TE,RA,CI,C,E HS S C 11/94 5 HE,PE,RA S S 6/95 5 HE,PS,TE,RA,LC S S 10/95 4 HE,PS,RA,CI S HS 4/96 3 HE, RA, CI S HS 6/96 6 HE,RE,PS,CI,TE,IE S HS 9/96 7 HE,PS,TE,CI,IE S HS 2/97 5 HE,CI,IEPS S HS 6/97 7 HE,CI,PS,TE,IE S HS 9/97 7 HE,CI,PS,TE,IE S HS 3/98 4 HE,PS,IE S S 6/98 3 HE,CI,IE S S 11/98 6 HE,CI,ES,IE S S 3/99 4 HE,ES S S 9/99 4 HE,ES,PS U S 12/99 2 HEJE U S 1/99 3 HE,PS U S Abbreviations: HE = Highway Engineer, TE = Transport Economist, PS = Procurement Specialist, IE = Infrastructure Expert, Cl = Construction Industry Specialist, ES = Environment Specialist, RA = Research Analyst, LC = Legal Counsel, C = Consultant 21 Annex B Borrower Comments Comments received from Mr. M. Pokataev, formerly Director of DORINVEST. Mr. A. Barbu Sector and Thematic Evaluation Team Leader Operations Evaluation Department World Bank Re: Draft Project Performance Assessment Report for the Highway Rehabilitation and Maintenance Project (Loan 3706-RU) Dear Mr. Barbu, Thank you for the opportunity to review and comment on the draft Performance Assessment Report (PAR) for the Highway Rehabilitation and Maintenance Project (HRMP). On the whole, the PAR gives a rather objective and very professional evaluation of the substance and outcome of the unusual and extremely interesting work performed by Russian specialists in a close collaboration with the Bank staff and experts in 1993-1998. As to the contents and assessments formulated in various sections of the report, I would like to make the following comments: 1. General I cannot agree with the statement that the management of the Russian road sector was not familiar with the Bank's rules and procedures. At the Project start-up phase, i.e. in May 1993, Mr. A.P. Nasonov (who was appointed the First Deputy Head of the Federal Highway Department soon thereafter), Mr. V.I. Sokolov (First Deputy Chairman, Leningrad Road Committee), and a number of FHD officials, including myself as a potential Dorinvest Director, took a short-term, but very efficient training course at the EDI. An in-depth, independent study of the Bank's documents by Mr. O.V. Skvortsov (First Deputy Head, FHD) allowed him to conduct a professional dialog on all Project-related issues with the Bank's key staff at all Project implementation stages. To my mind, even such short-term training of the Russian road sector managers facilitated a dynamic Project development in 1993-1997. At the initial stage of the Project, it might have been useful to have a more detailed study of experience generated by Hungary, which had implemented a similar project under conditions of economy in transit. In 1992, the Russian highway authorities had no intention to introduce commercial principles in highway construction and maintenance: they were only trying to pilot toll roads and contractual road maintenance. 2. Project Objectives and Design In the retrospect, not all Project objectives seem to be the top priorities. Objectively, the following activities proved to be the most topical and efficient: reforming contractor attitudes; a rapid assimilation of the international best practices by Russian road designers and contractors within the framework of cooperation with foreign companies; introduction of competitive bidding and market-based pricing; and establishment of an efficient contract management system. It may be worth noting that in spite of the lack of instructions from the Russian road sector management, all these innovations are now intensively used by most regional and local highway authorities in Russia. A 'Big Jump' in that part of the Project seems more reasonable than a 'series of consistent 22 steps', which had a negative impact on disbursement efficiency under a number of other investment projects financed by the Bank in Russia. While highly appreciating the Project benefits derived from cooperation with the US Federal Highway Administration, I still find it necessary to note an extremely useful contribution by the Project advisers from the Finnish Highway Administration who provided assistance with Dorinvest staff training and organization of workshops and study tours on bidding procedures for regional highway authorities and contract management and supervision units of Russian consulting firms at all Project implementation stages. 3. Project Implementation When assessing whether it was appropriate to establish Dorinvest as a sector-level PIU, I hold it necessary to emphasize the advantages of such PIU over a specialized commercial structure (e.g., Geningconsult) contracted for project management, which is common practice under many other projects. During the Project implementation period, Dorinvest organized theoretical and hands-on training in bidding practices, contract management and supervision for more than one hundred specialists from Russian regional highway authorities who are successfully using that knowledge in their work. From the fiduciary viewpoint, it should be noted that, with regard to the preparation of short lists for consulting services and contract supervision, the Bank's consultants were extremely inflexible insisting on the reduction of the lists to 4 bidders and on the leadership of foreign companies in joint ventures for consulting services. That largely accounted for weaknesses in technical projects. 4. Outcomes I cannot comment on that issue since I was not involved in the Project completion activities after September 1998 and do not have detailed information. 5. Rating To my mind, it is not quite fair to give a low rating to the institutional development efficiency. Large-scale improvements in road planning and financing, professional development of Russian contractors, assimilation of the international best practices by Russian road consultants and contractors, introduction of competitive bidding and market-based pricing, and establishment of an efficient contract management system outweigh the fact that numerous tasks formulated at the Project preparation stage were not completed. 6. Lessons Learned I cannot comment on that since I was not involved in the Project completion activities after September 1998. I would like to apologize for the delayed comments, which is due to the fact that I could start working on the PAR only when I got the translated version and also due to a very tight travel schedule. Sincerely yours, M. Pokataev Head, Main Road Construction Department ZAO LUKoil-Neftegazstroy