ICRR 12990 Report Number : ICRR12990 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/10/2008 PROJ ID : P008499 Appraisal Actual Project Name : Road Transport US$M ): Project Costs (US$M): 141.0 129.8 Restructuring Project Country : Kazakhstan Loan/ US$M ): Loan /Credit (US$M): 100.0 95.6 Sector Board : TR US$M): Cofinancing (US$M ): Sector (s): Roads and highways (97%) Central government administration (3%) Theme (s): Infrastructure services for private sector development (40% - P) Access to urban services and housing (20% - S) Rural services and infrastructure (20% - S) Regulation and competition policy (20% - S) L/C Number : L4437 Board Approval Date : 02/09/1999 Partners involved : Closing Date : 12/31/2004 12/31/2007 Evaluator : Panel Reviewer : Group Manager : Group : Ramachandra Jammi Roy Gilbert Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: To achieve more efficient road and road transport sub -sectors in Kazakhstan. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): (i) Rehabilitation of Priority Sections of National Roads – (Planned: US$113.4M; Actual: US$118.8M), including: (a) Civil works for the rehabilitation of several priority sections, including bridges and limited improvements in road geometry, of the national road network, totaling about 500 km, located along the country's principal North -South transport corridor, the Almaty-Karaganda-Astana road. (b) Consultant services for supervision of the road works, training, technical assistance to the Bureau of Investment Projects (BIP) and those engineering and procurement services for the Project not funded by either the Project Preparatory Facility (PPF) advance or from local resources . (ii) Strengthening of Routine Road Maintenance – (Planned: US$5.8M; Actual: US$4.4M) To improve maintenance capability of Department of Roads (DOR) and the (corporatized) Oblast road maintenance companies on the Almaty-Astana road. Contingent on the establishment of maintenance planning and implementation processes satisfactory to the Bank, the component would provide equipment and spare parts to complement training provided under the on-going Asian Development Bank (ADB) project. (iii) Institutional Strengthening of the Department of Roads and Development of the Kazakhstan Road Maintenance and Construction Industry – (Planned: US$1.7M; Actual: US$0.0M), including: (a) establishment of planning, budgeting and monitoring procedures for the national and regional road network, based on the assessment of road condition and economic analysis of alternative solutions, as well as the training of staff for the preparation of budgets; (b) establishment of procedures for the supervision of road works carried out under contracts on national and regional roads, and training of staff; (c) preparation of training programs for Kazakhstan contractors in bid preparation, contract administration, cost control, management, and quality control; and (d) strengthening of DOR's training capability in the above three areas . (iv) Improvement in Road Transport Policies and Regulations and in their Implementation (Planned: US$0.7M; Actual: US$0.0M) including: (a) development of laws and regulations pertinent to road transport; (b) strengthening of enforcement agencies and improvement of public information; (c) improvement of road data systems; and (d) development of a competitive market for the provision of inter -city and rural passenger transport services, including the development of independent bus stations . The component also included provision of consultancy services and training. (v) Improvement in Road Safety – (Planned: US$1.0M; Actual: US$0.0M), including: (a) setting up of a temporary road safety task force and the preparation of a preliminary National Road Safety Action Plan and Local Safety Plans with monitorable objectives; (b) improvements in road accident data for identifying black spots; (c) development of road safety research in Kazakhstan, including liaisons between Kazakhstan institutes and international road safety institutes; (d) improved enforcement of road safety rules and regulations, including the provision of appropriate equipment; (e) development of a better understanding of road safety problems; and (f) review of the existing emergency services. The component included the provision of consultancy services, training, study tours, and equipment. (vi) Project Management – (Planned: US$1.2M; Actual: US$0.3M) for the incremental project management costs, including office space, equipment and supplies, auditing services, training, communication, and travel . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost, Financing, Borrower Contribution : Borrower contribution was US$36.8M, around 3% more that US$35.7M at appraisal. Cost increases in civil construction (due to increase in bitumen prices and re -design of some road sections to accommodate heavy loads ) were compensated by the cancellation of one contract on the Astana-Karaganda segment, which is being funded outside of the project by the Government . Dates: The loan closing was extended two times for a total of three years to allow the completion of institutional components (iii-v in section c)and to accommodate the re-design of several road segments due to their accelerated deterioration. There were delays in the execution of most contracts, mainly due to inadequate equipment and inexperienced contractors . Also, delays were experienced in the procurement of equipment for routine maintenance, until satisfactory planning and implementation processes were established . 3. Relevance of Objectives & Design: The project development objective was consistent with the Bank ’s Assistance/Partnership Strategy (2001, 2004) and Kazakhstan’s transport development strategy in terms of supporting private sector -led growth and improving the competitiveness of the economy in general . The relevance of objectives is high . In the Bank's transport projects during the 1990s, there was a general emphasis on several institutional features including governance frameworks, cost recovery, competitive procurement and private sector development, (environmental) quality assurance of works and road safety . Most of these features are reflected in the various components of this project. In retrospect, the breadth of institutional components proved ambitious in a situation where there was not even a rudimentary transport policy, a medium term plan, a road condition data register, reasonable capacity and equipment for data collection, or use of quantitative information in programming of projects . Due to existing technical capacity and experience within the country, the project was on stronger ground with road works and adoption of contract -based routine maintenance procedures, cost control, and quality assurance . The design correctly anticipated the Government ’s unwillingness to adopt new laws, regulations and enforcement mechanisms, and the lack of cooperation between different ministries . Despite the Government's expression of commitment to the project, the Bank recognized potential risks relating to insufficient counterpart funding; institutional inertia in putting newly learned concepts and capabilities to work; disruptive reorganization within Government; and insufficient use of technical assistance . Relevance of design is rated substantial . 4. Achievement of Objectives (Efficacy): The overall objective of achieving more efficient road and road transport sectors in Kazakhstan was substantially achieved through rehabilitation of priority sections of national roads, and institutional and policy actions . Rehabilitation works were completed on the Almaty -Karaganda segment (468 km out of the planned 500 km) of the Almaty-Astana National Road with the exception of one segment between Almaty and Gulshad, which is being repaired. Traffic on the road has increased (by 140% in terms of all vehicles in the Almaty Region and 180% in the Karaganda region) and serves increased trade flows in the corridor . An routine road maintenance plan was developed for the Almaty -Astana road. A corporatized state owned enterprise working for the Road Committee (RC) under contract was created for routine maintenance, separate from the client organization. The government's road budget increased from US$ 165M in 2002 to US$1120M in 2007. Belatedly, a partially automated road data collection system was set up as part of the agreed technical assistance program. Three studies were completed on the following : planning, budgeting and monitoring road maintenance; policies and regulations for road transport; guidelines for road safety management . The outcomes of these studies are not clear. Between 2003 and 2005, eighty four “normative documents� and standards were revised and, where appropriate, harmonized according to international requirements . 5. Efficiency (not applicable to DPLs): An ex post cost-benefit analysis (CBA) conducted for the rehabilitation of the road sections between Almaty and Karaganda using the Highway Development and Management Model (HDM-4), yielded a Net Present Value (NPV) of US$269.1 million, and an Economic Rate of Return (ERR) of 290 percent. However, this ERR estimate is sensitive to the assumptions made for increases in passenger and freight traffic, which at 10% per annum for the next 15 years may be on the higher side . Taking into account the cost overrun of 50% and the time overrun of 3 years, overall efficiency is rated substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 32% 90% ICR estimate Yes 290% 90% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The objective of improving the efficiency of road transport was served by the rehabilitation of the Almaty -Astana national road which helped lower the cost of transport by about 20%, while cost of transport to users decreased by about 40% after factoring in savings in time and money . The travel time in northern section of the road decreased by 30% and in the southern section by 22%. The capacity of the Kazakhstan road administration and local contractors was strengthened, and procurement and financial management functions were improved . Notable progress was made in controlling the overloading of heavy vehicles which may have helped decrease wear and tear of the road . However, outcomes of the revised road transport policies and regulations are less clear . Based on substantial relevance, efficacy and efficiency, outcome is rated satisfactory . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The project’s main component relating to rehabilitation of the Almaty -Astana road and the benefits that it generates are not subject to any significant risk . Traffic levels are increasing, and the road continues to be in good condition, and maintenance funding has been increased . There is proactive control of overloading and good engineering surveillance of road conditions, which forms the basis for maintenance recommendations . Efforts are on to develop sources of low-sulphur bitumen (to substitute for the currently used bitumen which has a high sulphur and paraffin content) to improve maintenance and increase pavement life . There is a risk that traffic safety will deteriorate unless steps are taken to remove selected black spots, maintain enforcement of traffic rules, and increase awareness about traffic safety . On the institutional side the organization structure and separation of administrator / manager and supplier functions appear sustainable . Improvements in staff skills resulting from the institutionalized regional training programs also appear to be sustainable in the road sector entities even if there is significant staff turnover . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: While the quality of entry is rated substantial, in retrospect, the institutional targets proved to be too ambitious for a first road project in Kazakhstan . While the emphasis on institutional issues in transport projects was a common feature in transport projects at the time of project preparation, a prudent course in the country context would have been to have a more modest institutional development program which might be more effective in the long-run. Supervision was regular and frequent, and kept up high standards in an uncertain governance environment, as for instance in insisting on a satisfactory routine maintenance plan before permitting the procurement of modern equipment. After the government decided to fund the technical assistance components on its own, the Bank insisted on approving the terms of reference and supervising the consultancy work . Finally, missions paid attention to financial management and procurement and corrective actions were taken to provide for learning in this regard. at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: The Government showed commitment throughout the project, especially for the civil works components . The Government succeeded in separating the road maintenance and construction industry from road administration and either corporatized or privatized it, and resolved that all periodic maintenance and rehabilitation works would be awarded competitively. Funding for road sector maintenance was increased as agreed during preparation and was satisfactory in general . However, the Government did not increase road user charges to cover 100% of the cost of road use as per the relevant loan covenant, but provided funds through its budget . The Government resisted funding for some of the institutional components . especially where foreign consultants were to be involved, though arrangements were made to complete the tasks using local expertise . The implementing agency was effective from the start of the project with regard to the road rehabilitation works, and was persistent in dealing with defects that remained after completion . On the institutional side, the training program to supervise road works and improve contractor capacity was carried out swiftly, and terms of references were prepared for the Bank’s approval for the remaining technical assistance activities . However, the implementing agency took time in coming up with an acceptable maintenance plan for the project roads . This resulted in a late start for procuring the maintenance equipment . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Design: The project specified five development outcome /impact indicators, of which those related to institutional matters (ii to iv) were not readily measurable: (i) lower cost of road transport on Kazakhstan's main north -south national (Almaty-Astana) road; (ii) increased capability of the Roads Committee to plan, budget, implement, and monitor road expenditures; (iii) increased capability of the Kazakhstan road construction and maintenance industry; (iv) increased efficiency of Kazakhstan's road transport sector; and (v) reduced number and seriousness of accidents on the Almaty-Karaganda section of the Almaty -Astana road. Institutional indicators could have been more clearly specified in the M&E design (e.g. increased certification of contractors; number of responsive bids etc .). In gathering data on the physical condition of roads, the project could have promoted technically more advanced measuring techniques to replace the current visual road inspection methods . Implementation: M&E was implemented to a limited extent through monthly progress reports on project activities prepared by supervision teams complemented by reports provided by the Road Committee (RC). No data appears to have been collected for outcome indicators ii to iv mentioned under M&E "Design" above. Strengthening of institutional capacity was measured to the extent of the number of Oblast staff trained . Data on accident rates was collected from traffic counts and reports issued by the Traffic Police . Utilization: The outcome indicator on road user costs helped track the extent to which rehabilitation of the Astana-Almaty corridor had improved service, and the compilation of accident rates showed the extent to which safety was being improved. There is little evidence of how this and other output -related information was fed back into improving project implementation. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards: This was a Category B project. There was no land acquisition involved, and the Project's main negative impacts were limited to the road works (from construction equipment, asphalt mixing plants, quarries, erosion and run-offs, traffic safety at construction sites, etc .). An Environmental Management Plan (EMP) was prepared and included in the contract documents as part of the final engineering . Contractors were required to employ environmentally sound work methods and equipment, all of which were monitored by the supervision engineers. Overall, the project had positive impact on the environment through improved road condition resulting in less wear and tear on the vehicles, less pollution and less dust . Fiduciary compliance. The project maintained a financial management system that was acceptable to the Bank and provided reasonable assurances that the proceeds of the loan were used for the purposes for which the loan was granted. One issue was the overpayment of $ 400,000 under a contract due to an incorrect application of an exchange rate. This was eventually identified and resolved, but it was agreed that the implementing entity responsible for the overpayment had not met the financial management obligations specified in the Loan Agreement . Although most procurement of road works and consultants went smoothly, some issues required the Bank's attention. These included lapses in having variation orders approved by the Bank, delays for signing the contract for remedial works, and unauthorized payments for incidental services . The Bank requested changes in procurement practice by designating a senior implementing agency staff to be responsible for procurement and, when necessary, consult senior management for advice . Otherwise, the Government broadly complied with audit requirements . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Moderately Project design was ambitious with Satisfactory respect to the institutional component Borrower Performance : Satisfactory Moderately Lack of compliance with covenant on Satisfactory road user charges; Delay in preparing maintenance plan and launching consulting assignments Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: 1. Institutional reform should be designed with careful attention to the specific endowment, culture and needs of the existing organizational structures and processes, rather than attempting to fit a more general and prescriptive model. 2. Where local expertise is of a high order, there is likely to be resistance to the use of international consultants, whose services should then be used judiciously to fill genuine gaps in capacity . 14. Assessment Recommended? Yes No Why? The outcomes of institutional interventions, some of which were funded by the government are yet to be fully understood. It will be instructive to review progress in this regard with more tangible evidence, which may hold useful lessons for follow-up projects as well as for similar projects in other client countries . 15. Comments on Quality of ICR: The ICR is written in a clear, analytical and objective manner and makes good use of the information that is available, but could have been more critical in deriving useful lessons form the project experience . Overall, the ICR quality is rated satisfactory. a.Quality of ICR Rating : Satisfactory