ICRR 12855 Report Number : ICRR12855 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/29/2008 PROJ ID : P045052 Appraisal Actual Project Name : Road Maintenance US$M ): Project Costs (US$M): 65.90 69.74 And Development Country : Nepal Loan /Credit (US$M): Loan/ US$M ): 54.50 53.47 Sector Board : TR US$M): Cofinancing (US$M ): Sector (s): Roads and highways (55%) Central government administration (30%) Other social services (14%) Law and justice (1%) Theme (s): Rural services and infrastructure (50% - P) Rural markets (25% - S) Municipal governance and institution building (25% - S) L/C Number : C3293 Board Approval Date : 11/23/1999 Partners involved : Closing Date : 12/31/2004 06/30/2007 Evaluator : Panel Reviewer : Group Manager : Group : Antti P. Talvitie Ridley Nelson Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: The original PDO, as written in the PAD, had five objectives : 1. Achieve sustainable maintenance, rehabilitation, and construction of economically justifiable roads in SRN within the PIP (priority investment plan) framework; 2. Improve access to district headquarters not currently served by road, and reduce vehicle transport costs, and delays in project areas; 3. Promote more sustainable funding and more efficient public sector management of road maintenance; 4. Adopt and disseminate environmentally sustainable road construction and maintenance practices; and 5. Generate rural employment through adoption of labor-based technologies and long-term employment for the additional generated economic activity . The DCA has different wording for several of the PDOs : (i) Strengthening the road sector planning by building upon the Priority Investment Plan framework; (ii) Reducing vehicle transport costs and delays; (iii) Improving the access to district headquarters not currently served by road; (iv) Promoting more sustainable funding and more efficient public sector management of road maintenance; and (v) Generating temporary rural employment through adoption of labor -based road construction technologies and long-term employment for road maintenance and additional economic activities generated by the improved road network. The differences are the following : PAD objectives 1 and 4 do not have counterpart in the DCA, although PAD objective 1 has an overlap with DCA objective (i); Objective 2 in the PAD combines DCA objectives (ii) and (iii); and, PAD objectives 3 and 5 are equivalent to DCA objectives (iv) and (v). The ICR uses the objectives as stated in the PAD. This ICR review will also use the PAD wording as the basis of evaluating the project; the choice is based on the greater clarity and measurability of objective 1 and the overall similarity of intent. The PDO was revised in the second restructuring in 2004 to read: "to provide sustainable and efficient road service on the Strategic Road Network�. The rewording of the PDO was not intended to mean changes in expected outcomes and benefits. It is a consolidation of the original PDOs, but less concrete and therefore not an improvement. The number of performance indicators was reduced from 20 to 6 to reflect changes in project components and to permit a more effective monitoring of project progress . This evaluation is done against the original objectives because the restructuring was not formal . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? No c. Components (or Key Conditions in the case of DPLs, as appropriate): Originally, the project had five components : 1. Policy reform to setup the Roads Board Nepal (RBN), amend regulatory framework and develop the Road Fund. This component financed technical assistance and study tours to the Implementation Committee, RBN, and key decisionmakers to Road Funds elsewhere, support private sector capacity building, and office equipment for the RBN. Cost at appraisal: US$0.40 million; cost at completion US$0.40 million. 2. New Road Construction and Upgrading (NRDUC). This component targeted construction of dry-weather feeder road construction (197 km) and upgrading to gravel standard (253 km), and associated support to resettlement assistance, environmental monitoring, design, project coordination and construction supervision, and civil works for improvements to training and office facilities . Cost at appraisal: US$36.60 million; cost at completion US$9.73 million. 3. Rehabilitation of 160 km of strategic highway and feeder road network (SRN). This component targeted rehabilitation of three priority roads of SRN – Tansen-Syangja, Tansen-Tamghas and Kothi –Taulihwa road.Cost at appraisal: US$5.70 million; cost at completion US$26.23 million. 4. Periodic maintenance of strategic roads (470 km). This component financed periodic maintenance of 470 km on SRN. Cost at appraisal: US$5.50 million; cost at completion US$11.06 million. 5. Institutional strengthening and training. This component consisted of consulting services for institutional strengthening/training of the Department of Roads (DoR); third party audits of works; 5-year training and human resource (HR) development program for DoR and the Ministry of Works and transport (MOWT); equipment for office modernization and training; and, pre-investment and socioeconomic impact studies of project's road improvements. Cost at appraisal: US$4.80 million; cost at completion US$7.80 million. The DCA has largely the same component descriptions, excepting that (i) policy reform included the establishment of the Roads Board and the Road Fund to increase road funding through road user charges, and (ii) the institutional strengthening component was specified in much greater detail without mentioning it being a 5-year HR development program. The project was restructured twice, as detailed below, due to security situation and a public sector budget crisis. The restructuring affected project implementation and achievement of the PDO . First Restructuring. The mid-term review (MTR) in November 2002 recommended a minor restructuring due to the security situation and public sector budget constraint. This restructuring was approved by the Country Director early 2003. The PDO and components remained the same, adjustments were made to the scale and scope of works to the following components: 2. NRDUC: carrying out of (i) construction work of 50 km of dry-weather feeder roads (versus 197 km planned at appraisal) to provide road access to two district headquarters (versus five as planned) EoP; and (ii) upgrading of about 200 km of existing earth roads (versus 253 km planned at appraisal) to gravel standards to connect one district and improve access in two other districts by EoP (same as the original targets). The construction of new roads was dropped in three districts – Jumla, Jajarkot and Bajura – due to insurgency. 3. Road Rehabilitation Component: expanding the component to include financing of high priority rehabilitation works on 227.7 km (versus 160 km planned at appraisal) of SRN. 4. Periodic Maintenance Component: Implementation of (i) periodic maintenance of about 650 km of the strategic highway and feeder road network (versus 470 km planned at appraisal), and (ii) support of DoR’s annual routine and regular maintenance for about 2,700 km of strategic highway and feeder road network in FY2002-2003 and about 1,500 km in FY2003-2004 as well as emergency maintenance, which would enable the DoR to meet urgent maintenance needs. Components 1 and 5 remained unchanged and two new components were added : (6) Community road works (45 km); cost at completion US$0.92 million, and (7) Emergency repair of flood damaged roads (36 km); cost at completion US$3.00 million During the first restructuring IDA’s financing share of civil works was increased from 80% to 85%, because of the budget crisis to provide counterpart funding. Unused funds from the NRUC component were distributed to the maintenance, rehabilitation, institutional strengthening components, and the new components . Second restructuring. During the second restructuring in April 2004, approved by the Regional Vice President, the following adjustments were made in components 2 and 3: 3. Road Rehabilitation Component: carrying out of: (i) rehabilitation work to about 248 km of the strategic highway and feeder road network (versus 230 km at the first restructuring); (ii) upgrading work to about 125 km feeder roads; and (iii) provision of bituminous seal to about 141 km of rehabilitated gravel roads . 4. Periodic Maintenance Component: implementation of (i) periodic maintenance of about 676 km of the strategic highway and feeder road network (versus 470 km planned at appraisal); (ii) performance-based maintenance of about 114 km of strategic highway; and (iii) routine and regular maintenance of strategic highway and feeder road network as well as emergency maintenance . During the second restructuring further reallocations were made . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The increase in IDA’s financing share of civil works, as a result of the budget crisis, reduced the Borrower's share to 10 percent at completion when it was 17.3 percent at appraisal; a significant decrease . The PDO was reworded in (April 5) 2004 to consolidate the original PDO and to clarify it . This was also considered to be a minor restructuring and was approved by the Regional Vice President . The new PDO became simply: “to provide sustainable and efficient road service on the Strategic Road Network �. The rewording of the PDO was not intended to mean changes in expected outcomes and benefits . The Credit closing date was extended twice and approved by the Regional Vice President. During the second restructuring, in 2004, the closing date was extended by 24 months to December 31, 2006 to complete the increased scope of work, which had been added in the first restructuring. The second extension was for six months to June 30, 2007 to complete the remaining works which had been disrupted during the conflict in 2005 and early 2006. 3. Relevance of Objectives & Design: The PDO supports the CAS of 1998 and the CAS of 2004-2007. The former identifies the main issues as : poverty reduction in rural areas, promotion of higher agricultural output and non -farm income, improvement of poor governance and economic weaknesses, and ineffective public service delivery . The CAS of 2004-2007 is more explicit and lists four broad objectives : (i) to achieve sustained high and broad -based economic growth, focusing particularly on the rural economy; (ii) to accelerate human development through effective delivery of basic social services and economic infrastructure; (iii) to ensure social and economic inclusion of the poor, marginalized groups, and less developed regions; and (iv) to vigorously pursue good governance . The PDO supports the broad CAS objectives and was relevant at appraisal and also currently . The components of the project design are appropriate to the strategy, although the policy reform and institutional components had too demanding targets . The 20 performance indicators in the PAD were excessive and no baseline was present for several items, nor was data collection indicated to ensure indicator usefulness . Furthermore, the indicators to monitor the achievement of the project objectives, whether before or after restructuring, are not clearly linked to the PDO, but rather to project outputs . The revised indicators, while clearer, should have had a more explicit relation to the PDO. For example, for "sustainable and efficient road service ", the existence of the Road Fund is less important than having sufficient funding for routine and periodic maintenance at least for the improved roads . There also should have been a clear statement whether "sustainable funding" means funding of the Medium Term Expenditure Framework (MTEF) for roads (which was not funded). Likewise, if the Department of Roads (DoR) is "to operate in corporate manner" (indicator 4), it is not sufficient to have an indicator that only relates to IT, HR, and an Action Plan to do so in 10 years. A business plan with substantive content that DoR itself had developed and intends to follow would have been better, but, would have been qualified as clearly out of reach within one project . 4. Achievement of Objectives (Efficacy): Achievement of objectives. 1. Achieve sustainable maintenance, rehabilitation, and construction of economically justifiable roads in SRN within the PIP (priority investment plan) framework. To the extent this objective means maintenance, rehabilitation and construction of the planned project roads only, it was substantially achieved . However, the intent appears to have been that there is a sustainable funding mechanism for the MTEF, the achievement is rated modest . 2. Improve access to district headquarters not currently served by road, and reduce vehicle transport costs, and delays in project areas. The planned improvements for access to district headquarters and surveys of vehicle transport costs (Annex 5) indicate that the objective was substantially achieved. This assessment includes consideration of the insurgency and the revised target (access to 2 out of the initially planned 5 district headquarters), after the first (minor) restructuring. 3. Promote more sustainable funding and more efficient public sector management of road maintenance. Although the Road Fund law is being considered its provision for sustainable funding has not yet become reality. The management of the DoR has clearly improved . Achievement of this objective is modest . 4. Adopt and disseminate environmentally sustainable road construction and maintenance practices. Technical assistance and training for this objective was carried out and successful. Achievement of the objective is substantial. 5. Generate rural employment through adoption of labor-based technologies and long-term employment for the additional generated economic activity. The community works component was successful and created employment for local people, especially in the lower economic and social strata. The long-term sustainability is uncertain, however. Achievement of the objective is substantial . Overall the achievement of the objectives was modest . 5. Efficiency (not applicable to DPLs): The ERR of the project's road works is high . This is attributed to the introduction of new, low cost surface treatment technology for low volume roads . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal Yes 28% 88% ICR estimate Yes 46% 79% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project achieved its planned physical targets as revised and DoR staff capabilities were improved as intended . There were moderate shortcomings on the institutional framework and its implementation in terms of the Roads Board, maintenance funding, and usage of the MTEF for priorities, although some progress was made . The road works were implemented in compliance with the Bank's guidelines for the environment and resettlement, although at times the lack of security prevented timely compliance of the social safeguards . There also were delays in disbursements. There were positive effects on transport costs and tariffs,and there was 27- 112 % increase in traffic indicating increased economic activity . The project was associated with several positive social indicators . These are relevant outcomes to the PDO achievement . The monitoring indicators should have had better definitions and associated data collection mechanisms and a scope consistent with the PDO, as commented in Sections 3 and 10. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to development outcome is moderate . On the positive side is the progress made to improve the technical capacity and operating practices at DoR . There also appear to be staff motivation and better road tools to improve road management. The developments regarding the operating procedures and funding of the RBN are also positive . The lack of actual implementation of well -intended changes both in RBN management autonomy and funding level places the risk at the moderate level . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: In spite of QAG's HS rating at quality-at-entry, the project had substantial unresolved design issues . The PDO could have had a more explicit link with the CAS, and the project components should have had better monitoring framework supported by a data collection system . The risks did not mention the disruptive potential of the insurgency. This should have been noted since it was active at the time . The ICR also notes the insufficiency of computer aided design without geotechnical information . That issue, while clearly a risk, is complicated given the Nepalese terrain and its susceptibility to land slips . Often, in such circumstances, it is less expensive to go ahead with inadequate design and do the thorough geotechnical investigations as needed during the work (with due provisions for contingencies ). The project design was also overly optimistic about the institutional reforms, particularly with the road fund and funding of the MTEF . Although institutional reform had an 'S' risk rating, its mitigation measure and plan was unsophisticated . The implementation team recovered from the inadequacies at entry and showed flexibility and pragmatism to restructure the project and modify the components to fit the changing political context without compromising the intentions of the original PDO, even if the PDO, ambiguously defined, de facto reduced the targets . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: The Borrower adjusted less well than IDA to the implementation realities . There were difficulties with the road budget and the counterpart funds . The institutional framework and the Road Fund fell short of the initial aspirations; the MTEF was not adequately funded; and, DoR modernization proceeded slower than planned and was not completed at project closing . Though lukewarm to the reforms, as the adverse political context consumed the government's energies, it did not stand in the way but supported DoR to implement the project . DoR cooperated with IDA in making meaningful changes to the project and trained its staff to build capacity . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: The M&E design was flawed at the start . It was never implemented as designed in the PAD . For several indicators there was no baseline value. Others had an annual data collection mechanism through Highway Management Information System (HMIS). The M&E design was redone and improved at the second restructuring . The resulting six indicators are monitorable, but their consistency with the revised PDO is ambiguous . It is unclear if the indicators, other than those specifically linked to outputs (indicators 2 and 3), which could be monitored through the Road Condition Report from the HMIS, were used to monitor project progress . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The project complied with Nepal's and the Bank's environmental regulations, although there was a learning curve . Financial management training improved fiduciary performance and compliance and, with some suggestions for improvement, the audits were satisfactory to the Bank . Procurement was carried out with Bank guidelines and fairly . There were no project induced negative impacts; the project's association with increased literacy, improved school enrolment, and increased use of health services was positive . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: 1. Projects in conlict-affected areas should have a flexible design and implementation plan . Social conflict and insurgency affects project planning and implementation . The project team should have substantial leeway without undue process burden to redefine project components as long as the project objectives do not change . In this project the components were changed, but for overall benefit the change could have taken place earlier . 2. Local consultants and contractors are more effective than foreign entities in conflict areas . Projects in conflict areas could be implemented with local consultants and contractors, who can negotiate with the conflicting parties to implement projects with their safeguards . 3. Dissemination and application of project experience is best carried out by staff who have hands -on experience with all IFI funded projects. The project experience supports integrated project management, not a separate PIU . 4. Financial incentives improve compliance with safeguards . Contractors can be reluctant to follow EMP (environment management plan) if there is no financial incentive to do so . EMP - related items should be put in the bill of quantities and unit rates to motivate contractors to comply with (environmental) safeguards. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR had a difficult task to describe the achievements of a project whose design was less than clear . It would have been appropriate to evaluate the achievement of the objectives rather than the simply the degree to which the indicators were achieved. The former has a broader scope . Nonetheless, the ICR outlined the project implementation and its major phases, accomplishments and results in a fair manner . The beneficiary survey results in Annex 5 are useful. a.Quality of ICR Rating : Satisfactory