Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) Report Number: ICRR0022014 1. Project Data Project ID Project Name P122204 Second Secondary and Local Roads Project Country Practice Area(Lead) Georgia Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-81430,IDA-50780 30-Jun-2017 62,501,983.40 Bank Approval Date Closing Date (Actual) 15-Mar-2012 30-Jun-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 70,000,000.00 0.00 Revised Commitment 66,469,548.32 0.00 Actual 62,501,983.40 0.00 Prepared by Reviewed by ICR Review Coordinator Group Elisabeth Goller Peter Nigel Freeman Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project development objective (PDO), as mentioned in the project appraisal document (PAD, para 18) and the loan and financing agreements (LA, FA, schedule 1) was to "improve local connectivity and travel time for the selected secondary and local roads, and to strengthen the capacity of the roads department (RD) to manage the road network." Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) For the purpose of this review, the PDO will parsed as follows: (i) improve local connectivity for the selected secondary and local roads; (ii) improve travel time for the selected secondary and local roads; and (iii) strengthen the capacity of the RD to manage the road network. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components The project was structured along the following two components: Component 1: Rehabilitation and Improvement of Selected Secondary and Local Roads (cost at appraisal US$86.0 million, including contingencies; actual cost US$81.3 million). This component aimed at (i) rehabilitating and improving 19 road sections with a total length of 225 km, including piloting two design- build contracts; and (ii) rehabilitating, repairing, and maintaining 200 km of roads in the Kakheti Region using the Output and Performance-based Road Contract (OPRC) model. Component 2: Institutional Strengthening and Project Management (cost at appraisal US$1.5 million, including contingencies; actual cost US$1.8 million). This component aimed at strengthening the capacity of the RD and the foreign project unit in (i) project management and implementation; (ii) identifying, developing, and implementing road safety measures on secondary and local roads, including improving road safety engineering at the design stage and road safety audits; (iii) carrying out impact evaluations; (iv) developing the road asset management system; and (vi) managing and implementing alternative contracting methodologies. The project components and activities were not changed. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost The total project cost was US$83.1 million (ICR, para 15 and page 39), which is 95 percent of the appraisal estimate of US$87.5 million. The cost savings were due to the devaluation of the Georgian Lari vis-a-vis the US$ and the fact that the remaining two years of maintenance under the OPRC will be financed by the government. Financing The project was financed through an IDA credit and an IBRD loan for a total amount of US$62.5 million equivalent (ICR, datasheet, page 2), which is 89 percent of the IDA and IBRD financing of US$70.0 million Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) equivalent envisaged at appraisal (IDA credit of SDR25.8 million, equivalent to US$40.0 million, and IBRD loan of US$30 million (LA and FA)). The IDA credit and the IBRD loan were not fully disbursed because (i) the government repaid US$3.5 million, partially from the IDA credit and partially from the IBRD loan, to the World Bank after misprocurement was declared and (ii) an undisbursed balance of about US$3.4 million by project close under the IBRD loan due to the devaluation of the Georgian Lari, which could not be used for additional maintenance works because of procurement problems. It should be noted that these two amounts do not add to the undisbursed amount of US$7.5 million due to fluctuations in the US$/SDR exchange rate, which reduced the US$ equivalent of the credit. Borrower Contribution According to the Bank task team, the actual borrower contribution was US$20.6 million, which is 118 percent of the appraisal estimate of US$17.5 million. Dates and Project Restructuring The project was approved on March 15, 2012, became effective on May 24, 2012, and was originally expected to close on June 30, 2017. The project was restructured on July 9, 2015, when an amount of US$51.0 million was disbursed. The restructuring was necessary to extend the closing date by 24 months, to June 30, 2019 to implement the first five-year OPRC pilot in Georgia. This OPRC represented 46 percent of the IBRD loan, and its original bidding process had to be cancelled because the bid prices were three times higher than the budget estimate mainly because of the inclusion of maintenance of a gravel road in the contract in addition to paved roads. In addition to the closing date extension, the restructuring included changes to the results framework to adjust several indicator targets to the actual project circumstances, including the targets of two outcome indicator targets. The target of the outcome indicator of "average decrease in travel time" was made more ambitious by increasing it from 20 to 40 percent. The target for the outcome indicator of "roads managed under alternative contracting methodology" was adjusted downwards from 200 km to 137 km. Considering that two key outcome indicator targets were revised, this review will apply a split rating and will assess the project outcome based on the original and revised indicator targets. 3. Relevance of Objectives Rationale The relevance of the objective is considered substantial. At appraisal in 2012, the country had mostly recovered from the August 2008 war with Russia and the 2008-2009 global economic crisis. The economy grew by 7.2 percent in 2011, and the country continued its high levels of public investment, mainly in the road sector. This public investment had been aimed to boost the economic recovery by improving main road corridors and local connections and create temporary employment. Despite these investments, and progress being made in the sector, 70 percent of the secondary and core local roads required significant improvements. Therefore, improving local connectivity and travel time and strengthening the RD's network Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) management capacity was highly relevant. The PDO was in line with the strategic goal of Georgia's FY2010-13 World Bank Country Partnership Strategy of strengthening competitiveness for post-crisis growth, by facilitating improved road connections and reducing transport costs in the region. The PDO remained relevant by project close. At that time, 34 percent of the secondary roads were still in bad condition (ICR, page 32 - the ICR does not provide information on the condition of local roads). Georgia's FY2019-22 World Bank Country Partnership Framework envisages support to Georgia, among others, to enhance inclusive growth and competitiveness. In this area, better connectivity and reduced travel times are expected to improve access to markets and economic participation in the region. In addition, a strengthened RD was expected to better plan and manage the road infrastructure. This was the third World Bank project that supported secondary and local roads in Georgia.The first Secondary and Local Roads Project aimed at upgrading and rehabilitating roads and strengthening the road department's management capacity. The PDO of the Kakheti Regional Roads Improvement Project was to reduce transport costs and improve access and traffic safety. By aiming at improving local connectivity and travel time and strengthening road management, the project's ambitions were higher than under the first project and similar to the Kakheti project. However, the project is different and more sophisticated compared to the previous ones because it experimented with OPRCs. Consequently, the PDO's level of ambition was largely adequate. The PDO was realistic, but the term "connectivity" would have benefited from a clear definition. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve local connectivity for the selected secondary and local roads (under original outcome targets) Rationale The theory of change for subobjective 1, as reconstructed by IEG based on the information in the PAD, was that rehabilitating selected secondary and local roads and paying adequate attention to road safety features in this process would increase the km of local and secondary roads rehabilitated with such adequate features. This, in turn, would increase the km of secondary and local roads in good and fair condition, and more roads in good and fair condition would mean that local connectivity improved. In addition, maintaining these roads and using alternative contracting methods, especially the OPRC, which combines road maintenance and rehabilitation, were to enhance the sustainability of the roads and connectivity improvements. For the purpose of this review, IEG interprets connectivity as the ability of people to connect by road to their destinations, which not only requires roads or better roads, but in most cases also vehicles and transport services. Therefore, the theory of change was not only based on the assumption that the government has Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) adequate resources and capacity to improve the roads, but also that better roads will attract the necessary transport services. Outputs The following outputs were achieved during the life of the project to meet subobjective 1 (ICR, para 29 and annexes 1 and 6.3):  238 km of secondary and local roads rehabilitated, more than the original target of 225 km and the revised target of 200.5 km  20 road safety audits completed, significantly more than the original target of 5 and in line with the revised target of 20  117 km of roads maintained under the OPRC pilot  135.27 km of roads managed under alternative contracting methods, slightly less than the revised target of 137 km and significantly less than the original target of 200 km Outcomes Because of the road rehabilitated under the project, the secondary and core local road network in good and fair condition increased from 30 to 34 percent, in line with the original and unrevised target. The project directly benefited 122,000 residents (the original and unrevised target was 150,338 direct project beneficiaries) and provided them with a better connection to markets, services, and the Georgian road network (ICR, para 29). The project's socio-economic impact evaluation carried out for nine rehabilitated roads in 2015 also provided evidence on the project's contribution to improved connectivity (ICR para 32 and annex 6.5). The impact evaluation showed in a statistically significant way that, for the households in the treatment group, the frequency of bus use to most destinations, including schools, municipal centers, hospitals, and pharmacies, increased by about four percent. A larger increase of 10 percent was seen in the frequency of minibus journeys to schools. Correspondingly, walking decreased as a mode of travel for visiting markets, municipal centers, pharmacies, and schools. 37 percent of the respondents mentioned that the road rehabilitation has simplified their travel and made it easier for residents in villages to reach their destinations. 25 percent of the respondents also mentioned a reduction in travel time. Among the positive impacts of the road rehabilitation, the respondents identified better transport schedules, lower expenses for fuel, and increased communication between villages and the city. Local businesses reported having increased the distance driven by about 12 percent. Over 90 percent of both residents and businesses stated their access to various services has considerably improved or improved. The impact assessment also showed that the travel cost per km, which according to the task team refers to the operating cost per km considering fuel and maintenance but not to the time cost of the driver and/or passengers, decreased on average by 0.04 GEL. For households with a person with disabilities, it decreased by 1.9 GEL per km. However, the statistical significance of the travel cost-related findings was weak. The Bank task team clarified that the impact evaluation used a difference-in-difference methodology. 3285 households were surveyed, 2287 in the treatment area and 998 in the control area, before and after the road interventions. According the Bank task team, the impact evaluation was adequately done and comprehensive, but it had shortcomings in the presentation of the results. Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) Rating Substantial OBJECTIVE 1 REVISION 1 Revised Objective To improve local connectivity for the selected secondary and local roads (under revised outcome targets) Revised Rationale The theory of change remained unchanged under the revised outcome indicator targets. Outputs: The outputs are the same under the original and the revised outcome indicator targets. Outcomes: The outcomes are the same under the original and the revised outcome indicator targets. Revised Rating Substantial OBJECTIVE 2 Objective To improve travel time for the selected secondary and local roads (under original outcome targets) Rationale The theory of change for subobjective 2 was that rehabilitating selected secondary and local roads would decrease travel times on these roads, and this, in turn, would improve travel time for the selected secondary and local roads. Outputs The outputs achieved during the life of the project to meet subobjective 2 were the same as mentioned under bullet 1, 3 and 4 for subobjective 1 above. Outcomes The secondary and core local road rehabilitation under the project improved the condition of these roads, bringing the network in good and fair condition from 30 to 34 percent, in line with the original and unrevised target. These improvements reduced the average travel time on the 19 project-rehabilitated roads by 44.1 Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) percent, from 5.97 hours before to 3.34 hours after the rehabilitation. This reduction is significantly higher than the original target of 20 percent. The project's socio-economic impact evaluation of 2015 also confirmed an improvement in travel time for the households in the treatment group. This evaluation showed in a statistically significant way that the average time household members spent traveling per kilometer after the roads were rehabilitated decreased significantly, i.e. by 0.94 minutes in the treatment group for the whole sample. The effect is slightly higher for the bottom 40 percent of the sampled population by household income in the treatment group, where the average time spent per km decreased significantly by 0.97 minutes. Rating High OBJECTIVE 2 REVISION 1 Revised Objective To improve travel time for the selected secondary and local roads (under revised outcome targets) Revised Rationale The theory of change remained unchanged under the revised outcome indicator targets. Outputs: The outputs are the same under the original and the revised outcome indicator targets. Outcomes: The outcomes are the same under the original and the revised outcome indicator targets, except for the indicator of "average decrease in travel time", for which the project end value of 44.1 percent is in line with revised target of 44.0 percent. Revised Rating Substantial OBJECTIVE 3 Objective To strengthen the capacity of the RD to manage the road network (under original outcome targets) Rationale The theory of change for subobjective 3 was that (i) supporting project management and implementation, (ii) identifying, developing, and implementing road safety measures on secondary and local roads, including improving road safety engineering at the design stage and road safety audits, (iii) supporting impact evaluations, (iv) supporting the development of the road management system, and (vi) supporting the Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) management and implementation of alternative contracting methodologies would, among others, lead to more km of roads managed under alternative contracting methods. This, in turn, was to be an indication that the RD's road management capacity had been strengthened. The theory of change's underlying critical assumption was that the alternative contracting methods, especially the OPRC, were sufficiently attractive to local contractors. Outputs The following outputs were achieved during the life of the project to meet subobjective 1 (ICR, paras 38 to 48 and annexes 1 and 6.3):  Two alternative contracting methods, OPRC and design and build contract, piloted, in line with the target  20 road safety audits completed, significantly more than the original target of 5 and in line with the revised target of 20  22 standalone rural road safety schemes designed and implemented (ICR, page 36)  Consultancy services for road asset management provided over a period of five years, which according to the Bank task team were to support the continuous development of the road asset system, its running, and the preparation of the planning documents  Technical specifications for road asset management equipment, and equipment and data collection vehicle procured. The Bank task team pointed out that this included a systems for video logging, measuring roughness and geometry, traffic counters, skid resistance measurement equipment, and a 4x4 vehicle on which to mount the equipment  Feasibility studies, preliminary designs, and bidding documents for two roads to be rehabilitated under the design and build method prepared  Application program interface and visualization tools for the RD to connect with the government's e- procurement system developed and in use  Two impact evaluations completed, each with studies before and after the project. The Bank task team clarified that this included the project's socio-economic impact evaluation of 2015 mentioned under subobjective 1 and the 2019 road rehabilitation impact evaluation of the OPRC carried out in the Kakheti region The Bank task team noted that the following outputs were also achieved:  Informal evaluation of the Kakheti OPRC pilot by the project's monitoring consultant to correct shortcomings during its implementation and help shape future OPRCs  Workshop on OPRCs organized by the Bank for the RD staff and contractors  Training on OPRCs provided by the project's monitoring consultant to the RD staff  Technical and contractual documents for the Kakheti OPRC pilot. A consultancy services contract for the “Preparation of a Framework for the Design and Implementation of Concessions (Contracts) for Performance-Based Road Maintenance” was originally started under the Third East-West Highway Improvement Project and then moved to this project Outcomes Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) The project used the indicator of "km of roads managed under alternative contracting methods' as a proxy to measure the achievement of this subobjective. By project end, 135.27 km of roads were managed under alternative contracting methods, significantly less than the original target of 200 km. However, the Bank task team confirmed that the underachievement on this indicators does not indicate that the RD's capacity was not adequately strengthened. Rather, after a first bidding with significantly higher bid prices than the estimated budget, the RD reconsidered the roads to be included in this contract to better balance the contractual risks between the parties. This reduced the number of km of road under the OPRC. The Bank task team highlighted that, in addition to the above-mentioned workshop and training, the RD learned by doing under the OPRC pilot because the RD managed its implementation and was closely engaged in all steps of the process. The ICR (para 43) highlights that the experience and lessons learned by the RD from managing the OPRC pilot are being carried forward, and the RD is now developing a hybrid OPRC for 240 km of roads in the Guria region, incorporating lessons learned from the OPRC pilot under this project. The Guria OPRC is also supported by the Bank under the Secondary Road Asset Management project. The RD is preparing a further OPRC under an Asian Development Bank project. The ICR (paras 40 to 42) lists the hiccups in the implementation of the first OPRC, which were successfully overcome. It also highlights the residents' satisfaction with the road network in the area where the OPCR was implemented. The Bank task team clarified that this feedback is based on the 2019 road rehabilitation impact evaluation in the OPRC's area of influence. Therefore, the positive response can be attributed to the road improvements under the project in this area, but it is not clear if it is due to the OPRC modality or just the road rehabilitation. As for the design and build contracts, the ICR (para 39) highlights that using alternative contracting methods successfully is a sign of enhanced capacity. The Bank task team clarified that by introducing these methods to the RD and providing the resources to implement and manage them, the project enabled the RD to understand their benefits and created the capacity to move forward with them and other alternative contracting methods. These instruments provide the RD with tools for more flexibility in procuring works and maintenance. The Bank task team also pointed out that the project's support to road asset management increased the RD's capacity to collect road condition and traffic data and prepare the five-year rolling road program and the annual planning. According to the ICR (para 44), the current use of the road asset management system demonstrates a real strengthening of the RD's capacity to manage the network and represents one of the best practices in the region. However, the Bank task team and the ICR point out that the Bank has been supporting the introduction of road asset management in Georgia for more than a decade through a series of different projects, so not everything is attributable to this project. The impact evaluations provided the RD with a better understanding of the impact of the project investments on the rural population and demonstrate better the project's outcomes (ICR, para 45). As an indicator of the RD’s commitment to the use of impact evaluations, the ICR mentions that the RD undertook the final survey for the OPRC area from their national budget after the project closed. The project strengthened the RD's road safety capacity by introducing them to the concept of standalone small-scale road safety schemes for rural roads. Through the support of two project-financed consultancies, the RD identified 22 village locations, where they implemented road safety improvements, such as better signs, road markings, guard rails and safety islands. The Bank task team pointed out that this created the Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) necessary capacity to carry out small road safety schemes, which the RD plans to continue with their own funds. The application program interface and web-based visualization tools enabled the RD to use the government's e-procurement system, which improved transparency and efficiency in the procurement process and the utilization of public funds (ICR, para 48). The Bank task team clarified that the interface enabled the connection to the government's e-procurement system whereas the visualization tool generated procurement related statistics and data that allow for a better management of the procurement process. According to the Bank's 2018 enhanced procurement review, the e-procurement system minimized the risk of collusion between civil servants and bidders or between bidders. This is because contrary to the previous practice of using "paper-based" procurement, the implementing agency does not know which bidders submit or intend to submit bids (ICR, para 48). The review also found that the system increased the level of competition because the average number of bidders under this projects, which used "paper-based procurement", was much lower than under successive Bank project, which used e-procurement. Rating Substantial OBJECTIVE 3 REVISION 1 Revised Objective To strengthen the capacity of the RD to manage the road network (under revised outcome targets) Revised Rationale The theory of change for subobjective 3 remained unchanged under the revised outcome indicator targets. Outputs: The outputs are the same under the original and the revised outcome indicator targets. Outcomes: The outcomes are the same under the original and the revised outcome indicator targets, except for the indicator of "km of roads managed under alternative contracting methods", which with 135.27 km nearly reached the revised target of 137 km. Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) Rationale The first subobjective under the original outcome indicator targets is rated substantial, the second is rated high, and the third is rated substantial. Therefore, the overall efficacy of the PDO under the original outcome indicator targets is rated substantial. Overall Efficacy Rating Substantial OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale All subobjectives under the revised outcome indicator targets are rated substantial, therefore, the overall efficacy of the PDO under the revised outcome indicator targets is rated substantial. Overall Efficacy Revision 1 Rating Substantial 5. Efficiency Economic Efficiency At appraisal, the economic analysis covered six of the 19 road sections, totaling 61 km. The borrower carried out separate economic analyses for the remaining roads and the OPRC pilot during project implementation. The borrower used the Highway Development and Management (HDM-4) model, assumed a 20-year evaluation period, and applied a 12 percent discount rate. The assumptions given in the PAD appear realistic. The analysis showed an average Economic Internal Rate of Return (EIRR) of 19.8 percent and an average Net Present Value (NPV) of US$13.1 million. The sensitivity analysis showed an average EIRR of 15.8 percent if the costs were to rise by 15 percent and the benefits were to go down by 15 percent. The ICR (para 56) mentions that the economic analyses for all 19 road rehabilitations under the project were economically justified with an average EIRR of 18.5 percent. The sensitivity analysis demonstrated that most roads maintained an EIRR greater than 12 percent if the costs were to rise by 15 percent and the benefits were to go down by 15 percent. The ICR also points out that the evaluations were prepared by different consultants during 2011 and 2012 using different assumptions, so the EIRRs for individual roads may not be directly comparable. Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) For the OPRC, the borrower carried out an economic analysis for the 117 km of paved road sections to be included in this contract. This analysis showed an EIRRs that varied from 13.1 to 42.8 percent for the different sections. For the after project economic analysis of the 19 roads rehabilitated under the project, the borrower used the known traffic growth rates and the actual construction costs but kept the other values constant where applicable. As with the ex-ante analysis, a 20-year evaluation period was assumed and a 12 percent discount rate was applied. The analysis confirmed that the investment in rehabilitating the secondary and local roads was economically justified with an average EIRR of 14.9 percent and a NPV of US$11.11 million. However, the rate of return was lower than the appraisal estimate of 18.5 percent and some of the sections had an EIRR lower than 12 percent. The ICR (para 58) attributes the lower rate of return to the lower economic growth than forecasted and differences between the economic models used. For the OPRC, the borrower carried out the post-completion economic analysis for the 37.5 km of road rehabilitation works. The analysis showed that the OPRC was economically justified, with an EIRR of 15.3 percent and a NPV of US$2.25 million. Due to the different scope of the OPRC than originally planned and the way the EIRRs are presented in the PAD the ICR, it it not possible to compare the ex-ante and ex-post returns. Administrative and Operational Efficiency: As mentioned in section 2.e., the total project cost was five percent lower than the appraisal estimate due to currency devaluation and a remaining contractual balance under the OPRC to be paid by the borrower after project close. The cost per km of the rehabilitation under the OPRC was comparable with rehabilitation costs of the other roads under the project. The rehabilitation cost, which excludes the maintenance component of the OPRC, was US$0.31 million per km, compared to an average of US$0.28 million per km for rehabilitation using traditional input-based contracts. According to the ICR (para 61), the slightly higher cost per km might be attributable to the additional risks associated with the OPRC and the cost of design of the road works. The ICR (para 62) points out that the cost of the maintenance component of the OPRC was somewhat higher than the average for nine other maintenance contracts of the RD (GEL 12,481/km/year compared with GEL 11,100/km/year). However, unlike the other contracts, the OPRC included maintenance backlog clearing and the risk of penalties for not meeting service levels. This might justify a slightly higher cost per km. The two-year closing date extension and the support needed in the implementation of the first OPRC considerably increased the World Bank's supervision budget (ICR, para 65). They also increased the borrower's project implementation costs (the ICR does not provide figures). Overall, the project was implemented largely efficiently despite lower economic rates of return than at estimated at appraisal and delays and higher administrative costs in the implementation of the OPRC pilot, which are expectable. Efficiency Rating Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 76.00 Appraisal  18.50  Not Applicable 68.00 ICR Estimate  14.90  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome With a substantial relevance, efficacy, and efficiency, the overall outcome under the original outcome indicator targets is rated satisfactory (5). With a substantial relevance, efficacy, and efficiency, the overall outcome under the revised outcome indicator targets is rated satisfactory (5). Based on the share of the disbursements of 81.6 percent at the time of the restructuring on July 9, 2015 (US$51.0 million out of US$62.5 million), the overall outcome rating is satisfactory (81.6*5+18.4*5=5). a. Outcome Rating Satisfactory 7. Risk to Development Outcome The risk that the development outcome of this project will not be maintained is modest. The main risk relate to the following:  Road maintenance of local roads. Four of the rehabilitated roads are under the responsibility of municipalities and therefore do not come under the RD’s maintenance program. There is a risk that the maintenance of these roads will be less than optimal, but the roads are currently being maintained with the municipalities outsourcing maintenance on an annual basis (ICR para 117).  Future funding for maintenance of roads under the responsibility of the RD. According to the ICR (para 116), the RD maintains its roads by outsourcing maintenance to the private sector and maintenance funding is currently not an issue. However, there is a possibility that policy and economic changes might put this funding at risk. For the pilot OPRC, maintenance is included in this contract for a five-year period and the RD took over the funding of the remaining maintenance period at project close without any issues. Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204)  Significant staff changes in the RD. A significant change in technical staff in the RD could undo some of the capacity strengthening achievements under the project, especially in terms of alternative contracting methods, road asset management, and road safety. However, the Bank task team pointed out that most key technical staff in the RD has been there for many years and when the government changes, normally only the chairman, vice-chairman, and some heads of divisions are replaced. 8. Assessment of Bank Performance a. Quality-at-Entry The Bank task team largely designed the project based on the experience and lessons from the previous road projects in Georgia. The implementation arrangements stayed the same. The main differences with respect to the first Secondary and Local Roads Project was the strong focus on alternative contracting methods and the discontinuation of the support to capacity strengthening for municipalities. The reasons for the increased focus on alternative contracting methods are extensively documented in the PAD (annex 2, appendix 2) and the approach is well justified. The PAD does not mention why capacity strengthening for municipalities was not continued considering that the project still rehabilitated local roads under the responsibility of municipalities and the previous efforts to strengthen their capacity were not successful. Rightly and based on lessons from previous projects, the Bank team put a stronger focus on incorporating road safety considerations in road designs and strengthening the road departments road safety capacity. The team also used a more stringent approach to the evaluation of technical alternatives for road rehabilitation (PAD, para 15), using the HDM-4 model and ensuring that the country's key road priorities were included in the project. With respect to the alternative contracting methods, it was adequate to start with a pilot, which could be assessed and improved over time. The Bank task team identified most project implementation-related risks and considered the overall project risk as low. Although a low overall risk seems adequate in the light of Georgia's extensive experience with Bank-financed projects, the Bank task team failed to identify the risks related to the introduction of the first OPRC, including possible delays that materialized, and the risk related to the fact that road improvements alone might not be sufficient to improve connectivity. The Bank task team did also not propose mitigation measures for the governance risk because previous projects had no major procurement issues. The PAD (para 60) highlights the project readiness. When the project started, the RD had prepared the detailed designs and the bidding documents for six of the 19 road sections. The RD was also preparing the detailed designs for additional road sections and had started the selection of the supervision consultant. his made it possible for the RD to start the procurement of 18 works contracts within six months of project effectiveness. Preparations for the design and implementation of the OPRC were also underway by appraisal because the consultant had been appointed in March 2011 under the Third East West Highway Improvement Project. Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) The Bank task team carried out adequate safeguards and fiduciary assessments and ensured the preparation of the necessary safeguards documents. As mentioned below, the project had some shortcomings in M&E. Quality-at-Entry Rating Satisfactory b. Quality of supervision The Bank task team actively engaged with the RD and supported them in all aspects of project implementation. According to the ICR (para 112), the Bank task team closely worked with the RD to promote and move forward the OPRC pilot, which faced some resistance at times. The Bank task team identified the risks the failed OPRC bidding process posed to the achievement of the PDO and properly addressed it through project restructuring. However, as seen in section 12 below, the continuous support of a specialized technical consultant and a thorough industry and market analysis would have been beneficial. The Bank task team supported financial management through consulting services and was proactive in timely addressing the procurement and contractor performance challenges encountered throughout the project’s duration. This included (i) downgrading the procurement rating to moderately satisfactory to signal the problem of low bid participation in the first round of civil works contracts (ICR, para 107), (ii) convincing the RD to carry out an enhanced procurement review after the complaint by a potential bidder, which led to an INT case, (iii) preparing a procurement improvement plan for the RD, (iv) using project funds to develop an interface that enabled the RD to use the governments e-procurement system, and (v) preparing a web- based visualization tool to generate procurement-related data graphs, such as graphs on current and historical bid participation (ICR, paras 83 to 91 and 48). With respect to the low bid participation, the Bank task team clarified that because the contracts were small and awarded under national competitive bidding, they thought that Georgia had problems with its local construction industry. Therefore, they used the Bank budget to carry out a construction industry review in 2013. The Bank task team also pointed out that because of the low bid participation in this project, in the subsequent project they changed the contracting strategy, dividing the contracts into lots of different sizes. The Bank task team identified and raised issues related to the inadequacies in the disposal of waste and excess materials and pushed for proper action. The ICR (para 112) points out that although the change in the RD management during the project’s implementation constituted an opportunity, it also meant that the Bank task team had to build new relationships and repeatedly insist on the project’s objectives. The Bank task team carried out the supervision of this project together with the other road projects in the RD. Early practical experience from this project contributed to the design and implementation of third Secondary and Local Roads and the Secondary Road Asset Management projects, particularly in the areas of output and performance-based road contracting, design-build contracts, road asset management, road safety, and employment opportunities for women. Quality of Supervision Rating Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The RD was to be responsible for M&E and reporting. At appraisal, it had a dedicated M&E sub-unit with three staff. The results framework included three PDO indicators, each of them expected to measure one of the subobjectives. For two of the subobjectives, measuring their achievement was difficult, mainly due to shortcomings in the PDO indicators. With respect to the subobjective of improved connectivity, first, a definition of "connectivity" was missing. This made results measuring subjective. Second, using the core indicator that measures the percentage of the total classified roads in good and fair condition did not tell the whole story. Although more roads in good and fair condition enhance the ability of people to connect to their destinations, most people also need vehicles or transport services to be able to benefit from the road improvements to enhance their connectivity. Therefore, the project could have included an additional indicator that measured, for instance, the increased availability of transport services. Third, the core indicator is expected to capture the percentage of Georgia's “total” classified network in good and fair condition. In this case, it only referred to the 5,446 km of Georgia's secondary road network. This created an inconsistency in measuring the core indicators at Bank-wide level. Finally, using this core indicator to measure the achievement of project- specific objectives might cause an issue of attribution because road works additional to the project works might contribute to improvements of the "total" classified network". This was not an issue in this case because the borrower only captured the km of roads attributable to the project in the indicator end value. For the subobjective related to the RD's road management capacity strengthening, in hindsight, an indicator that measured the km of roads managed under alternative contracting methods was not the best choice for two reasons. First, although successfully managing roads under alternative contracting methods is an indication that the RD's capacity increased, the indicator did not include a reference to "successful" management, which also would have required a clear definition. Second, this indicator did not capture all key capacity strengthening results expected from the project, such as enhanced used of the road asset management system or the capacity to carry out road safety audits in-house. Therefore, additional indicators would have been desirable. The difficulties in measuring the project achievements caused by the shortcomings in these two indicators were overcome through additional evidence mainly from the two impact assessments carried out under this project (see section 4, subobjective 3). The project had four intermediate indicators adequate to measure project progress for certain activities. A few additional intermediate outcome indicator to capture key milestones, for instance, with respect to the road asset management system, could have been added. Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) All indicators had baselines and target values, but the targets for two indicators were too conservative and had to be revised. b. M&E Implementation The ICR (para 98) points out that M&E was satisfactory for the duration of the project. It mentions that the RD's small dedicated M&E unit collected information related to the project, including fiduciary and safeguards aspects, from the different RD units, the Transport Reform and Rehabilitation Center, and the supervision consultants. The information formed the basis of the RD’s semi-annual progress reports. The quality of the information was good. The ICR (para 99) also highlights that throughout the project's lifetime, all indicators included in the results framework were routinely updated and reported in the six-monthly Implementation Status and Results (ISRs). The Bank task team slightly revised the results framework during the project restructuring, mainly to adjust the indicator targets. They did not change the target for the indicator on the number of project beneficiaries, however. The project carried out two impact evaluations as mentioned above. c. M&E Utilization The M&E information was used to monitor progress in project implementation, towards the achievement of the PDO, and the safeguards compliance. As an example, the ICR (para 99) mentions that the Bank task team identified problems with the implementation of the OPRC pilot through the reporting on the indicators and that this led to remedial actions and project restructuring. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as category B for environmental assessment purposes. The project's negative impacts were expected to be of small to medium scale, site-specific, and predominantly confined to the construction phase. The following safeguards policies were triggered: Environmental Assessment OP/BP4.01 and Involuntary Resettlement OP/BP4.12. The RD prepared an environmental and social management framework. The contractors used this framework to prepare the site-specific environmental management plans for each road section. The RD also Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) prepared a resettlement policy framework. The RD's environmental unit was primarily responsible to monitor and ensure safeguards compliance. In terms of environmental safeguards compliance, the major issue during project implementation was the poor control of the disposal of excess materials and construction waste. Although the environmental management plans addressed the disposal of materials and waste, several contractors struggled to apply adequate mitigation measures and minimize damage. The problem was aggravated by weaknesses in Georgia's regulatory framework and country-wide deficiencies in waste management infrastructure. The issues with disposal management was the reason why the environmental safeguards performance was rated moderately satisfactory during more than half of the project duration. At the request of the Bank task team, the RD largely addressed the issues through better control by the supervision consultants. The environmental safeguard compliance was rated satisfactory by project end (ICR, para 104). Social safeguard compliance was rated satisfactory during the duration of the project. Although the project was not expected to require land acquisitions or resettlements, the Involuntary Resettlement safeguards policy was triggered as a precaution. During project implementation, no land acquisition or physical displacement was necessary as it was possible to design all the works to be carried out within the existing rights of way (ICR, para 105). b. Fiduciary Compliance The ICR (para 106) mentions that financial management was rated satisfactory throughout project implementation. The project benefited from the financial management arrangements of the previous projects. The project provided consulting services to the Transport Reform and Rehabilitation Center, responsible for financial management, to strengthen their capacity further. The Bank task team pointed out that interim financial reports (IFRs) and financial audit reports were submitted on time, and that all financial audit report had unqualified opinions. Procurement was rated satisfactory in the beginning of project implementation. It was considered moderately satisfactory because of poor participation of bidders in the procurement processes of the first round of civil works (ICR, para 107). The project had also delays in procuring the OPRC, which was the first contract of this type in the country, and there was a need to terminate three work contracts due to the contractor's bankruptcy (ICR, paras 83 to 86). In August 2018, the Bank declared misprocurement because of procurement irregularities in two design and build contracts and imposed sanctions for collusion on two contractors and one individual (ICR, para 87). The Bank task team also lowered the procurement rating to moderately unsatisfactory. The RD implemented remedial actions and strengthened its procurement performance (ICR, paras 89 to 91). Nevertheless, the Bank task team kept the procurement rating as moderately unsatisfactory until the project closed. c. Unintended impacts (Positive or Negative) --- d. Other Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment The underachievement on the indicator of "km of roads Moderately managed under alternative Outcome Satisfactory Satisfactory contracting methods" is not evidence that the capacity of the RD was not strengthened Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The following lessons have mostly been derived and summarized from the ICR, with minor additions of the IEG evaluator: Introducing new contracting methods requires the adaption of the World Bank's standard contract modal to the local context. In addition, flexibility in adjusting and improving the contract scope on a continuous basis is needed. It also requires more time for bidding and implementation than conventional contracts, with which the RD and the construction industry are familiar. The implementation experience of the first OPRC in Georgia demonstrated the need for flexibility and the ability to adjust and improve the pilot at any stage of its implementation. Although one of the objectives of the pilot OPRC was to introduce a shift in the risk allocation between the government and the contractor, the shift moved too much risk to the contractor and the risk allocation had to be revised after the failure of the first bidding process. The redesign of the contract scope and the revision of the bidding documents resulted in a successful second bid, but delayed the project. In addition, even after this second attempt, the maintenance performance indicators needed to be refined after contract award because some of them were irrelevant or difficult to measure. Overall, this demonstrates the need to adapt the OPRC model in the World Bank’s standard bidding documents to the local context and circumstances, and to allocate adequate time for bidding and contract modifications. The implementation of an OPRC pilot requires specialized technical support and training to avoid delays. To implement an OPRC for the first time in a country is technically complex and it introduces new concepts to the sector. In this case, as already mentioned, the first attempt to procure the OPRC was unsuccessful. Although the RD benefited from consultancy services at the design and supervision stages and the advice of the Bank task team, they might have further benefited from having highly specialized third party technical support available at key stages of the pilot. The consultant could also have provided targeted training for key RD staff. Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) Strong ownership in the RD and efforts by the Bank team to recreate this ownership after management changes are vital to introduce OPRCs. The experience of the project highlights strong ownership of the RD to introduce an OPRC pilot is a critical factor in its successful implementation. It also demonstrated that there is a necessity for the Bank task team to proactively work with the client during management transitions. In this case, shortly after the start of the OPRC, changes in the RD management and at higher levels in the government required substantial time for the new leadership to better understand and appreciate the benefits of the pilot. This learning process stalled the implementation of the OPRC for several months. It only resumed when the new RD management showed ownership following the Bank team's efforts. Special attention to the construction industry and a careful market analysis are necessary when preparing bidding documents and cost estimates for new road works contracting methods. In this case, the OPRC bidding process was launched with insufficient attention to the capacity of the local construction industry and a careful market analysis. This led the bidding process to fail because the lowest bid was three times the cost estimate. The high prices were due to shortcomings in the cost estimates and insufficient familiarity of the bidders with the new contract type. A careful market analysis prior to piloting the OPRC would have been useful. This analysis should have considered the allocation of risks, which needed to be carefully described in the bidding documents and adequately priced in the cost estimates. The analysis could have been used to guide the procurement strategy and improve the risk allocation in the bidding documents. An early and more intensive dialogue with potential bidders could have helped them understand the new contract type and price their bids correctly. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is concise, neatly-structured, and generally well written. It provides a clear overview of the project's objectives, scope, and changes over time. The limited amount of "telling" pictures, a nice timeline of the World Bank involvement in secondary and local roads in Georgia, and some additional annexes, such as the one on the alignment of the PDO with the World Bank Country Partnership Framework, are useful. The ICR follows the Bank Guidance on Implementation Completion and Results Reports for Investment Project Financing Operations. It includes the project’s theory of change with the necessary assumptions. This theory of change lacks, however, the PDO outcomes, such as the "km of road under alternative contracting methods" and the "reduction in travel time". The ICR largely used the theory of change as guidance to present the achievement of the PDO. This presentation is results-focused and also includes the PDO outcomes. Because the PDO indicators had limitations, the ICR uses evidence from other studies and surveys to show the achievement of the subobjectives. There are, however, some weaknesses in the presentation of this evidence. For instance, the results of the impact evaluation as reported in para 32 are not fully clear. The ICR also cites Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Secondary and Local Roads Project (P122204) additional facts to evidence achievements, particularly for the third subobjective. This effort, although commendable, has also some presentational shortcomings. For example, the ICR does not manage to clearly link the evidence on road improvements in Kakheti in para 42 to the RD's capacity strengthening. The remaining parts of the ICR are comprehensive and complete, and the lessons are solid and informative. A minor shortcoming consists of not having mentioned the borrower contribution. The ICR would also have benefited from a reference to key financial management obligations, such as the regular and timely submission of IFRs and financial audit reports and the qualifications of the latter. However, it points out that the project's financial management was satisfactory throughout project implementation. a. Quality of ICR Rating Substantial Page 21 of 21