Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) Report Number: ICRR0022297 1. Project Data Project ID Project Name P132270 Central Asia Road Links - Phase 1 Country Practice Area(Lead) Central Asia Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-54300,IDA-H9340 30-Apr-2019 34,894,847.72 Bank Approval Date Closing Date (Actual) 22-Apr-2014 15-Nov-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 45,000,000.00 0.00 Revised Commitment 39,274,244.32 0.00 Actual 34,894,847.72 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Peter Nigel Freeman Victoria Alexeeva IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) as stated in the Financing Agreement (Schedule 1, page 5) and the Project Appraisal Document (PAD, page 6) was: "To increase transport connectivity between the Kyrgyz Republic and Tajikistan along priority cross- border road links in Batken Oblast, whilst supporting improvements in road operations and maintenance practices ". Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) This assessment is based on the two sub-objectives: (1) To increase transport connectivity between the Kyrgyz Republic and Tajikistan along priority cross-border road links in Batken Oblast: and (2) To support improvements in road operations and maintenance practices. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components This project was the first phase of the Central Asia Road Links Program (CARS - 1), aimed at increasing transport connectivity between neighboring countries in Central Asia along priority cross-border road links and supporting improvements in road management practices. The road sections financed under this project prioritized rehabilitation of the cross-border road-links between Kyrgyz Republic and the Republic of Tajikistan in Batken Oblast. There were three components (PAD, pages 8 -9). 1. Rehabilitation of Priority Road Sections in Batken Oblast. The estimated cost at appraisal was US$51.3 million (IDA financing of US$42.3 million). The team clarified that about US$5.3 million from this component was reallocated to component two. The actual cost was US$31.0 million. This component planned to finance the rehabilitation of 56 kilometers (km) of road sections in Batken Oblast, near the Osh - Batken and Isfana (OBI) road corridor. The road sections included: (i) the Isfana - Kairagagach - Madaniyat up to the border crossing (36.08 km): Batken - Tortkul (14.30 km): and the Batken - Kyzyl Bel/Guliston up to the border crossing (6.3 km). 2. Improvement of Road Operations and Maintenance Practices. The estimated cost at appraisal was US$1.7 million, fully IDA financed. The actual cost was US$8.20 million. The actual cost was higher due to the reallocation of spending from component one. This component planned to finance the provision of goods and consultants' services for improving road operations and maintenance practices. The road maintenance practices were to be improved, through establishing a Service Level Agreement between the Ministry of Transport and Roads (MOTR) of the Kyrgyz Republic and the OBI corridor Roads Department (UAD). Under this agreement, UAD was responsible for providing road maintenance (summer and winter) and other emergency works on the corridor. This component also planned to support the OBI RD and its Local Maintenance Units (DEPs) with technical assistance to OBI - UAD on international practices on maintenance techniques. 3. Project Management and Implementation. The estimated cost at appraisal was US$1.0 million, fully IDA financed. The actual cost was US$0.9 million. Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) This component planned to provide project management support through training activities, installing a financial management software and preparing financial audits. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. The estimated cost at appraisal was US$54.0 million. The team clarified that about US$3.5 million of the IDA credit and grant was cancelled on November 12, 2019. The actual cost was US$40.1 million. Project financing. The project was financed by a national IDA credit/ grant of US$20.5 million and a regional IDA credit of US$24.5 million. The total bank financing for the project was US$45.0 million. US$3.5 million of Bank financing for the project was cancelled. The amount disbursed at closure was US$40.1 million. There was parallel financing from the Asian Development Bank (ADB) for activities aimed at improving border crossing facilities and complementary corridor road links along the Osh - Khushad axis from the Japan International Cooperation Agency (JICA), the European Union (EU), the European Bank for International Reconstruction and Development (EBRD), the Islamic Development Bank and the Exim Bank for China. Borrower contribution. The borrower contribution was planned at US$7.0 million at appraisal. The team clarified that the borrower only contributed US$4.8 million. Dates. The project approved on April 22, 2014, became effective on December 14, 2014, and was scheduled to close on April 30, 2019. The project closed about seven months later on November 15, 2019. Other changes. There were five Level 2 restructurings. The first restructuring on May 30, 2015, extended the lapsed dated legal covenant in the Financing Agreement for employing a road engineer, a contract administrator specialist and a lawyer by six months (from March 14, 2015, to September 14, 2015), to provide additional time for the Investment Project Implementation Group (IPIG) to complete staff recruitment, which was delayed due to the late initiation and selection process by the IPIG. The second restructuring reallocated US$5.7 million from component one to component two activities, to extend a pilot on Weigh - In - Motion Systems (WIMS) equipment to three more locations. The third restructuring on March 6, 2019, extended the closing date by five months from April 30, 2019, to September 30, 2019, for completing the rehabilitation works on seven km of the road network that had been delayed due to severe weather conditions in 2018. The fourth restructuring on September 26, 2019, extended the closing date from September 30, 2019, to November 15, 2019, to allow additional time for the road maintenance equipment to be delivered and inspected before project closure. The fifth restructuring on November 13, 2019, cancelled the unutilized amount of US$3.5 million, and reallocated funding from component one to component two activities. Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) 3. Relevance of Objectives Rationale Country, regional and sector context. The Kyrgyz Republic, is one of the most remote and poorest countries in Central Asia, with an economy mainly dependent on gold and mineral production, steel, cotton and reexports. Being landlocked, the road transport mode plays a major role in the economy, with 90% of passengers and 60% of freight using this mode. Given this dependence, improving road links between the Kyrgyz Republic and Tajikistan and improving road maintenance practices was expected to contribute to the Kyrgyz Republic's potential to expand participation in trade and supply chains and encourage regional integration through an east-west horizontal road program to improve connectivity. Country and regional strategy. The Kyrgyz government's Mid Term Development Plan for 2018- 2020 identified improving road connectivity and managing road operations and road maintenance practices as priorities. The Government's strategy articulated in the Main Development Directions for the transport sector for 2016-2025 reiterated the need for improving transport connectivity, through closing infrastructure gaps and preserving road assets. At the regional level, the Transport and Trade Facilitation Strategy and Action Plan of the Central Asia region for 2008- 2017, endorsed by the Central Asia Regional Economic Cooperation (CAREC), identified rehabilitation of six regional strategic transport corridors, based on their potential impact on regional economic growth and poverty reduction. Bank strategy. The PDOs continue to be well aligned with the Bank strategy. At appraisal, the Country Partnership Strategy (CPS) for the Kyrgyz Republic for 2014- 2017, explicitly identified the need for sustaining growth, through diversifying exports, reducing economic distances, improving connectivity, and lowering road transport costs, both within the country and through regional connectivity. The PDOs are well aligned with two of the focus areas of the Country Partnership Framework (CPF) for 2019-2022. The focus area two of the CPF underscored the need for raising productivity through improving transport connectivity. The focus area three of the CPF identified the need for improving economic opportunities and building resilience through regional development. The Bank has executed other transport projects in the Kyrgyz Republic, including the National Republic Road Project. This project was the first phase of the regional Central Asia Road Links (CARs) Program, aimed at improving transport connectivity between the Kyrgyz Republic and Tajikistan. Though the PDOs to improve connectivity were important in the country and regional corridor contexts, the PDOs as formulated were relatively unambitious, compared with the Bank's usual criteria for a regional project (that is, having activities in two or more regional countries). The PDOs were focused on rehabilitating and improving maintenance for priority road links in Batken Oblast (a specific location in the Kyrgyz Republic). In this case, the Asian Development Bank (ADB) appears to have the responsibility for improving cross-border facilities. There is little discussion about the role of the other development partners in the ICR, even though this is relevant to the context of this phase 1 project. The timing of the project was not ideal given the instability in the region, manifested in Kyrgyz Republic's border disputes with Tajikistan. Rating Relevance TBL Rating Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To increase transport connectivity between the Kyrgyz Republic and Tajikistan along priority cross-border road links in Batken Oblast. Rationale Theory of change. The causal links between the project activities, outputs, and outcomes were logical. Rehabilitation of priority road sections through well-maintained physical infrastructure and enhanced cross-border facilitation were likely to lower transport costs and reduce travel times in Batken Oblast. These activities could plausibly be expected to improve connectivity between the Kyrgyz Republic and Tajikistan, and to the long-term development program of improving regional integration. However in this project, given that activities were confined to one oblast and that there were no activities associated with border-crossing procedures, the word connectivity might have been inappropriate. Outputs (ICR, pages 6 -7 and 27 -28).  56.40 km of main roads in the Batken Oblast were rehabilitated, slightly exceeding the target of 56 km.  Of the total classified road network of 420 km in Kyrgyz Republic, 78% of the roads were in good/fair condition when the project closed on November 2019 (as compared to 48% in November 2013). This was slightly short of the original target of 80%. Outcomes (ICR, pages 6 - 7 and 25 -26).  The activities were expected to lead to a 10% annual increase in freight volume through the relevant border crossing points (from 500,000 tons at the baseline to 550,000 tons at closure). The actual achieved value was substantially lower at 166,023 tons a year. The ICR (paragraph 23) notes this this indicator was not achieved due to the substantial overestimation of freight volume at the baseline. The ICR notes that there were also disputed border issues, which led to the government decree of not allowing trucks to cross during daytime through this route, although this was only in the first year of implementation. The ICR notes that the Bank did not revise the target value during implementation, despite the implementing agency's request to modify the target value. It is not clear from the ICR whether there was no historical data on freight volume through these border crossing points. In the absence of this data, there is no conclusive evidence on the extent to which the project aided in increasing freight volume on the project-intervened border crossing points.  The market accessibility index of Batken Oblast was expected to increase to 5.50 in November 2019 (from 4.60 at the baseline). (This index, developed by the Development Research Group (DEC) of the Bank for the Kyrgyz Republic, is a measure of potential economic connectivity between villages/towns and big cities). The ICR (paragraph 23) notes that the project documents contained no details on the Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) methodology followed in calculating the baseline for this index, and the staff of the Investment Project Implementation Group (IPIG) were not trained in calculating the index. The ICR notes that the ICR team attempted to calculate this index by reaching out to the DEC Group, but was unable to do so for lack of feedback from them. Although the market accessibility index uses poverty data, the project design did not include any indicators to assess the attribution of the wider program of the Central Asia Road Links program on poverty levels. Given that the PDO indicators envisioned at appraisal were not useable, traditional connectivity measures - decrease in travel time and cost on the project areas - was used as a proxy for monitoring transport connectivity. While the proxy would have been appropriate for a national project, travel time and cost, they were less so for a regional project.  The average travel time on Lot one road segment connecting Isfana with Kairagach decreased by 11.11% for service cars, 31.85% for pickups and 24.21% for cars between 2016 and 2019.  The average travel time on Lot two road segment connecting Batken - Kyzyl -Bel with Isfana decreased by 19% for service cars, 19% for pickups and 24% for cars between 2016 and 2019. There were no targets for these indicators.  Road user costs by truck along the project road sections decreased to 0.42 US$/ vehicle km from 0.52 US$/ vehicle km at the baseline, as targeted. Given that the indicators specified at appraisal were not usable, and decrease in user costs and travel time is necessary (although insufficient) for improving connectivity between adjacent countries, this review concludes that the project's contribution to realizing the PDO is modest. Rating Modest OBJECTIVE 2 Objective To support improvements in road operations and maintenance practices. Rationale Theory of change. The causal links between the project activities, outputs, and outcomes were logical. Providing maintenance equipment and establishing service level agreement between the OBI Roads department and the MOTR for summer and winter road maintenance and other emergency works on the OBI road segments, technical assistance on international practices in road maintenance, and review of regional standards on axle load limits, were likely to improve the government's capacity for maintaining and preserving the road assets. Outputs (ICR, pages 7 -9 and pages 27 -31).  Road maintenance equipment was procured for OBD- RD. Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270)  Technical assistance was provided for establishing the Service Level Agreement (SLA) between the OBI Road Department (OBI- RD) and the Ministry of Transport and Communication (MOTR). The Financing Agreement of the project had an effectiveness condition that specified that the OBI RD was to be responsible for road maintenance works along the OBI - Tajikistan road corridor as per the SLA (the aim of the SLA was to move away from the traditional form of force account works, that is, payments based on the consumed resources, and to link payments to results). An annual SLA was signed for undertaking road maintenance activities along the road corridor for only one year (from April 1, 2014, to March 31, 2015). The ICR (paragraph 82) noted that the signed SLA was of insufficient quality, did not precisely specify duties and responsibilities, had inconsistencies in defining service levels and did not express sanctions for non-compliance in monetary terms. The team clarified that the contract had not been renewed since March 31, 2015 for factors stated such as: (i) significant delays in handing over the newly constructed road sections to Local Maintenance Units (DEPs) that did not allow the DEPs to receive the required funding for maintenance; (ii) inadequate maintenance equipment with DEPs; and (iii) lack of appropriate legislation for the payment structure.  However, a strategic plan on transport control was developed as targeted, for formulating main directions in the axle- load control system. Based on the recommendations of this plan, the Ministry of Transport and Communication (MOTR) installed eight high-speed weight-in-motion systems (HSWIMS) to support the overload control mechanism and enforce axle load control. According to the clarifications provided by the team, this document was to serve as a base document by the MOTR in determining the important directions in axle load control. The ICR notes that when the project closed, the strategic plan document was still subject to modifications, and had not yet been approved by the MOTR. This said, legislation was approved on September 2, 2020, after the project closed, to enable the enforcement of axle load and vehicle dimension control. Outcomes (ICR, pages 7 - 8).  Annual budget allocation for routine maintenance for the road segments increased to US$242,489 in November 2019. This represented a 21% increase relative to the target of 200,000 US$. The ICR (paragraph 28) notes that while this represented a substantial increase, an analysis, conducted by an international consultant who supported the implementation of this component, concluded that this amount was insufficient for the level of maintenance as specified in the SLA. Nevertheless, it was a significant step forward.  A transport cost and travel time survey along the entire road corridor was conducted at closure, to assess the impact of the road maintenance activities, as per the practice of other transport projects, to gauge user satisfaction. The survey provided evidence that transport cost reduction and travel time reduction had been achieved in the project intervened areas. In sum, given that the annual budget allocation for road maintenance was increased, and a transport cost and travel time survey was conducted to gauge user satisfaction, efficacy of this objective is rated as substantial. Rating Substantial OVERALL EFF TBL Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) OBJ_TBL OVERALL EFFICACY Rationale Efficacy of the second objective - support improvement in road operations and maintenance - is substantial. Efficacy of the first objective - To increase transport connectivity between transport connectivity between the Krygyz Republic and Tajikistan along priority cross-border road links in Batken Oblast - is modest, as the proxies used for monitoring performance were inappropriate. However, it is likely that the road links completed under this project, along with the other links that are being rehabilitated as part of the regional program, is likely to improve transport connectivity at the regional level, and thereby eventually aid in increasing cross-border trade. Overall Efficacy Rating Substantial 5. Efficiency Economic analysis. A cost-benefit analysis was conducted using the Highway Development and Management Model (HDM- 4) model for road rehabilitation activities along the corridor. These activities accounted for 95% of the appraised estimate and 77% of the actual cost. The project benefits were assumed to come from savings in road maintenance, savings in vehicle operating costs, and travel time savings. The ICR (paragraphs 33 and 34) reports that actual costs of these activities were 5.3% higher and actual traffic flows were 22% lower than estimated at appraisal on the corridor. The ex-post Economic Internal Rate of Return (EIRR) was marginally lower at 16.2% as compared to the ex-ante EIRR of 16.6%, due to the higher than expected actual cost and lower than expected traffic flows along the corridor. Administrative and operational issues. There were several administrative and operational shortcomings. One, despite the preparatory work done in the context of the Bank-financed project, there were design changes (realigning the roads for bypassing disputed road segments with Tajikistan). These changes were due to the weak quality control oversight of the original design (ICR, paragraph 54). Two, key positions (financial manager, road engineer, environmental specialist and contract manager) remained unfulfilled for most of the implementation period. This was exacerbated by a high turnover of the MOTR leadership, with five consecutive ministers leading the Ministry during the implementation period. These factors hindered the implementation of civil works and institutional development activities. Three, while the design included technical assistance to the MOTR for preparing a plan for developing an axle load control system, the plan had not yet been effectively adopted by the MOTR when the project closed. Four, the original design had earmarked funds for procurement of road maintenance equipment. Most of these funds were diverted at government request for procuring equipment for High -Speed- Weigh-Motion (HSWIM) equipment at two pilot locations (later extended to three more locations). The actual cost of establishing the equipment was five times the original cost, due to a combination of factors, including technical design issues relating to the physical layout of the stations, technological challenges related to the absence of power and internet, system architecture and software components and to integrate the new HSWIM systems with the existing Low speed Weigh in Motion systems. Further, the diversion for funding for HSWIM equipment compromised the funds for procuring road maintenance equipment for most of the implementation period (although the goods were eventually procured near to project closing). There were then challenges associated with installing the HSWIM equipment, due to the Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) weak implementation capacity (although they were installed and operational by the time the project closed). And five, the design envisioned an annual user survey for assessing user satisfaction with the maintenance works on the rehabilitated roads. Efforts to institutionalize this application was undermined by the lack of financial and human capacity, and the survey had not been administered by project closure. In sum, efficiency is rated as modest, in view of the serious administrative and operational shortcomings during implementation. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 95.00 Appraisal  16.60  Not Applicable 77.00 ICR Estimate  16.20  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of PDOs is substantial. Efficiency is modest, due to the administrative and operational shortcomings. Efficacy of the second objective - support improvement in road operations and maintenance practices - is rated as substantial. Efficacy of the first objective - to increase transport connectivity between the Krygyz Republic and Tajikistan along priority between the Krygyz Republic along priority cross-border road links in Batken Oblast - is modest, because of the inappropriate proxies used for monitoring performance. However, given that the project activities did improve transport connectivity and this can be expected to increase cross- border trade, on balance overall outcome is rated as moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Political risk. There is risk to the activities financed under the project, given the political instability in the region and disputes over borders (over issues such as water for irrigation purposes and pasture grounds) between the Kyrgyz Republic and the Republic of Tajikistan (ICR, paragraph 88). The ICR also Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) notes that the likelihood of border issues is relatively high, given that the Kairagach/Madaniyat border crossing was closed during the first year of the project. Government risk. There is high risk to sustainability, given that there is still no secure basis for funding road maintenance activities. It is also unclear whether the government will implement the recommendations developed by the project for maintenance of the High Speed Weigh in Motion equipment installed under the project. 8. Assessment of Bank Performance a. Quality-at-Entry The implementation arrangements were appropriate, with the regional coordination mechanism of the Central Asia Regional Cooperation (CAREC) responsible for coordination, and the IPIG of the MOTR responsible for implementation (PAD, paragraphs 37 and 38). The IPIG and the MOTR had implemented Bank-financed projects and were familiar with the Bank's fiduciary requirements. Nevertheless this was seen as a high risk project. The preparation team leveraged financing from other donors for supporting complementary activities in the road corridor. Resources from the prior Bank financed project (the National Road Rehabilitation Project) aided in advance procurement for this project. Several risks were identified including substantial risk with governance, and even with the mitigation measures, the overall project risk was rated as substantial (PAD, page 13). Appropriate arrangements were made at appraisal for safeguards and fiduciary compliance (discussed in section 10). There were, however, shortcomings at Quality-at-Entry. One, regional dimensions for a project with regional IDA financing, were indirect involving other financiers. While the PAD (paragraph 29) noted that project activities were to include a review of regional standards and methods for enforcing axle load limits, it was not clear as to what was meant by regional standards - that is whether they referred to regional standards or bilateral standards between the Kyrgyz Republic and Tajikistan (given that the project interventions were only in the Kyrgyz Republic). Two, there were shortcomings in the result framework for a regional project and M&E design shortcomings, with some key indicators that were either unusable or inadequate for monitoring project performance. (discussed in section 9a). The expected freight volume was substantially overestimated and the target for the increase in road maintenance funding was set too low. The Bank performance at Quality-at-Entry is rated moderately satisfactory, given the difficult country situation and sector context, in which the project was prepared. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Formal supervision missions were held twice a year as per the norm. The support provided by the team aided in fiduciary and safeguards compliance (discussed in section 10). The team allocated resources at government request for installing High - Speed - Weigh - In Motion equipment (HSWIMS) that were Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) expected to have positive effects on sustainability. The government at times lacked ownership of the project but implemented the regulatory reforms recommended to enforce axle load and vehicle dimension control. There were significant supervision shortcomings, however. The lack of candor on the part of the supervision team undermined the ability of the Mid-Term Review (MTR) to make effective corrective proposals. The ICR (paragraph 57) acknowledges that problems faced by the project were well documented by the MTR. These problems included: (i) inadequate staff capacity and absence of key operational staff; (ii) adverse impacts on road maintenance activities in the wake of the Bank's decision, at the government's request, to divert funds earmarked for road maintenance for installing High-Speed Weigh - in -Motion (HSWIM) equipment (although the equipment was procured by project closing); (iii) challenges associated with installing the HSWIM equipment; (iv) challenges associated with administering a user survey; and (v) non usability and inadequacy of key outcome indicators (discussed in section 9a). Although these issues were known, and the Bank had disbursed only 20% of the IDA grant/credit, the development outcome was assessed as satisfactory by the supervision team. The change in Task Team Leader (TTL) shortly before completion posed a risk to continuity in project oversight. In view of these shortcomings, quality of supervision is rated as moderately unsatisfactory. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The theory of change adequately represented the logic framework, and how the project activities and outputs were to aid in realizing the PDOs. The Ministry of Transport and Communication (MOTC) was responsible for M&E. There were significant shortcomings in key outcome indicators, with some indicators unusable and some with modest targets. As an example of the former was the indicator for monitoring the sub-objective of improving transport connectivity between the Kyrgyz Republic and Tajikistan, through monitoring the freight volume relative to the baseline. This indicator had a significantly overestimated baseline value. While reducing the transport cost and reducing travel time are part of improving connectivity along a road corridor, they are not sufficient, given that freight movement would also depend on the cost and time associated with border crossing formalities. The indicator associated with improvements in the market accessibility index for the Kyrgyz Republic was unusable. The PDO indicators aimed at monitoring the sub-objective of improving road operations and maintenance practices also had shortcomings. Regarding the indicator aimed at monitoring the annual budget allocations for road maintenance activities. The amount allocated did exceed the target, but was insufficient. Moreover, the user satisfaction survey could not be administered. And compliance by the OBI Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) Road department should have been a key outcome indicator (and not an intermediate indicator), considering its importance in supporting the second sub-objective of the project. b. M&E Implementation The ICR (paragraph 60) does not provide details, but notes that M&E implementation was weak. Despite the weakness of the original M&E indicators, which were noted in the MTR, none of the indicators were revised. c. M&E Utilization The M&E system was used for the narrow goal of monitoring project performance. The ICR (paragraph 61) notes that the project M&E system did not fit into any wider M&E system, but may have laid the ground for follow-on projects within the broader Central Asia Road Links (CARs) initiative. Is sum, M&E is rated as modest, given the shortcomings in design and no attempts at revising the indicators during implementation. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was categorized as a Category B (partial assessment) model under World Bank safeguard policies. Two safeguard policies were triggered at appraisal: Environmental Assessment (OP/BP 4.01) and Involuntary Resettlement (OP/BP 4.12). (PAD, paragraph 56). Environmental Assessment. The PAD (paragraph 55) notes that the adverse environmental project impacts were expected to be minimal and site-specific (such as, air pollution from trucks, loss of vegetation, and disposal of soil waste). A site-specific Environment Impact Assessment (EIA) was prepared and publicly disclosed at appraisal (PAD, paragraph 56). The ICR (paragraph 64) states that there was compliance with environmental safeguards. The ICR also notes that there were two issues during implementation. One, there was an issue of non-compliance with spills, that was only addressed by the contractor during the defect liability period; and two, there was a sighting of an endangered "red list" snake at a construction site in 2018. A biodiversity assessment was conducted, and as a mitigation measure, additional culvert underpasses were constructed along the route to address the matter. Involuntary Resettlement. The PAD (paragraph 54) noted that there could be some permanent acquisition of small, private lands along the road sections. A draft Resettlement Policy Framework (RPF) was prepared and publicly disclosed at appraisal. The ICR (paragraph 65) notes that there was compliance with social safeguards. The ICR notes that the feasibility report for a road segment identified 29 households, which could be affected. Resettlement Action Plans were prepared and the MOTR addressed the issue by Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) preparing alternative designs, which excluded the need for land acquisition. In another road section where land acquisition was required the MOTR provided alternative land lots for the affected people. The ICR (paragraph 66) notes that Grievance Redressal Mechanisms (GRMs) were established both at the local and central levels to receive and resolve complaints. The ICR (paragraph 66) notes that 129 grievances filed under the project were successfully resolved. b. Fiduciary Compliance Financial management. A financial management assessment was conducted at appraisal. The assessment concluded that the financial management arrangements were satisfactory, and the financial risk was rated as moderate. In general, there was compliance with financial management. The ICR (paragraph 76) notes that the Investment Project Implementation Group (IPIG) complied with the financial covenants and unqualified audits were submitted in a timely fashion. The final audit for 2019 was submitted by the due date, and was deemed to be satisfactory by the Bank. Procurement. An assessment of the procurement arrangements of the Ministry of Transport and Communication (MOTC), and the Investment Project Implementation Group (IPIG) of the MOTC was conducted at appraisal. Both the MOTC and IPIG were familiar with Bank procedures. The procurement risk was rated as substantial, due to the size of the civil works and the country and sector environment. The ICR (paragraph 69) notes that IPIG was adequately staffed at appraisal. However, there were procurement delays in the last year due to decreased capacity and changes in the IPIG. The ICR notes that the Bank's procurement specialist in Bishkek provided support in addressing such procurement issues. The ICR notes that in the last year of implementation there were inconsistencies and overspending of the major consultancy contract for civil works, which necessitated an exceptional waiver from Bank management. There was no case of misprocurement during implementation. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) Quality of ICR --- Substantial 12. Lessons The ICR draws the following three main lessons from the experience of implementing this project, presented with some adaptation of language: 1. The regional dimensions of a project need to be fully considered for a project with regional IDA financing. The PDOs in this project aimed at improving connectivity between the Kyrgyz Republic and Tajikistan, along priority cross-border links in the Batken Oblast. However, all the priority links were in the Kyrgyz Republic. The border issues and performance of the two countries were not part of the project design, and the only evidence of cross-border collaboration or dialogue between the two countries was in the regional forum. 2. The absorptive capacity of the implementing agency or in the country context needs to be carefully considered at design. This project aimed at administering a user satisfaction survey through an Information Technology (IT) system. The implementing agency had neither the financial nor the human resources for administering such a survey. Use of standard sociological surveys and focus group discussions should be considered in countries with remote areas, where either the internet penetration is weak, or the local authorities do not have the capacity or resources to maintain high technology support. 3. The sustainability of project investments, especially in road sector projects, needs to be carefully framed at design. The High -Speed - Weigh -in Motion (HSWIM) equipment and road maintenance equipment are necessary to improve road operations and maintenance practices. While the original cost of this aspect was US$1.7 million by project completion, the actual cost was five times the original amount at US$8.2 million. Spending of this magnitude should be accompanied by activities that support a long-term view in managing road assets, including through axle load control and capacity building to maintain a functional Road Asset Management System (RAMS). 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well-written and provides a detailed overview of the project. It candidly discusses the issues that arose during implementation, and also candidly acknowledges the faults during supervision that undermined taking appropriate corrective actions following the Mid-Term Review. It uses proxy indicators, as the original indicators were not usable, but they fall short of measuring connectivity. Given the many shortcomings, the ratings in the ICR are unrealistically high. The quality of evidence and analysis is nevertheless aligned to the messages outlined in the ICR. The ICR draws good lessons from the experience of implementing this projects, Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Central Asia Road Links - Phase 1 (P132270) which are useful and based on evidence. A minor additional shortcoming is the excessive length. The main text of the ICR at 24 pages is greater than the recommended length of 15 pages. a. Quality of ICR Rating Substantial Page 15 of 15