Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) Report Number: ICRR0021778 1. Project Data Project ID Project Name P126321 KE-National Urban Transport Improvement Country Practice Area(Lead) Kenya Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-51400 31-Dec-2018 61,868,482.30 Bank Approval Date Closing Date (Actual) 02-Aug-2012 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 300,000,000.00 0.00 Revised Commitment 92,531,367.04 0.00 Actual 61,868,482.30 0.00 Prepared by Reviewed by ICR Review Coordinator Group Kavita Mathur Peter Nigel Freeman Victoria Alexeeva IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) The Project Development Objectives (PDOs) as stated in the Financing Agreement (page 6) and the Project Appraisal Document (PAD para 46) were to: (a) improve the efficiency of road transport along the Northern Corridor; (b) improve the institutional capacity and arrangements in the urban transport sub-sector; and (c) promote private sector participation in the operation, financing and management of transport systems. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components Component A: Support to Kenya National Highways Authority (KeNHA) to Upgrade the Urban Road Transport Infrastructure (cost at appraisal US$311.15 million, actual cost US$59.9 million). This component was expected to finance: (a) the expansion and upgrading the Northern Corridor road section through Nairobi from Jomo Kenyatta International Airport (JKIA) turnoff to Rironi road, as well as access roads; (b) construction of the Kisumu northern bypass road; (c) construction and rehabilitation of non-motorized transport facilities, including foot paths, cycle tracks, pedestrian bridges and underpasses; (d) feasibility and detailed engineering studies; and (e) KeNHA capacity building. Component B: Support to Kenya Urban Road Authority (KURA) and Kenya Railway Corporation (KRC) to Develop Selected Mass Transit Corridors (cost at appraisal US$80.46 million, actual cost US$50.15 million). This component consisted of two sub-components: Sub-component B1. Support to KURA (cost at appraisal US$64.83 million, actual cost US$48.13 million). This sub- component would finance: (a) feasibility studies, including detailed designs and preparation of bidding documents for selected Bus Rapid Transit (BRT) corridors; (b) technical assistance for the provision of public transport and associated services; (c) activities to improve traffic management; (d) implementation of regulatory reforms; (e) construction of the Meru bypass road; (f) capacity building of KURA’s staff; and (g) establishment of a program to address HIV/AIDS in all the roads contracts. Subcomponent B2. Support to KRC (cost at appraisal US$15.63 million, actual cost US$2.02 million). This sub-component would finance: (a) feasibility studies and detailed designs and preparation of the necessary bidding documents for construction of selected commuter rail systems in Nairobi and other major towns; (b) preparation of bidding and contract documents for the selection of private sector operators for commuter rail operations; and (c) building the capacity of KRC in IT, management of private sector involvement, planning, and management of contracts. Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) Component C: Institutional Strengthening and Capacity Building of the Ministry of Transport (MoT) and Ministry of Roads (MoR) (cost at appraisal US$21.50 million, actual cost US$14.42 million). This component consisted of two sub-components: Sub-component C1. Support to MoT (cost at appraisal US$16.90 million, actual cost US$12.32 million). This sub- component would support MoT in the implementation of selected activities of the Integrated National Transport Policy, including the establishment of the National Road Transport and Safety Authority; establishment and strengthening the capacity of the Nairobi Metropolitan Transport Authority; strengthening the capacity of the East Africa School of Aviation, promoting private sector participation in the aviation sub-sector; and building the capacity of the MoT staff. Sub-component C2. Support to MoR (cost at appraisal US$4.6 million, actual cost US$2.1 million). This sub-component would strengthening the capacity of MoR staff; build the capacity of the newly established National Construction Authority and assist in the preparation and implementation of regulations for the National Construction Authority Act (2011); and build the capacity of the newly established Engineers Board of Kenya and developing the implementing regulations for the Engineers Act (2011). Note: The cost at appraisal include physical contingencies of US$23.9 million and price contingencies of US$28.84 million) (PAD page 19. table 4). Actual cost is based on the revised project amount in US$ equivalent, not the actual disbursement amount in US$ at project closure (ICR page 8, footnote 5). The ICR does not discuss the rationale for using this. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The actual project cost was US$124.47 million, substantially lower than the appraisal estimate of US$413.11 million. This was mainly because Jomo Kenyatta International Airport (JKIA) - Southern bypass -James Gichuru sections was dropped from the project as the government was promoting public transport. Financing. The project was financed through an IDA Credit. The original credit amount was US$300 million. The actual disbursements were US$62.01 million at project closure (ICR para 12). The project was restructured on December 27, 2018 (level 2 restructuring) and the entire undisbursed amount of the IDA credit of US$207.47 million equivalent was cancelled. The gap between net commitment and actual disbursement in US$ is due to currency exchange rate variation. Co-financing. There was no co-financing. Borrower Contribution. At appraisal, the Borrower was expected to contribute US$113.11 million. The actual Borrower contribution was nil. Dates. The project was approved on August 2, 2012, became effective on December 24, 2012 and closed on December 31, 2018 as scheduled. Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) 3. Relevance of Objectives Rationale Country and Sector Context: In Kenya, the multiplicity of government agencies were involved in the provision of urban infrastructure and services. This posed a challenge in coordination, planning and execution of programs. There were as many as 15 organizations involved in urban transport with overlapping and contradictory mandates and responsibilities. The ineffective institutional structures and weak legal and regulatory framework diminished quality, reliability and safety for public transport users, particularly in urban areas. For Nairobi, urban sections of the Northern Corridor are important transport route to the Central Business District (CBD) and Jomo Kenyatta International Airport (JKIA). It is also a crucial artery for its land-locked neighbors. At appraisal, these sections were heavily congested and with rapid urbanization required capacity expansion. Alignment with Strategy: The project’s first and second objectives are consistent with outcome 1.2 and outcome 2 "Domain of Engagement 1: Competitiveness and Sustainability— Growth to Eradicate Poverty" of the current Bank Country Partnership Strategy for Kenya for FY14-20. Outcome 1.2 aims at achieving "enhanced logistics and distribution network and more efficient gateways" while outcome 2 aims to "strengthen planning and management of urban growth" (CPS page 38). The third objective is aligned with the outcome 3, focuses on improving the enabling environment for private investment. The PDOs are aligned with the Government of Kenya's long term development strategy “Vision 2030”, which promotes increased investments in infrastructure, including the development of an efficient public transport system and ICT as the key to improve productivity, promote competitiveness, improve connectivity and reduce travel time and cost saving, all of which are necessary for the transformation of Kenya into a middle-income country (ICR paras 2 and 14). However, after Board approval there was a change in the government’s priority towards promoting public transport and the development of Mass Rapid Transit System such as Bus Rapid Transit system. It became necessary to retrofit the project to respond to the changing priorities. However, the project was not restructured, and the first project objective was no longer aligned to the government strategy. Previous Sector Experience: This project was a result of Bank's re-engagement in the Kenya's urban transport sub-sector (PAD para 41). Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) Rating Relevance TBL Rating Modest 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Improve the efficiency of road transport along the Northern Corridor. Rationale The theory of change for this objective was that the expansion and upgrade of the northern corridor road sections, including the bypass, would reduce average travel time and vehicle operating costs. These activities and outputs would result in improvement in the efficiency of road transport along the Northern Corridor. Due to change in priority by the government focusing on promoting public transport, particularly including BRT aspects in enhancement of the JKIA-Rironi sections, the project design was revised and Jomo Kenyatta International Airport (JKIA)- Southern bypass-James Gichuru sections was dropped from the project. The project, therefore, covered only James Gichuru road junction -Rironi section (26 km) of the Jomo Kenyatta International Airport (JKIA) - Rironi road (53km). The project included preparatory BRT activities such as feasibility and detailed engineering designs, but these were not casually linked to the objective of improving the efficiency of road transport as no physical works were envisaged under the project. Outputs The following engineering/feasibility studies were carried out: (a) detailed engineering designs incorporating BRT and preparation of bidding documents for the Jomo Kenyatta International Airport (JKIA) - Likoni road junction/Southern Bypass road section (Lot 1) were completed; (b) a preliminary design for the Likoni/Southern Bypass road junction-James Gichuru road junction road section (Lot 2) was completed; Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) (c) a detailed engineering design of Kisumu Bypass was prepared and was under design review at the time of the writing of the ICR; and (d) final Intelligent Transportation System (ITS) design and associated Civil Works plan in Nairobi, including concept design for Traffic Management Center, were completed. At project closure, the construction of Meru Bypass was substantially completed (about 80%) and all but 5 km of the road was opened to traffic. The construction of Meru Bypass was a small contribution of the project as it represented 30% of the total project costs (ICR para 25). The civil works of James Gichuru-Rironi road section were only 12% completed. Outcomes Due to the limited physical progress of James Gichuru road junction-Rironi section and the dropping-off of the JKIA Southern bypass-James Gichuru sections, the two PDO level indicators (a) reduction in average travel time from Junction Jomo Kenyatta International Airport (JKIA)-Rironi road; and (b) reduction in vehicle operating costs on Junction JKIA-Rironi road, were not achieved. Regarding Meru bypass, as the ICR (para 18) notes, this activity has transformed the face of Meru town and eased traffic movement in and out of Meru town. The travel time during rush hour between Meru town (Central Business District) and Maua and Isiolo roads was reduced by 37% and 21% respectively. No targets were set. Rating Negligible OBJECTIVE 2 Objective Improve the institutional capacity and arrangements in the urban transport sub-sector. Rationale The theory of change for this objective is that the institutional and policy reforms targeted towards the urban sub-sector and the establishment and capacity building of the regulatory bodies would improve the institutional capacity and arrangement of the urban transport sub-sector. Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) With the major institutional changes in the structure of government, including merging the functions of roads and other modes of transport under one Ministry, and the change in government’s priority that focused on promoting public transport, the focus on the institutional capacity needs of the urban transport sector further gained importance. However, the project components were not revised to address this specifically. There were no outcome indicator to measure the achievement of this objective in the results framework. The only indicator was “improved institutional capacity in the urban transport sub-sector” and the unit of measurement was yes/no. In this section, outputs intermediate outcomes are discussed together. • The National Transport and Safety Authority (NTSA), a lead agency for road safety, was established on October 26, 2012. NTSA is fully operational. The institutional arrangements in the management of road safety has improved and road safety has been mainstreamed through the operationalization of NTSA. NTSA is providing a clear leadership and is coordinating with traffic enforcement officials. The ICR reports (para 19) that with the acquisition of traffic enforcement equipment and public awareness on road safety, there has been an increased compliance with the traffic law. All this has contributed to a reduction in fatalities on the Kenyan roads (ICR para 19). The number of road crashes along the Northern Corridor (JKIA-Rironi section) decreased by 30% and 22% in 2016 and 2017, respectively, exceeding the 15% target (ICR para 17). • The Nairobi Metropolitan Area Transport Authority (NaMATA) was established on February 2, 2017 to oversee the establishment of an integrated, efficient, effective and sustainable public transport system within the Nairobi Metropolitan Area. According to the ICR, the Board of Directors and acting Chief Executive Officer of NaMATA had been appointed and it was being operationalized. NaMATA's website provides full details of its functions but does not include staffing and budgeting information. • In Kenya, there was no law or authority to deal with cases of poor workmanship particularly in the building industry that led to deaths at construction sites or collapse of poorly constructed buildings and other structures such as bridges. Therefore, the National Construction Authority (NCA) was established in December 2011 as an oversight body for the construction industry. The ICR reports that the capacity of the newly created NCA was strengthened. However, it does not provide any details on how it was strengthened but notes that the NCA has contributed to the enforcement of building code; for the first-time contractors presiding over substandard structures that collapsed were taken to Court (para 19). • The Engineering Board of Kenya (EBK), a regulatory body for engineering practice was established in 2011. The project financed a customized Enterprise Resource Planning system, which enabled EBK to accept on-line application/registration of engineers. The on-line registration is expected to reduce registration time by 60% compared to the paper-based system. Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) • The capacity of Kenya Urban Roads Authority (KURA) to manage urban trunk roads was enhanced. KURA developed a master plan for Intelligent Transportation System (ITS) in Nairobi. The plan includes the analysis and preliminary designs of 102 critical junctions and a design for a traffic management center. The Korean Exim Bank is preparing a project to finance the implementation of the master plan. A draft BRT design framework was prepared and the feasibility study for BRT line 1 was completed. • The project supported Kenya Railways Corporation (KRC) to develop a commuter rail master plan for the Nairobi Metropolitan Area. The plan identifies the interventions necessary to improve the level of service on the commuter railway network as well as the regulatory, institutional and legal requirements. By project closure, the KRC had started implementing some of the recommendations that are critical for the revival of the commuter rail services in Nairobi (ICR para 19). • The installation of 3D simulator has increased the training capacity of traffic controller at the East Africa School of Aviation (EASA) from 27 to 54 trainee per session. International Civil Aviation Organization (ICAO) certified EASA as the 9th regional center of excellence. EASA can now conduct flight safety courses for safety inspectors in collaboration with ICAO, Singapore Aviation Academy, and the Incheon Aviation Academy of Korea. The school is attracting students from Africa and other regions. Although, the project supported the capacity building of urban transport institutions, the capacity of these institutions to manage the urban transport sector is still weak. Several efforts were initiated under this project – including the increased focus on urban transport and mass transit, and priority actions of the commuter rail master plan. These may easily stall without sustained engagement. The sustainability risks related to NaMATA are substantial because not all stakeholders have fully endorsed their roles (ICR para 76).Therefore, sustained capacity building will be essential to ensure NaMATA is able to fulfill its role coordinating and leading urban transport initiatives in Nairobi Metropolitan Area (NMA), including hosting dialogue among key stakeholders and establishing coordination and implementation mechanisms. The achievement of this objective is modest. Rating Modest OBJECTIVE 3 Objective Promote private sector participation in the operation, financing and management of transport systems. Rationale The theory of change for this objective assumes that the establishment of a PPP department in KeNHA and the preparatory works for BRT and commuter rail in Nairobi will promote private sector participation in the operation, financing and Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) management of transport systems. However, this would require regulatory changes and capacity building with respect to PPP, which the project was not addressing. The ICR reports (para 22) that the activities supporting this objective were partly implemented under another Bank financed project - Kenya Infrastructure Finance/PPP Project (P121019). A PPP department was established by KeNHA. The PPP department is functional and one road section of Nairobi–Nakuru–Mau Summit Highway was offered for tolling under PPP. The preparatory activity for PPP engagement for BRT and commuter rail had not been finalized by project closure. Since the project did not support any activities for the achievement of this objective, this objective is rated negligible. Rating Negligible OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale --- Overall Efficacy Rating Primary Reason Negligible Low achievement 5. Efficiency Economic Efficiency The ex-post economic analysis was done only for Meru bypass (80% of which was completed at ICR mission) and followed the same methodology at appraisal. Net benefits were computed using the Bank's Highway Development and Management Model (HDM-4), which simulated life-cycle conditions and costs and provided economic decision criteria for multiple road design and maintenance alternatives. Costs included the latest cost figures from the actual bids (and any amendments thereof) and estimated maintenance cost. Latest traffic data was used (ICR para 25). The expected benefits included vehicle operation costs savings and travel time savings. The ex-post Economic Rate of Return (ERR) over 20 years after the completion of construction was 30.7% compared to the ex-ante ERR of 17.2%. Since the physical progress of James Gichuru- Rironi road section was only 18% at ICR mission, no ERR was calculated. Cost and Time Overruns Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) The adoption of the new constitution and changes in the GoK’s priorities towards promoting public transport, particularly including BRT aspects in enhancement of JKIA-Rironi sections, impacted the major civil works planned under the project. The cost increases were mainly due to increases in the scope of work, such as additional road safety interventions and additional lanes for future BRT. This required retrofitting the project and resulted in inordinate delays to civil works and increased resettlement impacts and costs (ICR para 13). The cost of civil works for Meru Bypass and James Gichuru-Rironi section were substantially higher (44%) than estimated at appraisal (the cost increased from US$29.4 million at appraisal to US$42.2 million). There were implementation delays due to: • Delays in release of counterpart funds for the compensation of project affected persons (ICR para 13). • Major institutional changes in the structure of the Kenyan government as well as transport sector, including merging the functions of roads and other modes of transport under one Ministry (ICR para 13). Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re- estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  17.20  Not Applicable 30.00 ICR Estimate  30.70  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objectives is modest. The project did not achieve its objectives and the efficacy is rated negligible. The efficiency in the use of project resources is rated modest. Therefore, the overall outcome rating of the project is Unsatisfactory. a. Outcome Rating Unsatisfactory Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) 7. Risk to Development Outcome Institutional Risk. The project supported the establishment of National Transport and Safety Authority (NTSA) and Nairobi Metropolitan Area Transport Authority (NaMATA), and the institutional enhancement of Kenya National Highways Authority (KeNHA), Kenya Urban Roads Authority (KURA), Kenya Railways Corporation (KRC), East African School of Aviation (EASA), National Construction Authority (NCA), and Engineering Board of Kenya (EBK). Although, the capacity is weak, by project closure, except for NaMATA, all these institutions were operational. There are sustainability risks related to NaMATA because not all stakeholders have fully endorsed their roles (ICR para 76). 8. Assessment of Bank Performance a.Quality-at-Entry At the time of project appraisal, the Government of Kenya (GoK) was implementing a number of reforms across the transport sector (including in roads, aviation, railways, and maritime), with the support of the Bank. However, the urban transport was lagging behind the other sub-sectors and was facing institutional challenges such as insufficient staff capacity, inadequate framework for transport policy and planning, and lack of transport corridor management (PAD para 8). The project was designed taking these factors into account. The overall implementation risk was assessed as "high" because of the potential risks that included (i) uncertainties regarding possible changes in the institutional set-up in the transport sector and implementation arrangements arising from the execution of the newly adopted Constitution; (ii) the forthcoming elections and the transition to a new Government; and (iii) likely resistance from transporters and bus operators to the introduction of any new regulatory framework for public transport. The main shortcoming of the project design was the exclusion of the development of a Mass Rapid Transit System (MRTS) in Nairobi. At appraisal, an Integrated Transport Policy (INTP) was waiting for the Parliament’s endorsement (ICR para 6). One of the recommendations of INTP was the introduction of the MRTS. This was supported by a mass rapid transit study financed by the African Development Bank in 2011. The study identified nine corridors as potential BRT routes. At appraisal, the GoK requested the Bank to include the MRTS design in the project. However, MRTS was excluded from the project due to the complexity and time required to develop MRTS in Nairobi (ICR para 36). Component B included preparatory activities such as feasibility and detailed engineering designs. The civil works related to MRTS and/or public transport were to be included in a follow-on project. The ICR notes (para 37) that the decision to proceed to Board approval relied on the following assumptions: the final designs – including the BRT aspects – would be ready within a year and would cause only minimal implementation delays, and a follow-on project would cover the additional civil works. These assumptions were optimistic because BRT was at the concept stage in the Nairobi Metropolitan Area and there was limited urban transport capacity and experience in in the sector to plan a BRT operation. Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) After Board approval, in December 2012 the Kenyan Parliament endorsed the new Integrated National Transport Policy (INTP) and decided to prioritize the development of MRTS such as BRT systems. Therefore, it became necessary to retrofit the project to the new policy. Incorporating BRT into the JKIA-Rironi road design process adversely impacted the project's (a) implementation schedule, (b) total project cost, and (c) the scale of involuntary resettlement and land acquisition. The project should have been restructured (Level 1) immediately at this early stage, but this was not done (ICR para 69). Quality-at-Entry Rating Unsatisfactory b. Quality of supervision During supervision, the Bank team visited the project sites and prepared comprehensive Aide-Memoires. The location of the TTL in the country office provided daily support to the implementing agencies. The Bank also played a leadership role among the development partners in the urban transport sector (ICR para 71). However, there were some serious shortcomings during supervision: • The Bank team did not provide timely safeguards guidance to finalize the safeguards document for a Level 1 restructuring (ICR para 72). As mentioned above, although a Level 1 restructuring was planned, it was not carried out. • There were delays in the updating of the RAP. The updating of RAP for James Gichuru-Rironi section was agreed by October 2014, but the first section i.e. the Zambezi-Rironi section (out of three sections) was cleared only in August 2017. • There was a high turnover of project TTLs that adversely affect project implementation. For example, the second replacement of a TTL happened just after the government requested additional financing, and the third replacement was a few months after the concept review meeting for additional financing that advised project restructuring without additional financing. This resulted in delays in project implementation. • There were areas where the Bank could have been more proactive, such as (a) it could have found a solution for "insufficient and delayed counterpart funding", which was a widespread challenge for the Bank’s entire portfolio in Kenya; (b) addressed the delays in the procurement process of Kisumu Northern Bypass; and (c) it could have insisted that an M&E consultant be hired. Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) Quality of Supervision Rating Unsatisfactory Overall Bank Performance Rating Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The main outcome indicator for the first PDO "improve the efficiency of road transport along the Northern Corridor" was the reduction of travel time on the Northern Corridor from Junction Jomo Kenyatta International Airport (JKIA)- Rironi road. To improve precision, the indicator could have been more specific, reflecting selected road sections traversing through urban areas of three counties of Nairobi, Kisumu and Meru. The indicator for the second PDO "improve the institutional capacity and arrangements in the urban transport sub-sector" was an output indicator. It just recorded that Nairobi Metropolitan Transport Authority (NMTA) and National Road Transport and Safety Authority (RTSA) were established and functional. It did not measure how the institutional capacity was enhanced. Regarding the third PDO "promote private sector participation in the operation, financing and management of transport systems", the indicator was inadequate. As the project focused on preparatory work on Mass Rapid Transit System, offering of BRT and commuter rail services for PPP was premature for this project (ICR para 49). b. M&E Implementation The project planned to hire an M&E consultant from an accredited university in Kenya to (a) compile the data from each implementing agency, (b) collect additional data and information from various stakeholder groups, (c) analyze data and risks, (d) assess the progress of the project, and (e) recommend actions. However, there were significant delays in the recruitment of an M&E consultant and the consultant was never hired. Each implementing agency collected the relevant indicators, but there was no analytical input to improve performance (ICR para 50). c. M&E Utilization The ICR reports (para 51) that data collection was undertaken but due to absence of an M&E consultant the utilization of data and analysis was not carried out. Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) M&E Quality Rating Negligible 10. Other Issues a. Safeguards At appraisal, the project was classified as environment category B as the project activities involved rehabilitation/expansion of existing roads within the right of way and two bypasses that would not traverse natural habitats. The environmental impacts were expected to be moderate and reversible (ICR para 96). The following three safeguards policies were triggered: Environmental Assessment OP/BP 4.01, Involuntary Resettlement (OP/BP 4.12), and Physical Cultural Resources (OP/BP 4.10). However, incorporating BRT aspects into the JKIA-Rironi road section required greater resettlement areas and land acquisition, which could potentially result in significant, large scale and irreversible adverse environmental and social impacts and led to the re-classification of the project to Category A. While the updated Integrated Safeguards Data Sheet was approved and disclosed, the envisaged Level I Restructuring was not finalized (ICR para 53). Environmental assessment OP/BP 4.01. Potential environmental impacts expected from project activities during the construction phase were soil erosion and disturbance of water flows, water pollution, traffic disruption, noise, air and dust pollution and temporary disturbance of flora and fauna. All the Environmental and Social Impact Assessments (ESIAs) and Resettlement Action Plans (RAPs), including an Environmental Social Management Plan (ESMP) for roads sections to be rehabilitated or constructed, were prepared and disclosed in the Bank‘s InfoShop and in-country. The ICR reports (para 55) that both implementing agencies - Kenya National Highways Authority (KeNHA) and Kenya Urban Roads Authority (KURA) had adequate and competent staff for the implementation of the Environmental and Social safeguards. In addition, the supervision consultants and contractors for the project had full time Environmental Specialists and/or Health & Safety Officers (for contractors). A fatal accident along James Gichuru- Rironi section occurred on October 5, 2017. This was due to inadequate traffic management at the construction site. An investigation was conducted and an action plan was prepared and implemented adequately. However, by project closure the compensation to the family of the deceased victim had not taken place as the case was still pending before the magistrate’s court in Limuru. By project closure, slope protection works such as construction of gabions were also pending. These works were necessary given the hilly terrain and high rainfall conditions in the Kathita East bridge and other road sections of the Meru Bypass. Involuntary Resettlement (OP/BP 4.12). No land acquisition was anticipated at appraisal. The project was expected to affect 344 vendors (resulting in 1,322 project affected persons (PAD Annex 7). On December 13, 2018, the project affected persons on the James Gichuru road -Rironi section submitted a petition to the Bank alleging delay in payment for land Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) acquisition and associated compensation. Other issues raised in the petition include: (a) health and safety concerns; (b) dust emissions; (c) access to homes and businesses; and (d) restoration of community water connections. On request by PAPs, a Bank team met with the representatives of the project affected persons on December 19, 2018 and discussed the issues raised in the petition (ICR para 60). The ICR does not discuss how this was resolved. The total number of project affected persons were 2,236 of which 1,322 were compensated. There were still 914 outstanding project affected persons to be processed at project closure. According to the ICR (para 63), the GoK is obligated to fully implement the RAP and ESIA of the project. The World Bank conveyed this message to GoK in a letter dated March 5, 2019 as follows: “GoK is required to continue honoring safeguards commitments assumed before the cancellation of the credit and to implement ongoing safeguards activities - including but not limited to all resettlement action plans - in line with the disclosed Safeguards Instruments after the closing date for the credit. The Bank team will continue to provide periodic oversight as you implement the safeguards instruments.” Physical Cultural Resources (OP/BP 4.10). During implementation, public feedback led KeNHA to take steps to preserve an important cultural resource, the Sigona House, which could have been affected by the proposed service road to James Gichuru-Rironi Road. b. Fiduciary Compliance Financial Management. The financial management (FM) of the project was rated moderately satisfactory in the latest ISR. The implementing agencies maintained qualified accountants throughout implementation and the project-level budgeting, funds flows, internal controls, financial reporting and audit arrangements were adequate (ICR para 64). In general, quarterly IFRs and annual audit reports were submitted to the Bank within the stipulated timelines. However, during the initial years there was high turnover of project accountants, which resulted in delays in submission of quarterly IFRs. Some of the audit reports were qualified but the qualification was not on fiduciary issues. They related to implementation challenges such as delays in contract and outstanding payments/bills that attracted interest and penalties (ICR para 65). Procurement. The procurement performance was rated moderately satisfactory in the latest ISR. Although there was no mis- procurement, there were a number of procurement issues: (i) delays in processing of procurement documents; (ii) inadequate or incomplete engineering designs, land acquisition and compensation payments. These challenges during design stages spilled over to contracts implementation phases and induced inordinate extensions of time and cost overruns for various activities/contracts (ICR para 67). Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Unsatisfactory Unsatisfactory Bank Performance Unsatisfactory Unsatisfactory Quality of M&E Negligible Negligible Quality of ICR --- Modest 12. Lessons Several important lessons were selected from the ICR, with some adaptation of the language: • In countries where the sector policy environment is changing rapidly, but the degree of a new policy impact on the project design is unknown, the project design needs to be flexible or the Board approval could be postponed. The project experience shows that the impact of adoption of the Integrated National Transport Policy and the subsequent focus on mass transit on the design of the project was unknown until 2013. While the project restructuring was necessary at an earlier stage to respond to the shift in focus, the restructuring process was halted. When such situations occur, the project team needs to undertake an additional risk analysis, present the anticipated issues to management, and obtain guidance and support from the Bank’s Country Management Unit and GP management. • Urban transport projects are typically complex, involve a variety of stakeholders, and require longer planning and implementation periods. The existence of a lead agency and a champion for the project is critical for success, including for managing the diverse interests. Rigorous stakeholder analysis needs to be conducted, and institutional arrangements must be clarified and streamlined, either as part of the project or before the project commences. • Continuous engagement is critical to preserve the momentum of institutional and policy reforms in the urban transport sub-sector. Several efforts were initiated under this project – including the increased focus on urban transport and mass transit, and priority actions of the commuter rail master plan. These may easily stall without sustained engagement. Likewise, sustained capacity building will be essential to ensure the Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) Nairobi Metropolitan Area Transport Authority (NaMATA) is able to fulfill its role coordinating and leading urban transport initiatives in Nairobi Metropolitan Area (NMA), including hosting dialogue among key stakeholders and establishing coordination and implementation mechanisms. 13. Assessment Recommended? No 14. Comments on Quality of ICR On the whole the ICR was fairly candid and had sufficient information to capture the positive and negative aspects of the project. However, the level of detail and the clarity of analysis was not adequate. There were also shortcomings in the internal consistency. Quality of analysis. The ICR lacks adequate detail and clarity of arguments on several aspects. There is no clear argument as to why a level 1 restructuring was not carried out. Given the fundamental nature of the restructuring, the ICR does not provide critical reasoning on the fact that level 1 restructuring would have been the appropriate course of action, including a review of the results framework. The coverage of relevance of the PDO is cursory. In assessing the relevance of objectives, the ICR only mentions the project’s consistency with the Bank’s Country Partnership Framework (CPF) and Performance and Learning Review (PLR). The Bank ICR guidance for investment operations (para 35) clearly states that in addition to the extent to which an operation’s development objectives are consistent with the Bank CPF/CPS at closure, the evaluation considers whether the Bank’s implementation support was responsive to changing needs and the operation remained important to achieving CPF/CPS development objectives. Internal consistency. The narrative loosely supports the ratings. This is reflected in an overly positive spin, which is not in sync with the negligible performance. The text does not sufficiently emphasize the seriousness of the shortcomings that led to an unsatisfactory project. Also, there is a weak logical linking in some parts of the report, as for example, in the discussion of Bank performance (e.g., the supporting text does not really explain the effects of a high task team leader turnover (ICR, para 72)). The explanation of costs was not clear. The team explained that during the life of the project, IDA credit in US$ equivalent changed from US$300 million at appraisal to US$269 million at closing. The difference in revised amount of US$92.5 million equivalent and actual amount of US$61.8 million in the financing table in page 2 of the ICR is mainly because of the exchange rate fluctuations between the US$ and SDR. In both cases, the SDR figure for actual and revised amounts is the same at SDR 43,959,046. Because of the fluctuations in the exchange rate, the ICR (footnote 5 page 8) reports "Actual cost is based on the revised project amount in US$ equivalent, not the actual disbursement amount in US$ at project closure". Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-National Urban Transport Improvement (P126321) a. Quality of ICR Rating Modest Page 18 of 18