Approach Paper Mobile Metropolises: An IEG Evaluation of the World Bank’s Group Support for Urban Transport May 16, 2016 Background and Context Introduction to the Evaluation 1. By 2050, population growth and increasing urbanization are projected to add another 2.5 billion people to the world’s current urban population of 3.9 billioni. While cities are the main engineer for growth and many immigrants are better off by moving to the cities, urban life is not short of challenges which include but are not limited to congestion, pollution, poverty and inequality. To make cities inclusive, safe, resilient and sustainable, every urban component has a role to play. An urban transport system that provides an efficient movement of goods and people, including the disadvantaged, and facilitates access to jobs and socio-economic opportunities in a sustainable and equitable way could substantially contribute to the achievement of the World Bank Group’s (WBG) twin goals of eliminating extreme poverty by 2030 and boosting shared prosperity. Urban transport is defined as the set of all modes of moving people (and goods) within a defined urban area. The importance of urban transport development was confirmed by the Sustainable Development Goals adopted in September 2015, which have an explicit target 11.2 on urban transportii. The World Bank Group has been steadily increasing its support to the urban transport development agenda since the 1970s. However, comprehensive program reviews are rare, even at a regional level.iii In this context, the Independent Evaluation Group (IEG) is conducting an evaluation of the effectiveness of the WBG in supporting its client countries for developing urban transport that contributes to its accessibility, affordability, sustainability and accountability to promote urban-based inclusive growth. This evaluation will contribute to each of the three IEG Strategic Engagement Areas (SEAs): “Sustained Service Delivery for the Poor” (detailed below) through its treatment of urban transport service delivery to the poor and of behavioral change leading to modal shifting; “Environmental Sustainability” because urban transport can critically affect air pollution and greenhouse gas emissions in rapidly motorizing cities, and “Inclusive Growth” because urban transport links lower income workers to markets and can make cities more efficient engines of growth. Description of the context and issues (the world beyond WBG) 2. Cities are considered dominant contributors to growth, even in poor and middle-income countries: economic activities in urban areas account for 80 percent of GDP in more urban and industrialized countries and 50 percent in less developed countries.iv Thus, it is important that cities function efficiently to maximize their contribution to GDP. Urbanization brings with it tremendous efficiencies in terms of agglomeration economies. At the same time, the substantial growth of city population and economic activity, increasing urban density, and growing motorization place heavy demands on urban transport systems. Several key challenges are already obvious in the urban area around the world. 3. Congestion and Efficiency: With increasing urbanization and expanded private car ownership, city transportation systems are under pressure from a growing number of vehicles on the streets that impair mobility and create congestion problems. Slow travel speeds, traffic jams 1 and gridlock increase commuting time, reduce accessibility, and negatively affect productivity thus offsetting the economic benefits of urban agglomerations. Empirical studies suggest that traffic congestion in cities cost billions of dollars in wasted time and fuel.v 4. Climate Change, Pollution and health: Transport is a major source of pollution, accounting for half of global oil consumption and nearly 20 percent of world energy use, of which about 40 percent is used in urban transport.vi By 2025, under a business-as-usual scenario, urban transport emissions are estimated to increase two fold to nearly 1 billion annual tons of CO2 equivalent, and 90 percent of growth in urban transport emissions will come from private motorized travel. According to the World Health Organization, about half of the urban population monitored in 1600 cities across 91 countries is exposed to air pollution that is at least 2.5 times higher than the recommended level.vii Exposure to vehicle emissions, particularly in urban areas with high traffic volumes and high concentration of air pollutants, is associated with various adverse health conditions, including heart disease, stroke, chronic lung diseases, obesity and lower respiratory infections.viii Increasingly, climate change is also raising questions about the resilience of urban transport systems to its impact, including flooding, heat and extreme weather events.ix Shifting users to more sustainable modes of travel through supply, demand and adaptation measures is thus a vital mitigation strategy. 5. Safety: Road injuries are the world’s eighth-leading cause of death and the number one killer of young people aged 15 to 24.x The agglomeration of private passenger cars, pedestrians, bicyclists, often poor and inadequate public transport, including substantial informal transport, combined with weak regulation and traffic law enforcement, significantly increases safety risks in urban areas. Africa has the highest proportion of pedestrian and cyclist deaths at 43 percent of all road traffic deaths.xi In urban African cities, school children are known to be at a particularly high risk. Security is critically to system accessibility, and women especially can face harassment, violence and intimidation where security is inadequate.xii 6. Poverty and Inclusion: Urban poor and other vulnerable groups often lack access to basic services, including transportation. Spatial disconnection of the urban poor from jobs can be an important contributor to poverty. The unskilled are more vulnerable to such disconnections, especially where cities over time become more geographically decentralized.xiii Poor households bear a disproportionate impact of weak urban transport systems (see Starkey and Hine 2014 for literature review).xiv They have to spend a larger portion of their income on transport or they cannot afford motorized transport at all. Fewer transport choices for vulnerable groups, including women, limit access to higher-income generating opportunities (see Berg et al 2015 for the synthesis of empirical studies). Thus key questions relate to how accessible and affordable services are to the poor, and whether services are accountable to poor users. 7. Political Economy and Governance: Urban transport is critically shaped by political decision making and, as it involves the distribution of valuable benefits, is subject to the interaction of interest groups with political systems operating at the national, regional and municipal levels. Decisions about investments, pricing, subsidies, fees, routing and award of contracts may be subject to a variety of formal and informal influences and different degrees of public accountability and transparency.xv As in other areas of public policy and investment, the urban poor may have considerably less voice than other groups in shaping decisions. In addition, weak capacity of public institutions may limit service quality. 2 8. While urbanization is irreversible, what could be changed is the way the city organizes itself and its services, and how it will look like in the future. It is clear that with the current rate of motorization (use of private automobiles), growing travel demands, and dependence on private motor vehicles in cities, urban mobility and efficiency in most major cities is not sustainable – economically, socially, or environmentally. Solutions do exist for a wide variety of urban transport and planning challenges. These include integrating the urban land use plan with transport plan to reduce transport needs; providing targeted (demand-side) support to poor and vulnerable groups; Encouraging modal shifting from private cars to public transport systems through behavior change intervention such as providing incentives and regulations; providing resources (building BRT, metro line), and shifting mental models of transport users to encourage more social beneficial modal choices. It also means improving service delivery arrangements to better target intended beneficiaries; limiting car emissions by encouraging switching to hybrid and electric engines and car-sharing; and adopting new technologies like the use of big data and open data to help move cars faster. 9. Concrete actions have been taken by many governments and international institutions. The new Sustainable Development Goals (SDGs) demonstrate international recognition that society must urgently find solutions for sustained mobility and improved access, along with a substantial reduction of the negative socio-economic impacts of transport. Eight MDBs including the WBG have committed in COP21, including to invest $175 billion of loans and grants for more sustainable transport in developing countries by 2022, with $65 billion committed by 2015. Policies and regulations to manage travel demand and support sustainable and high capacity public transport options are being sought in urban areas in both developing and developed countries. There are about 200 cities around the world with metro systems,xvi and construction and expansion is continuing in many cities, including in China, India and Brazil. Bus Rapid Transit systems have been introduced in about 200 cities around the world as well; mostly in Latin America, Europe, and Asia.xvii During the last decade, BRTs and bus systems with BRT characteristics have been piloted in such African cities as Lagos, Dar es Salaam, Accra, Kumasi, Dakar, and Kampala. A stronger emphasis is now put on promoting bicycling and walking in urban environments. Good urban policies emphasize integrated planning of public transport and measures to promote more mixed land use with close proximity to housing, jobs, and green spaces. These take place not only in cities of the developed countries, such as Seoul, Toronto, Portland, but also in Curitiba and Guatemala City. London, Singapore, and Oslo, for example, have responded with congestion charges to regulate traffic in the city cores. The Chinese government adopted policies to prioritize public transportation; Shanghai resorted to such measures to restrict car ownership through auctioning of new car registrations. Solutions to manage travel demand are being pursued in cities that are gradually adopting parking policies, for example Kiev, Moscow, and Cairo. 10. In spite of these trends, there is still much to do. Currently, the role of public transport still constitutes a small fraction of urban trips, in particular in cities with a large share of unregulated private paratransit and informal operators. In most of Sub-Saharan Africa and poorer cities in South and Southeast Asia, public and non-motorized transport services are either inadequate or non-existent. Only a handful of Sub-Saharan Africa cities (such as Addis Ababa, Ethiopia; Abidjan, Côte d’Ivoire; and Ouagadougou, Burkina Faso) have institutionalized public bus services, which account for 25 to 35 percent of all motorized trips.xviii The complexity of issues reflects a large number of stakeholders in urban transport, including different levels and capacity of governments and institutions responsible for planning and management of urban transport, urban land use planning, transport providers and operators, private sector, various 3 transport network users, and urban residents. It may also reflects difficult political tradeoffs in public transport that local leaders don’t want to make. Solutions to create sustainable, efficient, inclusive, and affordable urban mobility are urgent. WBG Policies and Interventions The WBG’s strategy for urban transport development has evolved with client needs and its own evolving global development agenda. 11. Since the urban transport development program’s inception in early 1970s, the WBG has increased steadily in commitment value, especially in the past decade. In the 1980s, the World Bank’s Urban Transport sector strategy (1986) concentrated on economic and financial viability of urban transport systems. In this light, it stressed efficient management of existing transport capacity, good traffic management, and efficient pricing. The strategy discouraged subsidies, instead relying on competition and reduced regulation to afford consumers (and cities) with appropriate services. It questioned the value to the urban poor of capital-intensive projects that might not be cost effective in countries with limited resources. 12. In the 1990s, the Bank’s Sustainable Transport: Priorities for Sector Reform provided a review of the Bank’s overall transport sector experience, and recommended a broader Bank perspective, emphasizing the integrity of economic, social, and environmental dimensions of a sustainable transport policy. Its social dimension placed considerable emphasis on accessibility of transport systems to the poor and urged a balancing among multiple objectives including economic and financial sustainability, environmental and ecological sustainability, and social sustainability, with special emphasis on accessibility to the poor. Urban development strategy had begun to emphasize that the livability of cities depends on their being economically competitive, financially sustainable, well governed, and well managed. 13. The World Bank’s 2002 Sector Review, Cities on the Move, represented a further step forward in sophistication and complexity in several dimensions. It explicitly recognized that the World Bank Group did not operate in a first best world, and that political systems, limited administrative capacity, and subsidies were part of the reality in which the Bank must operate. It advocated an integrated approach to urban transport where land use planning and transport planning coordinated. It recognized the role of urban transport in contributing to poverty reduction both indirectly, through its impact on the city economy and hence on economic growth, and directly, through its impact on the daily needs of poor people. 14. Cities on the Move notes five trends in urban transport, and proposes WBG engagement in each to strengthen the focus on poverty reduction, including the first explicit embrace of “well-targeted subsidies of services essential to the poor”; as a second-best solution as part of a WB urban transport strategy; facilitate decentralization; mobilize private participation in supply; increase transport safety and security and protect the environment (see Attachment 6 for further details). In 2008, the World Bank Transport Sector Board produced “A Framework for Urban Transport Projects: Guidance for World Bank Staff” to clarify strategy based on the somewhat more literary Cities on the Move by providing a more “operationally oriented approach” to guide the Bank’s regional and country strategies and project design and to provide a basis for comparing and evaluating projects and their results. 4 15. Subsequent World Bank Group strategies have discussed urban transport but have not advanced or altered the agenda from Cities on the Move, which continues to be as the most recent (or prevailing) strategy document. Four relatively recent ones of relevance (described in Attachment 6) are: (i) Safe, Clean and Affordable: Transport for Development: The World Bank’s Transport Business Strategy for 2008-12 (2008); (ii) 2009 World Bank Sustainable Infrastructure Action Plan 2009-11; (iii)Toward a Green, Clean, and Resilient World for All: A World Bank Group Environment Strategy 2012-2016; (iv)Systems of Cities: Harnessing Urbanization for Growth and Poverty Alleviation: WBG Urban and Local Government Strategy (2013); (v) Cities at a Crossroads: Unlocking the Potential for Green Urban Transport (2014). 16. A review of several recent IFC Roadmaps found no strategic discussion of urban transport at the institutional level. However, IFC prepared regional transport strategy papers for the LAC and MENA Regions and supports PPPs. While MIGA is very involved in political risk guarantees in infrastructure, it has no strategic objectives for urban transport, approaching it on a case-by-case basis. IEG Evaluations relevant to the Urban Transport Evaluation 17. IEG carried out an evaluation “Improving Institutional Capability and Financial Viability to Sustain Transport: An Evaluation of World Bank Group Support since 2002 (March 2013). This evaluation assessed the effectiveness of World Bank Group support to countries in sustaining the provision of overall transport infrastructure and services focusing on the financial and institutional sustainability of a nation’s transport system. The Urban Transport evaluation will build on this Transport evaluation but will have different scope and focus especially when at the time of the Transport evaluation only a small number of urban transport operations were completed. 18. Seven other evaluations and two learning notes also included topics related to urban transport development: A Decade of Action in Transport: An Evaluation of World Bank Assistance to the Transport Sector, 1995–2005 (2007); Environmental Sustainability: An Evaluation of World Bank Group Support (2008); Improving Municipal Management for Cities to Succeed (2009); Gender and Development: An Evaluation of World Bank Support (2010); IEG Three-Phase Evaluation on Climate Change (2009, 2010, 2013); World Bank Group Support to Public-Private Partnerships: Lessons from Experience in Client Countries, FY02–12 (2013); Global Program Review of the WBG’s Partnership with the Global Environment Facility (GEF, 2013); Learning Note: Additional Financing for Transport and Information and Communication Technology (ICT) (2015); Learning Note: Making Roads Safer: Learning from the World Bank's Experience (2014). The summary of full findings from these evaluations and learning notes is in Attachment 9. 19. The team also notes the relevance of the FY17 IEG Evaluation of WBG’s support for management of air and water pollution (Environmental Management of Air and Water Resources). This evaluation will focus on local pollution issues impairing economic growth and effecting the lives of the poor. Given the complementarity of the two evaluations (and the importance of mass transit and NMT as pollution control strategies), the two teams will confer closely and regularly.  5 PORTFOLIO OF WBG INTERVENTIONS 20. In the urban transport sector, the WBG is engaged in both upstream policy work and downstream investment activities. The WBG’s portfolio is summarized in Table 1 below. The more detailed preliminary portfolio review is in Attachment 7. Table 1. WBG Urban Transport Portfolio (active or closed/matured FY2007-16) Number of WBG Net UT projects (AF is Commitment commitment Number of Number of Institution Core/Non- counted as part of Amount for UT Closed Active core the parent project) ($ million) ($ million) projects projects IFCAS Core 7 8.72 8.72 2 5 IFCIS** 20 9 11 Core 515.58 515.58 MIGA Core 6 762.9 762.9 1* 5 WB Core 114 16,594 11,799 47 67 Non-core 156 17,483 5,778 78 78 WB-ESW Core 54 Non-core 33 Total 390 35,364 19,405 138 163 Source: IEG based on Business Intelligence. * One project from MIGA status is “Not Active” will consider as closed. ** WBG commitment for one IFCIS project (strictly confidential) is not available and has not been reported. However the project has been included in the number of projects. 21. WBG analytic, policy and institutional work. These works are to support the upstream development in the area of urban transport. These work are usually in the format of Advisory Services and Analytics (ASA), the Reimbursable Advisory Services (RASs) and trust fund financed technical assistances, either as a stand-alone services or as a complement to lending programs. The Bank policy advice and ASA focused on a number of areas such as: urban transport institutional management; regulation, sector reform; interurban connectivity; institutions and governance to manage infrastructure; road safety; and private participation in infrastructure financing. 22. During the period FY07-16, the Bank delivered 87 Economic and Sector Works (ESWs) in urban transport. A majority, about 77 percent, were in middle income countries (with 41 percent in lower-middle income and 36 percent in upper-middle income countries). Only 6 percent were in low-income countries. 23. IFC accounted for seven Advisory Services (AS) projects with an expenditure of US$11.0 million. All were in middle income countries - 56 percent in lower-middle income countries and 33 percent in upper-middle countries. 24. WBG Lending portfolio. The portfolio for this study includes all urban transport projects that were either approved or reached closure or maturity during the 10-year period spanning FY2007-FY 2016 for a total of US$19.4 billion. 25. The World Bank approved 203 projects of which 93 were or will be closed by June 30, 2016. Within the same time period, an additional 67 projects approved prior to FY07 were also closed. In total, 160 projects were closed. So 270 World Bank transport financing projects are 6 included in the scope of this study, with the total commitment amount of US$17.6 billion. Urban transport financing peaked in FY2010 with 35 projects and US$4 billion in commitments – with the approval of the US$1.5 billion Mexico Framework for Green Growth Development Policy Financing (DPF) and US$0.65 billion Brazil Sao Paulo Metro Line 5 projects. About 88 percent of the projects are Investment Project Financing (IPF) with only 12 percent as DPFs. There is only one Program-for-Results project, approved in FY14 in Vietnam, the National Urban Development Program. Of a universe of 270 projects, 42 percent were mapped to the Transport Global Practice (GP) and 58 percent were not. 26. Projects are concentrated in middle income countries. 78 percent of projects and 85 percent of the commitment are in middle-income countries (MICs). This leaves 22 percent of projects and 15 percent of commitment value in low-income countries (LICs). Regionally, Latin America and the Caribbean (LAC) had the largest share of WB lending commitments (42 percent), followed by East Asia and the Pacific Region (EAP, 30 percent). Europe and Central Asia Region (ECA) was the lowest with 2 percent. 27. Four countries account for more than half of the urban transport commitments: China, Brazil, India, and Vietnam. 28. IFC had 27 (urban transport projects approved (among IFCIS and IFCAS), with a total commitment value of US$ 524.30 million. 58 percent of IFC commitments were in upper-middle income countries (26 percent were in high-income countries, followed by lower-middle income countries, 10 percent), while 0.47 percent of IFC commitments was in low-income countries. In terms of regional distribution, 74 percent of the IFC investments were in ECA, followed by LCR (10 percent) and Middle East and North America (MENA, 6.3 percent). Investments in Sub- Saharan Africa represented less than 0.5 percent of IFC commitments in urban Transport. 29. MIGA had only 6 projects with a combined guarantee value of US$ 762.9 million. MIGA guarantees were approved for lower and MICs, with 81 percent in upper-middle income countries (Turkey with five operations) and 19 percent in lower-middle income (Cote d’Ivoire with one operation). There were no MIGA operations in the SA, EAP, LAC and MENA regions. 30. Modal Characteristics of the World Bank Group Urban Transport Portfolio:  About 40 percent of the projects are focused on urban roads including upgrading, rehabilitation, or new constructions in major cities as well as smaller towns.  Around 30 percent supported bus operations and services, including bus rapid transit (BRT) systems.  The Bank provided investments and technical assistance in support of non-motorized transport (NMT) in 20 percent of UT financing projects.  The Bank also provided operational and technical support for subway systems and light rail operations and services in 10 percent of the projects.  Within the lending portfolio, the Bank has provided assistance in institutional strengthening, capacity building, policy advice and strengthening financing for urban transport in client countries. 31. The WBG’s partnerships tackling issues related to urban transport range from those supporting measures to mitigate climate change impact, such as the Global Environment Facility (GEF) and Clean Technology Fund (CTF); programs with broader infrastructure focus, such as 7 Public Private Infrastructure Advisory Facility (PPIAF); and the transport sector programs, such as the Sub- Saharan Africa Transport Program (SSATP) and the Global Road Safety Facility that deal with different challenges in the sector – urban mobility and accessibility, and road safety. The two largest programs supporting WBG in urban transport are the GEF and the CTF. Purpose, Objectives, and Audience Purpose and Objective 32. The purpose of this evaluation is to obtain evidence-based findings, develop broadly- applicable lessons, and propose recommendations that could enhance WBG’s effectiveness in supporting client countries achieve credible progress towards achieving the target of Sustainable Development Goals for urban transport. Stakeholders and Audience 33. The primary audience for this evaluation study are the World Bank Group’s Boards of Directors, management, and staff involved in urban transport operations. The stakeholders that can benefit from this study include the Bank Group’s client governments, multilateral and bilateral developmental banks and donors, the private sector, concerned civil society organizations, and the ultimate beneficiaries of urban transport services. Evaluation Questions and Coverage/Scope Specific questions to be answered by the evaluation 34. The overarching evaluation question is “To what extent has the World Bank Group supported sustainable urban transport development in client countries that contributed to city’s efficiency and economic growth; environmental quality; the welfare of the poor and vulnerable groups; and road/traffic safety?” 35. This broad question links to the theory of change in several ways, acknowledging the multiple (and potentially competing) objectives of the WBG in this sector, as well as the multiple dimensions of IEG major evaluations of WBG engagement and performance. Clearly, the multiple objectives can be seen as contributors to the twin goals of elimination of extreme poverty (growing urban economies have been prime vehicles for mobility out of poverty) and shared prosperity, with attention to the equity of access of disadvantaged and vulnerable groups, including the poor, women, and the disabled. It is important to note (and clear from both the literature and WBG strategy) that achieving urban economic efficiency and growth requires a careful balancing act between economies and diseconomies of agglomeration, which in turn necessitates a mix of transport and broader spatial planning. Environmental and road safety concerns affect the quality of life, critical to both goals. In fact, both environmental and poverty focus are signature features of the WBG strategy, at least since 2008, when it adapted a more holistic approach. 36. The key evaluation questions to be answered in this evaluation elaborate a main evaluation question with subordinate in the areas of relevance, effectiveness (including impact), efficiency, and work quality. Embedded in these questions are questions about how 8 models of behavioral change and service delivery to the poor were understood and applied by the WBG and its clients.  Relevance: To what extent has the World Bank Group’s support for urban transport been relevant to client countries (and cities) and their poor, female and other vulnerable populations’ priority needs, as well as to local priority? This question focuses attention on the economic and development rationale for WBG engagement in urban transport, WBG’s strategic objectives, WBG’s allocation of its resources and instruments in the context of country and city conditions, tailoring of interventions to local physical, institutional and policy conditions, adapting them to the needs of disadvantaged groups, allocation of resources across cities, and use of collaboration and partnerships where conditions merited it.  Effectiveness (Efficacy): To what extent has the World Bank Group been effective in achieving its objectives (improved accessibility and mobility; environmental sustainability; the welfare of poor, women and vulnerable groups; and road/traffic safety) with regard to urban transport development? This question focuses on how effectively interventions have enhanced city and country-level capabilities, improved regulation and oversight, improved accessibility and mobility, strengthened environmental sustainability, improved road safety and helped (enhanced the welfare of) the vulnerable (including the poor, women and the disabled). Behavioral change and improved service delivery are essential factors to ensure effective urban transport development. They are embedded in the main evaluation parameters (accessibility and mobility, sustainability, safety and inclusion). This and other evaluations in the Services SEA of IEG seeks to understand how interventions shape behaviors of transport users and improve urban transport service delivery through economic and noneconomic means. This evaluation will document the use, type, and outcome of such efforts to change urban transport users' behavior (modal shifting) and to improve the effectiveness of urban transport service delivery.  Efficiency: To what extent are World Bank Group interventions in urban transport efficient from both program and institutional perspectives? This question aims to elicit the extent to which WBG interventions (or the systems they supported) reached beneficiaries at a reasonable cost and were well-utilized and financially viable.  Work Quality: To what extent has the World Bank Group achieved high standards in managing factors within its control and coordinating its work internally and externally? This question focuses on how well the WBG designed, supported the implementation, executed the safeguard policies and tracked the results of its urban transport portfolio, and how well it used collaboration, coordination, or complementarities across the Bank group and with other players and partners. 37. As elaborated in Attachment 2, each of these questions has subordinate questions that must be answered in order to construct a full and detailed response. Although most subordinate questions will be examined across the portfolio, some questions may only be answered for countries where a case study or a mission is conducted. Each relies on data from a variety of sources, ranging from a literature review to user data collected at the system level, in particular urban areas. And each will require responses based on imperfect or incomplete data, often insufficiently disaggregated or parallel. 9 Assessing Performance 38. Attachment 2 also describes what the team anticipates or has already found as key sources of data as well as the limitations of those sources in providing the basis to assess the key characteristics of WBG performance. Among the several ways performance will be assessed are: 1. First, the team will analyze all available project level-evaluative evidence on closed projects in the portfolio over the evaluation period, including validated evidence from ICRRs, EvNotes on XPSRs and PCRs, PERs, and PPARs. In addition, it will examine patterns and trends in unvalidated self-evaluative evidence (including for Bank AAA work) but will appropriately caution readers about the potential biases and limitations of the source in characterizing such data. 2. Since all the WBG’s investment projects have indicators (at the outcome, intermediate outcome and output levels) to measure the project’s performance, the evaluation will try to generate a comparable body of project-level key performance indicators (KPIs) to assess the performance of urban transport intervention at output, intermediate outcome, and outcome levels. The study will generate a set of commonly used project-level KPI to the extent possible to enable comparison and aggregation. Recognizing limitations and inconsistencies in the underlying data, the KPIs exercise will be used as one of tools to assess the performance and effectiveness. 3. The team will conduct a broad literature review, including a review of the impact literature. The literature is diverse, often focusing on a single mode (e.g., BRT, NMT, or metro) or a single policy objective (e.g., modal shifting, pollution abatement, PPPs, or pricing). Integrating this diverse literature will not only provide public good, but also a perspective on the track record of many types of engagements and contexts of engagements familiar to the WBG. While much of the literature may provide only very limited explicit performance information on the WBG portfolio, it will provide evidence on the relevance of the approach and instruments and the effectiveness of different instruments and mechanisms in comparable contexts. The literature review may yield critical performance indicators for some cities and countries. In addition it may reveal gaps in the available body of knowledge. 4. Country case studies will provide direct evidence of performance in country context, and can lend insights on complementarities, synergies, coordination, and sequencing of work by the institutions of the WBG and partnerships with other donors. Country case studies will yield insights from field-level staff, clients at the national and subnational levels, beneficiaries, other donors and other stakeholders, and local experts. The selection method (see para.46) will assure a representation of different country conditions in terms of stage of development and urban density. 5. Partnerships: While the WBG urban transport projects co-financed by GEF and CTF will be part of the evaluation’s portfolio review, IEG will assess the support of two programs—the Sub- Saharan Africa Transport Program (SSATP) and the Public-Private Infrastructure Advisory Facility (PPIAF) — to the urban transport sector. These programs are selected as longstanding providers of technical assistance, capacity building and knowledge work in the urban transport sector. The SSATP focuses on facilitating policy development and related capacity-building in the transport sector of Africa since 1987. The program has made a stronger focus on connectivity, urban mobility and accessibility, and road safety in the last decade. The Public-Private Infrastructure Advisory Facility (PPIAF), while has relatively limited focus on urban transport sector, provides catalytic support to promote public private partnerships in the area. 10 6. The methodology of this assessment will be closely aligned with the evaluation framework of the main evaluation. The evaluation will assess how relevant and effective is the support provided by these programs; and how well the programs support the WBG to achieve its sector objectives and international commitments? In the case of PPIAF, it would be important to learn how well the PPIAF’s main objective of promoting PPPs has worked in urban transport sector and, what are the main lessons? The review will focus on activities of these two programs that are particularly pertinent to the objective of the main evaluation. 7. Working papers: Two working papers on behavior change (in urban transport service, water supply and sanitation service, and primary health service) and service delivery (in the said three areas) are being prepared in parallel by other IEG teams. Within the urban transport context, a behavioral change framework will be used to shed light on changing users’ behavior to prompt modal shifting; while a service delivery framework explores arrangements to ensure accessible, affordable, sustainable and accountable service delivery to the targeted beneficiaries. The theoretical background, methodological guidance and portfolio coding tools on behavior change and service delivery developed by these two working papers teams will be applied to guide relevant aspects of this. 39. By integrating these key sources of performance information, the team should produce insights into both how effective the WBG approach to urban transport is, but also where there is potential to improve its effectiveness, either by discontinuing lower performing approaches or by replicating higher performing ones. Scope 40. This evaluation covers the entire WBG portfolio (lending, AAA, investment, advisory, and guarantees) either approved or closed/matured over a ten-year period from FY2007 to FY2016, which means those urban transport projects approved prior to FY2007 but closed within the evaluation period will be assessed as well. Projects mapped to the sector and theme codes covering urban transport were screened. For the WB projects, the codes are TC (urban transport), TZ (General Transportation), BV (Public Administration-transportation) and theme code 102 (City-Wide infrastructure and Service Delivery). For IFC, activities with an Urban Transport code. For MIGA a keyword search yielded an initial portfolio identification. Those projects either with objectives or activities in urban transport development were included in the scope of evaluation. Objectives usually include improving the efficiency and effectiveness of urban transport system, improving the institutional capacity of planning, managing and operating the urban transport system, and improving the accessibility and mobility of the targeted beneficiaries including the urban poor and the vulnerable groups. The activities include providing urban transport infrastructure and multiple modes of urban transport services, as well as traffic management measures and institution capacity building and strengthening activities. It covers portfolio relating to public and private provision as well as public-private partnerships. It also covers urban transport for both people and goods (freight), although the WBG does little regarding urban freight. 41. However, it excludes a number of related topics: intercity and non-urban transport; urban planning and development activities not involving an UT component; environmental and safety regulations and policies not concerning UT; and efforts to extend non-UT services to the disadvantaged groups. 11 Evaluation Design and Evaluability Assessment Evaluation design 42. Theory of Change: (Figure 1) From the World Bank Group’s point of view, its activities and interventions seek to influence the policies and actions of national and subnational governments and the private sector to strengthen the enabling environment and transport systems including infrastructure. Overall, the intermediate outcome categories listed in Figure 1 are broadly amenable to definition, measurement, and attribution to WBG interventions. With respect to development outcomes and their contribution to the twin goals, while they can be clearly defined, measurement and attribution may be less clear or available (particularly in ways that lend themselves to attribution to urban transport activities) than in the case of intermediate outcomes. Therefore, for characterizing development outcomes, there will be greater reliance on the project level data, the existing impact evaluations, and the beneficiary survey to be carried out under the evaluation. The country case-studies will also attempt to identify development outcomes to which the WBG may have contributed, utilizing contribution analysis (see Attachment 3) to make reasonable connections between activities and outcomes controlling for a variety of other contributing or inhibiting factors. 43. The study will assess both quantitative and qualitative results data at the level of individual projects, investments, or guarantee operations through the portfolio review. The study will also draw upon the findings of field-based PPARs and country case studies for a sub-sample of countries and projects purposefully selected to represent all regions, multiple country income levels and important project characteristics, supplemented by beneficiary survey. 44. Evaluation Design Matrix. Each of the above-described methods and instruments will provide inputs to one or more aspects of the evaluation questions. The portfolio review will focus on answering the questions on the relevance of WBG’s engagement and the performance in urban transport sector, drawing from all evaluated and all IEG-validated projects. The PPARs answer questions on the results on the ground, the sustainability of the development outcomes and lessons based on a deeper assessment of select urban transport projects. The country case studies will answer the relevance, depth and the achievement of the WBG’s UT engagement at the country and city levels, while taking into account country and city context and a consideration of the roles plays by the other partners. Case studies provide a special opportunity to understand, not only context, but also the sequencing and coordination of activities and complementarities, overlaps and synergies of the activities of different actors. In some country case studies, surveys may be used to better understand quality and impact. In some cases there will be an overlap of questions addressed by different instruments in order to facilitate triangulation. To the extent possible, the transport users’ behavior change and urban transport service delivery related aspects and questions will be examined applying all the instruments. The interface of evaluation questions with the evaluative instruments is laid out in the Attachment 2. 45. Based on the evaluative approach described above, the evaluative analysis will draw upon several evaluative instruments (a) Project Portfolio Review; (b) Extended Literature Review; (c) Country Case Study; (d) Country Partnership/Assistance Strategy Review; (e) Beneficiary survey; and (f)) Staff, Government and Beneficiary Interviews. In the interest of space, the purpose and coverage of each instrument are elaborated in Attachment 3. 12 Sampling Strategy 46. Based on the criteria laid out in the “Scope” section above, (i) 270 WB investment projects, 27 IFC Investment service projects and 6 MIGA guarantees; (ii) 87 WB analytical activities and 7 IFC advisory service, were identified as UT portfolio and subject to the portfolio review. Among these projects, 114 investment projects mapped to Transport & ICT GP with activities coded as TC and TZ or those not mapped to Transport & ICT GP but with substantial investment, i.e., more than 50 percent of the project commitment in urban transport are considered as core UT projects. 156 investments not mapped to the T&I GP but supporting municipal government or the urban communities on urban services including urban transport service are classified as non-core projects. Figure 1: Urban Transport Evaluation Theory of Change Source: IEG based on WBG Strategy Documents. 47. PPAR selection: 125 (47 core and 78 non-core) urban transport projects were closed in FY2007-2016. Ten (six core and four non-core) projects have been selected for PPARs to be delivered in FY16-FY17 to feed into the UT study based on project features and regional distribution. The proposed PPAR list is attached in Annex 14. 48. Selection of country case studies: The WBG’s UT portfolio is spread among 88 countries. In selecting countries for case studies, IEG applied criteria which included the intensity of WBG’s support to urban transport development in the country; Status of the portfolio (counties selected should have closed projects to ensure a meaningful assessment of the portfolio performance); income level; urbanization level; urban transport development stage; relevant 13 policy, regulatory, and institutional conditions; level of private sector participation; diversity of instruments applied; and engagement of multiple WBG institutions, including financial commitments. Based on these criteria, 14 countries were selected for country case studies with five of them (China, Brazil, India, Turkey, and Uganda) selected for field missions and the remaining 9 of them for desk review. This includes two low income countries (with one as FCV country), five lower middle income countries and seven upper middle income countries. The list of countries selected for the country case studies is in Attachment 8. Together, the 14 countries’ portfolios encompass about two thirds of WBG’s commitment value in urban transport development. 49. Sampling of beneficiaries: Beneficiary surveys are envisioned to be carried out as part of the PPAR or country case studies in at least 2 countries. Such surveys would use stratified random sampling to assure that they capture respondents representing the poor, women and the disabled. The surveys will be associated with specific projects where project objectives included benefiting the poor, the women, or the disabled. Data requirements, Design Strengths, sources, and their limitations 50. It is envisaged that multiple sets of data are required for the study. Attachment 2 elaborates on data sources and likely challenges to collecting desired data and indicators, in full cognizance that some of the required data will be missing at each of four levels: the project level, the country level, the city level and the beneficiary level. Attachment 4 further details data sources and their limitations for each of these levels. Quality Assurance Process 51. The Urban Transport evaluation will adhere to recommended quality assurance processes.  In the pre-approach paper stage, key stakeholders in WBG management were consulted in WB, IFC, and MIGA.  A preliminary portfolio review has been conducted and is reflected in the approach paper, along with the criteria for portfolio selection.  The (lending) portfolio review, internal and external literature review, CAS/CPS review etc. will focus on the same evaluation parameters though they have different scopes. Constant quality check and team meetings will ensure the consistency and that the entire evaluation team has the same understanding of the evluation questions and methodology.  During AP preparation, the team has consulted with IEG Gender Advisors, IEGKC, and other experts on key aspects of the evalaution.  The team will coordinate with the Service Delivery SEA and will confer actively with the teams responsible for the Service Delivery and Behavior Change working papers.  The AP has been drafted in accordance with the IEG Quality Assurance Standards for Approach Papers. The team participated in an Approach Paper Clinic during the preparation in order to better assure conformity with quality standards.  Peer reviewers of notable expertise have been identified and engaged.  An advisory panel will be used during the lifespan of the evaluation. Attachment 5 identifies the peer reviewers and advisory panel members and their qualifications.  Upon appointment of an IEG Methods Advisor, the team will consult and actively communicate about methods and their application. 14  After the approach paper is cleared, the team will consult management on the identified portfolio to assure that important elements have not been missed. The team will maintain a periodic dialogue with management counterparts as the evaluation progresses. Expected Outputs and Outreach Planned Reporting Vehicles 52. The primary output of the evaluation will be a report to the World Bank Group Board of Directors’ Committee on Development Effectiveness (CODE). The UT evaluation will be published and disseminated both internally and externally. In addition, IEG will develop presentations, quick notes, blogs, videos, an infographic, and other products as appropriate for different audiences, including WBG staff (particularly those working on urban transport), client countries, donors, and NGOs. Working papers or follow-on learning products are also possible. The final report, along with the Water Supply and Sanitation Report, and the behavior change and service delivery working papers will feed into a synthesis report for the Service Delivery SEA. The team hopes to make the full range of these outputs available to interested stakeholders. 53. During the evaluation preparation, the team will solicit feedback and comments from stakeholders, in particular World Bank Group management, the Urban Mobility Global Solution Group and urban transport practitioners, clients and stakeholders in industries, and government agencies in client countries to improve the evaluation’s accuracy and relevance. The team intends to use social media to reach a broader group of stakeholders during the evaluation. In both cases, the interlocutors in these processes would form a target audience for outputs through a variety of channels. Outreach Strategy 54. To maximize the value and use of findings and recommendations to strengthen development outcomes, IEG will implement an outreach plan during the evaluation and after the completion of the evaluation. IEG will launch the report both in Washington, DC, and abroad. The events will target key stakeholders, including staff at headquarters and country offices, other multilateral development banks and donors, government authorities, civil society organizations, counterpart officials and other donors. Internally, the team would reach out directly to the Urban Mobility Global Solution Group members in the WB and to relevant practitioners in IFC and MIGA via various platforms including the annual Transforming Transportation event co-hosted by the World Bank and the World Bank’s Transport and ICT Forum. Externally, IEG will seek a venue for an international launch or a set of venues for a “road show”. Possible dissemination venues beyond those hosted by IEG could include global events such as the annual Congress of Cities for Mobility, the International Conference of Urban Transport (CODATU), the Cities Alliance Assembly, the International Conference on Transport and the Environment; or regional or country specific events. As discussed above, social media will be a key part of the strategy as well. Finally, the team would hope to reach out to other ECG members who may be conducting parallel or related work to seek opportunities for either dissemination or joint events. 15 Resources Timeline IEG Urban Transport Evaluation Timetable IEG Management Review of the Approach Paper – One stop March 2016 Approach paper sent to CODE April 2016 IEG Management Review of the Draft Report – One stop December 2016 Report sent to WBG Management Early February 2017 Report sent to CODE Late March 2017 CODE Meeting 4th Quarter FY17 Budget 55. The budget for this evaluation is estimated at US$1,175,500, with about 60 percent fixed cost and 40 percent as variable cost, consistent with major IEG sector evaluations. Team and Skills Mix 56. The core evaluation team includes the following IEG staff: Fang Xu, Senior Evaluation Officer – Co-Task Team Leader (Co-TTL); Andrew Stone, Lead Evaluation Officer – Co-Task Team Leader (Co-TTL); Kavita Mathur, Evaluation Officer (IEGSD); Anahit Anna Aghumian, Evaluation Officer; other IEGFP staff/consultant: Victoria Alexeeva, Zukhra Shaabdullaeva, Margareth Celse-L'Hoste, Francesco Bolognesi, and Thao Thi Phuong Nguyen (Consultants). Slobodan Mitric and Jose Gomez-Ibanez will provide advisory service to the team. Zhi Liu, Gerhard Menckhoff, and Monica Kerretts-Makau are the peer reviewers for the study. Two working paper teams from the Sustained Service Delivery for the Poor SEA will also contribute to the study. The team may be strengthened by additional sector experts in the course of the study. The evaluation will be conducted under the general supervision of Midori Makino, Manager (IEGSD) and Marvin Taylor-Dormond, Director (IEGSP). i United Nations Habitat.2014. World Urbanization Prospects: The 2014 Revision, Highlights. Department of Economic and Social Affairs, Population Division. ii SDG 11.2 states: “By 2030, provide access to safe, affordable, accessible and sustainable transport systems for all, improving road safety, notably by expanding public transport, with special attention to the needs of those in vulnerable situations, women, children, persons with disabilities and older persons.” iii In the last 15 years, only the East and Central Asia region carried out a 10-year review of its urban transport lending program (World Bank, 2002). More recently, there was a review of the Bank-wide urban transport program for lending operation approved in the period 1999-2009, but this was a narrowly conceived review, focusing on strategic aspects only (Mitric, 2012). iv World Bank Group.2009. Reshaping Economic Geography. World Development Report. Chapter 4: Scale Economies and Agglomeration. v Litman, T. 2015. Sma. Global Status Report on Road Safety 2015. v Thompson Reuters Foundation News Most Dangerous Transport Systems for Women October 31, 2014 http://news.trust.org/spotlight/most-dangerous-transport-systems-for-women/?tab=stories v Berg et al. Transport Policies and Development (World Bank Policy Research Working Paper 7366, 2015). 16 v Starkey, P. and Hine, J. 2015. Poverty and sustainable transport literature review. Partnership on Sustainable Low Carbon Transport (SLoCaT). v “Despite large expenditures on urban transport systems, the current transportation problems in developing nations continue to worsen because of bad planning, lack of governance, and corruption.” M. Masood et al. Transportation Problems in Developing Countries, Pakistan: A Case in Point International Journal of Business and Management, Vol. 6, No. 11; November 2011 v http://mic-ro.com/metro/table.html v http://brtdata.org/ v UN Habitat. 2013. Planning and Design for Sustainable Urban Mobility: Global Report on Human Settlements. rt Congestion Relief: Comprehensive Evaluation of Traffic Congestion Costs and Congestion Reduction Strategies. Victoria Transport Policy Institute. vi International Energy Agency.2013. A Tale of Renewed Cities. vii World Health Organization. 2014. Air quality deteriorating in many of the world’s cities. Press Release. viii World Bank Group and Institute of Health Metrics and Evaluation. 2014. Transport for Health. ix “Adaptation and mitigation of urban transportation systems to climate change can be defined as a form of risk management. …There are two fundamental options for risk management in the transportation sector. One is by mitigation measures in cities around the world that reduce globally the climate hazard factor that city and thereby reduce climate risk…. The second option in reducing risk is adaptation.” C. Rozenzweig et al. Climate Change and Cities: First Assessment of the Urban Climate Change Research Network. (Cambridge University Press, 2011) x World Health Organization, Op cit. xi World Health Organization. Global Status Report on Road Safety 2015. xii Thompson Reuters Foundation News Most Dangerous Transport Systems for Women October 31, 2014 http://news.trust.org/spotlight/most-dangerous-transport-systems-for-women/?tab=stories xiii Berg et al. Transport Policies and Development (World Bank Policy Research Working Paper 7366, 2015). xiv Starkey, P. and Hine, J. 2015. Poverty and sustainable transport literature review. Partnership on Sustainable Low Carbon Transport (SLoCaT). xv “Despite large expenditures on urban transport systems, the current transportation problems in developing nations continue to worsen because of bad planning, lack of governance, and corruption.” M. Masood et al. Transportation Problems in Developing Countries, Pakistan: A Case in Point International Journal of Business and Management, Vol. 6, No. 11; November 2011 xvi http://mic-ro.com/metro/table.html xvii http://brtdata.org/ xviii UN Habitat. 2013. Planning and Design for Sustainable Urban Mobility: Global Report on Human Settlements. 17 Attachment 1 References Ardila-Gomez and Ortegon-Sanchez 2016 “Sustainable Urban Transport Financing from the Sidewalk to the Subway.” Berg, C., Deichmann, U., Liu, Y., and Selod, H. 2015. Transport Policies and Development. Policy Research working Paper. World Bank Group. Washington, D.C. Bertaud, Alain. 2014. Cities as Labour Markets. NYU Working Paper Harry T. Dimitriou and Ralph Gakenheimer (editors) Urban Transport in the Developing World: A Handbook of Policy and Practice (Northampton, MA: Edward Elgar Publishing, Inc., 2011) Carlos DORA and Jamie HOSKING Urban Transport and Health: A Review JOURNEYS | November 2012 (Publication of the Land Transport Academy of Singapore) World Bank.2009. Reshaping Economic Geography Cervero, R. B. Linking urban transport and land use in developing countries. Journal of Transport and Land Use, [S.l.], v. 6, n. 1, p. 7-24, Apr. 2013.Kopp, Andreas Dietrich. 2015. Reducing Greenhouse Gases (GHG) analysis in transport. (Washington: Transport and ICT connections; no. 8: World Bank Group Susan Hanson and Genevieve Giuliano (editors) The Geography of Urban Transportation – Third Edition (New York: Guilford Press, 2004) Starkey, P. and Hine, J. 2014. Poverty and Sustainable Transport: How Transport Affects Poor People with Policy Implications for Poverty Reduction; A Literature Review Suzuki, H., Cervero, R. and K. Iuchi (2013) Transforming Cities with Transit: Transit and Land Use Integration for Sustainable Urban Development. The World Bank Group. United Nations.2014. World Urbanization Prospects. The 2014 Revision/Highlights. World Bank Group and Institute of Health Metrics and Evaluation. 2014. Transport for Health. World Bank Group.2009. Reshaping Economic Geography. World Development Report. Chapter 4: Scale Economies and Agglomeration. World Bank Group.2012. Inclusive Green Growth: The Pathway to Sustainable Development. Washington D.C. World Bank Group. 2013. Building Sustainability in an Urbanizing World: A Partnership Report. Washington D.C. World Bank Group. 2013. Planning, Connecting & Financing Cities – Now”: a comprehensive view of transport priorities and strategies in Urban sector projects. Washington D.C.World Bank Group. 2015. Competitive Cities for Jobs and Growth: What. Who, and How. Washington D.C. World Bank Group. 2015. Urbanization Reviews, Washington D.C. 18 Attachment 2: Evaluation Design Matrix Table 1. Key Evaluation Questions Key Questions Information Sources, Data Data limitations Analysis Methods To what extent has the World Bank Group been relevant, effective and efficient in supporting the sustainable urban transport development in client countries that contributed to city’s economic efficiency and growth; environment; the welfare of poor and vulnerable groups; and road/traffic safety? Relevance: To what extent has the World Bank Group’s support for urban transport been relevant to client countries and their poor populations’ priority needs, conditions and readiness for reform? 1. Given the body of evidence from the Bibliography assembled from Literature is not literature (including impact literature), what is diverse sources, including comprehensive, and there is a the economic/developmental rationale for online databases, expert sparse impact literature. World Bank Group to be active in urban references and bibliographies/ transport? references in prior UT work. Internal and external literature review, including impact literature 2. What are the World Bank Group’s strategic Corporate Level: Review of Corporate strategies since objectives with regard to urban transport at the all corporate and 2008 have been very weak on corporate and country levels? sector/thematic strategies with Urban Transport. Critical a. Did the WBG group commit itself to: UT content; Dialogue with strategy document established • Urban development (through a more Key Members of the GPs and a framework but did not efficient urban transport system) or VPs + Staff; Interviews of commit WBG to actions. • Urban transport development? Key Partners + Country Potential challenge is where Clients. Portfolio Review. other sectors (not transport or Country Level: CAS/ SCD urban) took on UT relevant /CPF Review for countries activities. selected for desk and field review + structured interviews of country teams. 3. What are World Bank’s, IFC’s and MIGA’s, Portfolio Review of AAA, Limited data at intervention and World Bank’s diagnostic and support Lending, Investment, (subproject) level on instruments for urban transport? How do they Advisory, TA, Guarantees. relevance and consistency relate to each institution’s corporate strategy? Field Visits protocols will with strategy How do they differ from each other, and are triangulate feedback from they consistent and complementary? Multiple Government a. How does the use of instruments/ agencies. Feedback from interventions (including modal choice, e.g. rail, project teams and BRT, road) vary according to country beneficiaries. Focus Groups conditions and strategy? with Thematic TTLs and b. How does the use of instruments Country Teams. (interventions) vary by institution and in response to relevant ESW? c. How did the use of instruments vary according to whether the interventions involved direct or indirect service delivery? d. How did interventions use incentive (e.g. subsidy/tax/service quality) and cognitive (e.g. public education) instruments to encourage behavior change to prompt modal shifting? Was an explicit, theoretically or empirically- 19 Key Questions Information Sources, Data Data limitations Analysis Methods based model of behavioral change used? Were specific behaviors identified and targeted? 4. Was the WBG’s advisory services Broad patterns of variance Patterns of complementarity, complementary to observable through portfolio synergy, and coordination/ investment/lending/guarantee operations and in review (e.g. IDA/FCS/ region) collaboration evident appropriate sequence? and micro-evaluations with primarily from case studies, details and strategic alignment not always evident through from case studies. Presence of portfolio or micro-evaluative prior ESW evident from evidence. project documents. 5. How strategically did the World Bank Group allocate its resources for urban transport a.b&c. City-level needs a.b and c. Depends on to countries? assessments for case studies, specificity of project a. To what extent did WBG city-level PPARs. documents. More detail for interventions match WBG assessments of city- d. City-level masterplans for IEG validated projects, case level need? case studies, PPARs. studies and PPARs. b. How well did the WBG factor in the e. Country level – urbanization context into its support to the superficially through portfolio urban development and reflect in its resources review, in depth where allocation to urban transport projects, i,e., the PPARs, case studies. already formed urban landscape hence involving additional cost for resettlement when urban transport intervention kicks in? c. To what extent did the assessments collect data directly from service users of citizen beneficiaries? d. Were interventions relevant to the priority of the cities? The priorities of urban transport service users? e. Did the Bank engage systematically and over time with countries and cities? f. To what extent did project design and implementation take into account political economy, local capacity (including financial capacity) and governance conditions? g. Was there a well-defined goal for behavioral change (modal shifting) behavior change) linked to outcomes? Portfolio review on Difficult to systematically interventions addressing judge timing and institution strengthening, appropriateness of capacity building. Country intervention, limited data. case studies and PPARs as well. 6. Did World Bank Groups interventions take Project beneficiary discussion Detailed beneficiary data, into account the specific constraints and the in project documents, disaggregated by key groups needs of the poor and other excluded groups, monitoring indicators, project of the excluded (gender, including women and the disabled? objectives. Evidence from disability status, income- a. Did the interventions utilize specific evaluated and validated level) are rare. Baseline data evidence of the transport needs and constraints projects on development with matching post- confronting the poor, women and the disabled? impact, beneficiary work in completion data even rarer. country case studies. 20 Key Questions Information Sources, Data Data limitations Analysis Methods b. Were these groups consulted in the preparation and implementation of the interventions? Portfolio review and drill down for case study countries. 7. How well did the Bank coordinate with the Partnership review, portfolio Limitations of information on other donors or partnerships in the UT review (reference to other donor activities, development in the client cities or countries? collaboration), drill down in response bias, labor intensive country case studies and nature of original data PPARs. Selective interviews gathering. of key partners. Effectiveness (Efficacy): To what extent has the World Bank Group been effective in achieving its objectives with regard to urban transport development? 1. To what extent have strategies, projects and ISRs, ICRs, ICRRs, PPARs, Many projects will not have project components that sought to build city or XPSR EvNotes, PCRs, PES, recorded impacts; IFC country-level capability and enabling PERs, Partnership Reviews. projects, Macro (PRSCs and conditions for urban transport achieved their Country Studies, Country DPLs) will have system stated objectives? Missions. Interviews, External effects that will need to be a. Have such interventions effectively Evaluation materials. measured differently than addressed deficiencies in planning and Portfolio Review direct Bank investments in regulation regimes? Country Case Studies UT.  Have they been coordinated with urban Interviews planning in terms of use of space? Focus Groups  Is there an optimal balance between public and private provisions? b. Did the interventions effectively improve the Lack of consistent time series performance of transport regulatory and data/indicators operational agencies? c. Did the interventions account for service providers’ needs in interacting with servicer users? d. Where there were multiple donors or actors, what was the contribution of the Bank’s intervention? 2. Have interventions in one or more services improved overall urban transport system a. KPI analysis at project performance? level: reduced travel time and a-e. Extent/limitations of a. How did the interventions improve the reduced travel cost (of users KPIs/ indicators data and efficiency of the cities either through improved of the WB financed transport other project-relevant data, mobility and accessibility of transport users? projects). and difficulty of attributing 1. Improved efficiency through 1 and 2. Country case studies, observed changes in broad reallocation of city activities? PPARs, ICRs and impact indicators to WBG actions. 2. Improved efficiency through behavior studies. change: modal shifting away from private cars? 3. Any diminished efficiency due to displacement, dislocation, or b. KPI analysis. inconvenience among existing service Environmental impact studies. Country case studies. 21 Key Questions Information Sources, Data Data limitations Analysis Methods users and squeezing out the existing service providers? c. Road safety KPI analysis, b. How did the WBG Group’s intervention safety impact studies, country improve the environmental sustainability of the case studies. cities through its intervention in the UT area as d. Urban data collected in measured by reduced GHG emission? relation to ICRs, PPARs and c. Did the WBG try to improve the resilience of country case studies, and/or the city through transport adaptation impact studies. intervention? d. How did the WBG group help improve road safety in the city through its intervention in UT development? e. How did the World Bank Group interventions contribute to urban economic growth and prosperity for all? f. What can be learned from cases where the implementation of systemic interventions was particularly successful or failed? Are these consistent with conclusions of the literature? To what extent was behavior change an important contributor to success or failure? Did the interventions have any unanticipated (positive or negative) consequences? 3. Has the intervention from the World Bank Micro evaluative evidence, Limited data and more Group on urban transport development benefit PPARs country case studies limited disaggregation by the poor and the vulnerable group? and city data for case studies. vulnerable group, gender, a. Have the interventions delivered affordable income, etc. and accessible services for the poor and the Beneficiary assessment survey vulnerable group? in at least two PPARs or case Challenges in high quality b. Was the welfare (health, household income, studies. administration, sampling, education, life quality, etc.) of the poor and the response rates in survey underserved improve as a result of better administration. accessibility and improved affordability? c. If the poorest or the vulnerable groups did not benefit from the project, which types of users (including by income category) benefit the most from the project interventions? d. What were the main barriers in reaching the poorest and the vulnerable? Were opportunities created for their participation in the design or implementation of the interventions? Efficiency: To what extent are World Bank Group interventions in urban transport efficient instruments, from both a program and institutional perspective? 1. To what extent has support to urban Mainly from relevant Information on the cost and transport reached beneficiaries at a reasonable coverage of project benefits, especially for the cost? Do some approaches exhibit greater cost- evaluations (ISRs, ICRs, poor or other disadvantaged efficiency than others? ICRRs, PPARs, XPSR groups, will be rare. 2. Are the activities financially sustainable? EvNotes, PCRs, PES, PERs), a. Are World Bank and all subsidized World PPARs and Country Case Bank Group activities meeting the target rate Studies. Impact studies. of return? After World Bank Group interventions, is there an enduring benefit in 22 Key Questions Information Sources, Data Data limitations Analysis Methods correcting market failure? Are operating subsidies budgeted and fiscally sustainable? b. What is the profitability of IFC and mainstream MIGA activities? Do they have an enduring effect on markets and systemic capacity? c. If there is PPP arrangement, was there an appropriate distribution of risk and return between the public sector and the private sector? d. To what extent were impacts leveraged through demonstration effects, multiplier effects or synergies with other simultaneous or sequenced activities? Work Quality: To what extent has the World Bank achieved high standards in managing factors within its control and coordinating its work internally and externally? 1. Is the World Bank Group effectively managing factors within its control? a. How well did country strategies reflect the a. Review of country Biases in self-evaluation of World Bank Group’s research and knowledge strategies for case study work quality. Limitations of about urban transport? countries. coverage and extent of b. Is the World Bank Group meeting its b. Work quality coverage of validated evaluations of work established work quality standards in validated project evaluations quality. preparation, implementation, and supervision? (ICRRs, XPSR Evnotes, How does performance vary by country PERs, PPARs, PCRs, etc.). conditions and the presence or absence of For IFC, also an examination complementary or prior interventions? of DOTS monitoring data for i. Did preparation incorporate service UT projects, its quality and delivery lessons of prior work in the sector? consistency across the ii. Were lessons of previous behavior change portfolio. interventions incorporated? iii. Did supervision use real time feedback to adapt project elements as required? c. Interviews with c. Are World Bank Group monitoring and management and evolution of reporting standards related to urban transport portfolio (projects, interventions adequate for accountability and instruments, mechanisms) learning? Is Bank Group management using over time, observation in the resulting data to improve performance and country case studies of outcomes? program evolution. Limited validated data. i. Was monitoring sufficiently timely and relevant for projects to adapt to information collected? d &e. Safeguards and ii. Did evaluative information inform strategy, Performance Standards Data activities and project design in UT? Extraction of E&S Lessons d. To what extent were environment and social from the Safeguards and (E & S) standards and safeguards applied and performance standards data as monitored? part of the portfolio review of e. How well did the WBG take into WB, IFC and MIGA. consideration the governance, political Interviews with E&S economy into its project’s design and Specialists that have implementation? supported UT activities. Lack of rigorous framework f. Did the WBG work with local Country Mission Protocols for evaluating coordination, counterparts/institutions to assure there was collaboration, partnerships 23 Key Questions Information Sources, Data Data limitations Analysis Methods adequate capacity to produce needed M & E will include work Quality and of relevant indicators. information for effective project questions. Subjectivity of reporting. implementation, self-correction and learning? g. To what extent was modal shifting (behavior change) measured through M & E? a., b. Case studies, PPARs, structured interviews with staff, possible small staff survey, and review of portfolio of projects involving multiple WBG institutions. 2. To what extent did the WBG use collaboration, coordination or complementarities across the Bank group and with other players? a. How good is the collaboration between GPs c. Inventory of major relevant (WB) and Departments (IFC)? partnerships and selected b. Are the three World Bank Group interviews with actors from institutions leveraging their comparative both sides (WBG and external advantages to achieve synergies through partners), plus case study adequate coordination and sequencing of interviews with clients, interventions? To what extent have stakeholders, field officials complementary interventions contributed and partner organizations. to the effectiveness of assistance? Has the d. Case studies, PPARs and presence or absence of multiple activities interviews. and/or sequenced activities influenced outcomes? c. Is the WBG using partnerships effectively to leverage its capabilities and resources and exploit comparative advantages? What are the other public and private sector players (including other donors) in this area in specific cities or countries? How well has WBG worked with other donors/ development partners? d. What can we learn from successful or failed World Bank Group coordination across the various units contributing to the urban transport agenda? Which mechanisms of coordination (shared strategy, shared projects, formal or informal communication, etc.) are most and least effective? Source: IEG. 24 Attachment 3: Elaboration of Major Instruments to be used in Evaluation Project Portfolio Review. The portfolio analysis covers WB financing, IFC investments, MIGA guarantees, WB advisory and analytic activities (AAA), and IFC advisory services in the urban transport area. The review will be based on the Project Appraisal Documents (PADs), IFC Board Papers, MIGA President’s Reports, Implementation Completion and Results Reports (ICRs), IEG’s ICR Reviews, Expanded Project Supervision Reports (XPSRs), Project Completion Reports (PCRs), Project Evaluation Reports (PERs), and Project Performance Assessment Report (PPARs). The broad evaluative categories of this scheme are:  accessibility and mobility related outputs and outcomes arising from physical asset creation;  regulatory and policy improvements;  institutional capacity building;  improving financial viability and improved city environment through project level climate mitigation and adaptation measures;  benefits to the poor and vulnerable groups through project design targeting at this group of beneficiaries.  improved safety in the form of reduced traffic and transit system accidents and mortalities. In addition, the portfolio review will shed light on M&E systems and environmental and social safeguards, as well as project design and supervision quality. While all the UT projects will be reviewed to understand relevance, characteristics and trends in the portfolio, evaluative emphasis for efficiency, effectiveness and work quality will focus on closed or mature projects and investments and, for complex or multi-component WB projects, those with a major focus on UT. For the purposes of this evaluation, only projects focusing on transportation within a single metropolitan area are considered – intercity transport is excluded, as are ports, airports and other facilities designed for transport to destinations outside urban boundaries. The WB portfolio review began using thematic and sector codes, which appear to be fairly consistently applied in the WB. In IFC and MIGA, after using an initial keyword identification, Management was actively consulted to identify the full list of projects. With regard to the WB, once the list if finalized, a text analytic software will be applied to capture any projects that have not been correctly coded as Urban Transport but which fit the profile. Following this, the full project list will be presented to Management for comments. Literature Review. This will consist of two key elements:  A review of WBG’s working papers, publications, and research papers related to urban transport development (including any impact studies);  A review of external academic papers, reports and publications (including impact evaluations and systematic reviews)xix from academic journals, multilateral banks, and other sources. The literature review will follow the same organization of issues as laid out for the Project Portfolio review as discussed above, but focusing more on the interrelationship between urban transport development and city development, what comprises an efficient urban transport system, and what are the known relationships between various interventions and outcomes.xx In addition, the sector strategies from DEC, the Regions, the Transport & ICT GP, the GPSURR GP and the Environment GP will be reviewed in greater depth to assess the relevance of WBG’s intervention in urban transport development. 25 Country Case Studies: To add to the richness of the country-focused portfolio review, five field-based country case studies will be prepared. The proposed countries reflect different country income levels; size and density levels of cities; policy, regulatory, and institutional conditions; private sector participation levels; and nature, depth and diversity of WBG engagement. While each case study will treat fully the country context and country program as context, attention will be focused on a more major intervention (or series of interventions) within a single urban area. So, for example, if Colombia is selected, interventions in Bogota would be the likely focus. Case studies provide several opportunities:  Context-specific information on whether and how interventions have worked or failed to work and how they responded to country constraints and conditions.  Insights into sequencing, complementarity (synergies), and coordination of WBG activities and interactions with government, donor and private sector initiatives.  Relationships between multiple institutions, including the institutions of the Bank Group and governments, donors, clients, key stakeholders, and beneficiaries. Case studies provide a platform for engaging with and learning from beneficiaries (especially the poor) about what interventions actually delivered.  Sustainability of benefits after project maturity.  Learning from experience and feedback loops.  Rich and detailed illustrations of broader evaluative findings. Country Strategy review will be an integral part of country case studies. The country strategy review will draw upon Country Assistance/Partnership Strategies (CAS/CPS), CAS/CPS completion Reports, IEG’s CAS/CPS Completion Report Reviews, country-focused studies, and country project documents for all countries covered by field-based or desk –based country case studies, going back 10 years. In the field-based country case studies for this evaluation, IEG will apply contribution analysis using a sequenced methodology to identify how WBG activities affected observed outcomes. After constructing a results chain for interventions in a given country, the team will make explicit the risks, assumptions, and other contributing factors that may influence observed outcomes and impact. Desk work and interviews will establish the initial story about the contribution of the Bank Group program in each country to observed changes in financial inclusion. The field missions will be used to validate this story and fill in missing information required to understand how other (positively and negatively) contribution factors came into play and how reasonable were the assumptions underlying the hypothesized results chain. In the end, the aim for each country is to have a highly credible and well-evidenced account of the contribution of the World Bank Group program to financial inclusion. PPARs: Additionally, the evaluation will benefit from at least seven field-based Project Performance Assessment Reports (PPARs) of which six will be carried out during the lifespan of this evaluation. For projects covered by PPARs, additional evidence that is available beyond the ICRRs will be highlighted in the analysis, and will may provide some basis to consider any limitations to the ICRR data. The PPAR projects have been selected to cover a variety of countries/regions and UT development issues, and the involvement of WB, IFC, and MIGA to identify the extent of synergies between the units. The team purposely avoided an overlap between the PPAR projects countries and the countries for field visit countries cases, but with the 26 aim of including the PPAR countries as part of the country case desk review, so as to maximize the utilization of the PPAR tool and maximize the coverage. Beneficiary surveys will be carried out as part of the PPARs or country case studies. The purpose of the beneficiary surveys is to assess the actual outcomes of World Bank supported urban transport projects in terms of mobility, accessibility, affordability, and economic well-being of beneficiaries. While these surveys cannot hope to provide a representative view of beneficiaries from the global program, in select cities, these surveys will provide on who actually benefits from projects (relative to project targets); distributional impacts on different income groups in the beneficiary populations; actual realized benefits to beneficiaries of intervention; user satisfaction (in terms of travel cost/ time/safety/ comfort); access and related economic opportunities created by access. In addition, the survey should shed light on barriers to those who did not benefit from interventions and may yield insights on how further interventions might overcome such barriers. The questionnaire for the surveys will capture both qualitative and quantitative variables. Once the questionnaires are developed and tested, the design of the surveys will entail five main steps:  For each selected project, the team will carry background research to help define the relevant sample based on urban population, urban geographic, economic and policy characteristic and on transport system characteristics relevant to WBG interventions.  The sample will target populations (including non-users of the supported services or systems) from a range of income groups. Stratification may also assure specific geographic coverage and coverage of disadvantaged groups (poor, women, disabled, etc.).  The questionnaire will be customized to local conditions, language and sophistication of likely respondents, but with certain variables that are comparable.  In each survey location, a local consultant (organization) will be contracted to assist in identifying the sample, testing the survey and implementing the survey in the field. IEG will oversee the consultant and participate in training of enumerators.  The consultant will be responsible for data entry into a consolidated database and quality control, with remote oversight by IEG. Partnership Review: While the WBG urban transport projects co-financed by GEF and CTF will be part of the evaluation’s portfolio review, IEG will assess the support of two programs— the Sub- Saharan Africa Transport Program (SSATP) and the Public-Private Infrastructure Advisory Facility (PPIAF) — to the urban transport sector. These programs are selected as longstanding providers of technical assistance, capacity building and knowledge work in the urban transport sector. The SSATP has facilitated policy development and related capacity- building in Africa’s transport sector since 1987. In the last decade, the program strengthened focus on connectivity, urban mobility and accessibility, and road safety. Although the Public- Private Infrastructure Advisory Facility (PPIAF) has a very limited focus on urban transport sector, it provides catalytic support to promote public private partnerships in the area. The methodology of this assessment will be closely aligned with the main evaluation framework, but focused on the World Bank Group’s role in and use of these programs in pursuing its Urban Transport work. This partnership review will involve a desk review of key documents, consultations with key informants. It will also draw on available evidence on results from any other available evaluative data. Where relevant, country visits for the main evaluation will be utilized to extract information on these programs as well. 27 57. In seeking to answer these questions, data collection will: (i) ensure that attention to partnerships and MDTFs is mainstreamed in interviews with staff, partners, and client representatives; (ii) employ a review of documents and strategies; (iii) include portfolio analysis; (iv) involve focused in-depth interviews with staff, donors, and partnership representatives; and (v) include a review of existing evaluations of partnership programs and MDTFs. Staff and Stakeholder Interviews: Interviews will be conducted with relevant WBG sector staff – technical specialists, task team leaders, researchers, and managers to tap their experience and gather their views on issues of evaluative interest as they apply to them. PPARs and country field case studies provide an opportunity to interview field-based Bank staff, government officials, civil society, researchers, local NGOs and experts, and beneficiaries. Structured meetings with groups of staff will be also be carried out during the course of the study, including during team “learning days” where relevant HQ staff can be consulted systematically early in the evaluative process. 28 Attachment 4: Data Sources and their Limitations 1. At project level, relevant information will be extracted from the Business Intelligence database. Project performance information will be acquired from the ICRR database. This set of data will be used for the UT portfolio review and to assess the project performance. However, the team recognizes both the imperfection of the dataset and missing variables (or variables that are not disaggregated to capture outcomes and impact for specific urban areas, systems or groups) groups) that are desirable with regard to understanding both outcomes and impact. 2. At country level, Country Assistance Strategy/Country Partnership Strategy and the national transport development strategy will be reviewed to assess the relevance of the Bank’s operation in the UT development area. CAS Reviews and CPS reviews will provide a certain amount of data, but are often missing important indicators of impact and disaggregated indicators for disadvantaged groups. 3. City level data including the municipality budget information, the city development strategy, the transport masterplan, the city population information and the transport related information such as road density, traffic will be collected to understand the contribution of the WB’s intervention to the urban transport development in specific city. The capacity and data limitations of many developing country cities is well-known and may constrain desired data collection at the urban level. 4. At the beneficiary level, data on the welfare of sampled beneficiary will be collected to understand whether and what actual impact of the Bank’s intervention had on the targeted beneficiaries. However, beyond the evaluation’s limited ability to survey beneficiaries or interview them, beneficiary data is rare and often aggregated to levels that does not allow differentiation by income level, gender, disability or other characteristic. 29 Attachment 5: Peer Reviewers and Advisory Panel Members a. Gerhard Menckhoff is an urban transport expert who has a career of experience in civil engineering or urban and other transport systems and services, working as a World Bank staff member from 1981 to 2000 and a consultant to WBG and GEF since then. b. Zhi Liu is the director of the China program at the Lincoln Institute of Land Policy and of the Peking University – Lincoln Institute Center for Urban Development and Land Policy in Beijing, China. Liu has been with the World Bank for 18 years, with operational experience in East Asia, South Asia, and Latin America, where he managed economic sector studies and investment lending projects in the infrastructure and urban sectors. c. Monica Kerretts-Makau is a Senior Faculty at Strathmore Business School, specializing in institutional management and governance, as well as leadership, education (and pedagogy) and ICT. Dr. Kerretts-Makau currently sits in the Advisory Board of the World Banks World Development Report 2016 focusing on the role of ICT and Development. An advisory panel will be used during the lifespan of the evaluation. One member has been secured and a second is being pursued: d. Slobodan Mitric is a distinguished expert in the area of Urban Transport, including 25 years at the World Bank as an Urban Transport Specialist (later Lead) and authoring a key WB strategy for Urban Transport in 2008. He has published extensively, including on the evaluation of urban transport projects. e. Prof. Jose Gomez-Ibanez is the Derek C. Bok Professor of Urban Planning and Public Policy at Harvard University, where he holds a joint appointment at the Graduate School of Design and the John F. Kennedy School of Government. He teaches courses in economics, infrastructure and transportation policy in both schools. He is a co-developer of the World Bank’s Leaders in Urban Transport (LUTP) training. His research interests are in transportation, infrastructure, and economic development. 30 Attachment 6: Details on Relevant WB Strategies 1. Cities on the Move notes five trends in urban transport, and proposes WBG engagement in each to: • Strengthen the focus on poverty reduction – emphasizing both a full assessment of the needs of the poor and an emphasis on supporting their favored modes and routes of transport services, including non-motorized transport (NMT). This appears to be the first explicit embrace of “well-targeted subsidies of services essential to the poor” as a second-best solution as part of a WB urban transport strategy; • Facilitate decentralization – focusing on the Bank’s role in supporting the design and implementation of city development strategies, in supporting needed institutional reforms including through program lending, supporting sustainable financing at the municipal level; and, where needed, supporting development of dedicated metropolitan • Mobilize private participation in supply – aimed at building country capacity for and facilitating transactions in and regulation of private concessions, franchises and other PPP arrangements as well as coordination of publicly and privately provided transport services. This involves a range of services from training and technical assistance to guarantees and investments. • Increase transport safety and security – aimed at confronting threats to public safety, health and security in urban transport through adequate provisions for NMT, policy coordination, project design features, and support for public transport reform, supported by WBG policy dialogue, AAA and investments. • Protect the Environment, emphasizing the need to promote environmental sustainability by balancing road development with promotion of other transport means, including NMT, public transit (BRT and urban rail where appropriate), balanced coordinated planning and pricing of services, support for technological innovations to reduce carbon emissions, and restraining demand for personal motorized vehicle transport. 2. In 2008, the World Bank Transport Sector Board produced “A Framework for Urban Transport Projects: Guidance for World Bank Staff” to clarify strategy based on the somewhat more literary Cities on the Move by providing a more “operationally oriented approach” to guide the Bank’s regional and country strategies and project design and to provide a basis for comparing and evaluating projects and their results. Setting a high bar for realism, it notes that urban transit is characterized by heterogeneous demand, institutional fragmentation, under- funding, and unresolved problems in defining public and private sector roles. It acknowledges that most projects aim to have impact on both economic growth and equity/inclusion goals, as well as environmental objectives. It takes an empirical approach (based on 8 case studies and a portfolio review) to conclude what are the major instruments and approaches at the policy, institutional and investment level. 3. Subsequent World Bank Group strategies have discussed urban transport but have not advanced or altered the agenda from Cities on the Move, which continues to be referenced as the most recent (or prevailing) strategy document. Three relatively recent ones of relevance are:  Safe, Clean and Affordable… Transport for Development: The World Bank’s Transport Business Strategy for 2008-12 (2008) 31 This transport sector strategy largely embraces the agenda of Cities on the Move. It explicitly calls for WBG to “encourage cities to seek service and value-for-money benefits by separating services specification from delivery” so that cities can rely on regulated private operators through contracts or concessions. It emphasizes competition, but also coordination between services to optimize use of limited transport space, and attention to safety and environmental impact. Unlike Cities on the Move, it does not elaborate a detailed WBG agenda on urban transport.  2009 World Bank Sustainable Infrastructure Action Plan 2009-11 This action plan called for cross-sectoral collaboration on transport regarding the climate change agenda and envisions an increased WBG engagement in urban transportation services with attention to: building capacity in administration and regulation; and enhancing the role and quality of affordable passenger and freight transport through financing mechanisms, PPPs, demand management for private cars, new technologies, and recognition of the importance of NMT.  Toward a Green, Clean, and Resilient World for All: A World Bank Group Environment Strategy 2012-2016 The WBG environment strategy highlights urban transport as a means to address environmental goals, citing BRT approvingly and calling for “smarter urban design and development”. The latter includes “smart approaches to urban transport to reduce congestion” but no WBG means to the end are detailed.  Systems of Cities Harnessing Urbanization for Growth and Poverty Alleviation: WBG Urban and Local Government Strategy (2013) Again, urban transport is discussed but a specific strategy is not elaborated. Urban transport is seen as important to urban development to “facilitate economies of scale and specialization”. It is described as “the backbone of an efficient city system” and intimately connected to challenges of congestion, mobility, safety and pollution. In its discussion of environment, a “holistic” approach is advocated, including “appropriate infrastructure for pedestrians, cyclists, and other non-motorized transport in city planning” and “creating a variety of housing and transportation options that would minimize motorized vehicle transport for low-income groups in accessing jobs.” 32 Attachment 7: Preliminary Portfolio Review In the urban transport sector, the WBG is engaged in both upstream policy work and downstream investment activities. 1. WBG analytic, policy and institutional work. These works are to support the upstream development in the area of urban transport. These work are usually in the format of Advisory Services and Analytics (ASA), either as a stand-alone services or as a complement to lending programs. The Bank’s ASAs are currently delivered through five product lines – Economic and Sector Work (ESW), Technical Assistance (TA), External Training (TE); Impact Evaluation (IE) and Programmatic Approach (PA). The Bank policy advice and ASA focused on a number of areas such areas as: urban transport institutional management; regulation, sector reform; interurban connectivity; institutions and governance to manage infrastructure; road safety; and private participation in infrastructure financing. 2. During the period FY07-16, the Bank delivered 87 ASA products in urban transport. A majority, about 77 percent, were in middle income countries (with 41 percent in lower-middle income and 36 percent in upper-middle income countries). Only 6 percent were in low-income countries. Regionally, about a third of the ASA were in East Asia and the Pacific (EAP), followed by 21 percent in Europe and Central Asia (ECA). Both Latin America and the Caribbean (LCR) and the Middle East and North Africa (MENA) regions received 16 percent of the total projects. Africa (AFR) and South Asia (SAR) regions each received 10 percent of projects. 3. IFC accounted for 7 Advisory Services (AS) projects with an expenditure of US$8.7 million. All were approved in middle income countries – 56 percent in lower-middle income countries and 44 percent in upper-middle countries. These IFCAS were approved in Jordan, Mexico, the Philippines, the West Bank and Gaza and Bhutan (see Figure 2). Figure 2: Regional Distribution of Advisory Services and ESWs (share of number of project) 100 90 80 Share of number of project 70 60 50 40 29 29 29 26 30 21 15 14 16 20 10 9 10 0 0 2 0 0 ESWs IFCAS ESWs IFCAS ESWs IFCAS ESWs IFCAS ESWs IFCAS ESWs IFCAS ESWs IFCAS EAP ECA SSA LAC MENA SA Other Region Source: IEG Portfolio Review 33 . WBG Lending portfolio 4. The portfolio for this study includes all urban transport projects that were either approved or reached closure or maturity during the ten year period spanning FY2007-FY 2016. During this period, the World Bank Group (WBG) approved or closed/matured 290 investment projects in the urban transport across the three institutions, World Bank Lending commitments,xxi IFC long- term financing and MIGA guarantees, for a total of US$19.4 billion. Figure 3. Trend in the World Bank Urban Transport Lending (Number of Projects and Commitment Amount US$ million) 4,000 35 3,924 3,000 22 27 2,000 22 1,740 1,784 18 1,364 1,359 17 1,000 661 19 23 18 798 1,312 1,152 2 196 0 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Source: IEG Portfolio Review, Bussiness Intelligence. 5. The World Bank approved 203 projects of which 93 were or will be closed by June 30, 2016. Within the same time period, an additional 67 projects approved prior to FY07 were also closed. In total, 160 projects were closed. So 270 World Bank transport financing projects are included in the scope of this study, with 110 active and 160 closed, with the total commitment amount of US$17.6 billion. Urban transport financing peaked in FY2010 with 35 projects and US$4 billion in commitments – with the approval of the US$1.5 billion Mexico Framework for Green Growth Development Policy Financing (DPF) and US$ 0.65 billion Brazil Sao Paulo Metro Line 5 projects. About 88 percent of the projects are Investment Project Financing (IPF) with only 12 percent as DPFs. There is only one Program-for-Results project, approved in FY14 in Vietnam, the National Urban Development Program. Of a universe of 270 projects, 42 percent were mapped to the Transport Global Practice (GP) and 58 percent were not. 6. In terms of the lending instrument type, majority of the projects were Sector Investment Loans (SILs). Only 4 percent of the projects were of emergency nature. There is only one Program-for-Results project approved in FY14 in Vietnam – National Urban Development Program (see Figure 4 below). 34 Figure 4. Urban Transport Lending by Lending Instrument Type by Commitment Value APL 3% DPL ERL 12% 4% IPF 13% SIL 68% P4R 0% Source: IEG Portfolio Review, Business Intelligence 7. Projects are concentrated in middle income countries. 78 percent of projects are in middle-income countries (MICs), divided evenly between lower-middle income and upper- middle income countries. In terms of the commitment value, 85 is in MICs – 50 percent in upper-middle income and 35 percent in lower-middle income countries. This leaves 22 percent of projects and 15 percent of commitment value in low-income countries (LICs). 8. Regionally, Latin America and the Caribbean had the largest share of WB lending commitments (42 percent), followed by East Asia and the Pacific Region (30 percent). Europe and Central Asia Region was the lowest with 2 percent. The key WB interventions in the regions include: a. In Latin America & Caribbean the Bank is supporting initiatives such as the implementation of affordable bus rapid transit systems targeted to the poor, the modernization of urban and suburban rail systems, the coordination of land-use and transport policies, as well as the introduction of sector reforms to reorganize the provision of services and reduce negative environmental impacts. The Bank is also involved in projects aiming at improving air quality and lowering greenhouse gas emissions throughout the region, using resources from the Global Environment Facility. b. In East Asia and the Pacific Region, the World Bank supported activities such as improved traffic management and bus rapid transit to improve the efficiency of urban transport. c. In South Asia Region, the on-going priorities include continued engagement in various megacities in the region (such as Mumbai, Chennai, and Dhaka), as well as in other medium-size cities in India and Pakistan, where controlling carbon emissions and factoring in the climate change agenda become critical. d. In Africa, the World Bank supported measures to improve bus service reliability, bus network reach, traffic management, infrastructure, and development of metropolitan transport authorities and of bus rapid transit systems. 9. Four countries account for more than 50 percent of the urban transport commitments: China (40 projects, commitment amount US$5.6 billion), Brazil (17 projects, commitment 35 amount US$3.4 billion), India (12 projects, commitment amount US$1.3 billion), and Vietnam (11 projects, commitment amount US$ 0.9 billion). 10. 20 IFCIS operations, representing USD 515.58 million of IFC commitments were approved in 10 countries and for the MENA region. 58 percent of IFCIS commitments were for upper-middle income countries; 26% for high-income countries and 10 percent for lower-middle income countries. Investments in low-income countries represented only 0.5 percent of IFCIS total commitments and the regional operation in MENA accounted for 5.7 percent of IFCIS total commitments. The countries with most IFCIS operations were the Russian Federation (5 projects) and Turkey (4 projects). Altogether, these two countries gathered 70 percent of IFCIS commitments (USD 362 million). The 9 other IFCIS projects were approved in Indonesia, Montenegro, Brazil, Colombia, Guatemala, Pakistan, India, Burkina Faso and for the MENA region, for a total of USD 153.47 million of IFC commitments. 11. MIGA had only 6 projects with a combined guarantee value of US$ 762.9 million. MIGA guarantees were approved for lower and MICs, with 81 percent in upper-middle income countries (Turkey with five operations) and 19 percent in lower-middle income (Cote d’Ivoire with one operation). There were no MIGA operations in the South Asia, East Asia and Pacific, LAC and MENA regions. 12. Modal Characteristics of the World Bank Group Urban Transport Portfolio:  About 40 percent of the projects are focused on urban roads including upgrading, rehabilitation, or new constructions in major cities as well as smaller towns.  Around 30 percent supported bus operations and services, including bus rapid transit (BRT) systems. The Bank has helped pilot the BRT systems in Bogota, Lagos, Lima, Santiago, Mexico City, and Guatemala. It has also helped introduce new bus systems with BRT characteristics in Dar es Salaam, Accra and Kumasi, Hanoi, Dhaka, Dakar, Kampala, and the cities of China and India.  The Bank provided investments and technical assistance in support of non-motorized transport (NMT) in 20 percent of UT financing projects. The Bank support of non- motorized transport included operational and technical support to promote the use of bicycles and pedestrian walkways. Technical assistance included promotional campaigns at changing travel behavior to secure a modal shift to bicycles as well as safety strategy to minimize bicycle accidents. The civil works supporting NMT included rehabilitation and construction of bike lanes, bike parking facilities, and segregated facilities for pedestrians (underpasses, foot over bridges, sidewalks) aimed at facilitating inter-modal integration and improved access for non-motorized transport.  The Bank also provided operational and technical support for subway systems and light rail operations and services in 10 percent of the projects.  Within the lending portfolio, the Bank has provided assistance in institutional strengthening, capacity building, policy advice and strengthening financing for urban transport in client countries: This includes: (i) creation of a Metropolitan Transport Authority; (ii) setting up new entities to be in charge of urban transport management and regulation; (iii) developing a policy and strategic framework for the urban transport sector; (iv) setting up arrangements to finance urban transport activities; (v) raising the level of cost recovery; and (vi) increasing the private sector role in the provision of urban transport service. 36 Attachment 8: Country Case Study Selected Country Case Study Percentage of the total urban transport commitment Number of projects Income (%) No. Country Region Group IFCAS IFCIS MIGA WB AAA IFCAS IFCIS MIGA WB AAA TOTAL Congo, 1 AFR Low income 5 1 1,030 1,030 Dem. Rep. 2 Uganda* AFR Low income 4 1 619 619 Lower middle 3 Nigeria AFR 4 1 460 460 income Lower middle 4 Indonesia EAP 9 2 20 1,733 1,752 income 1 Lower middle 5 Vietnam EAP 11 6 1,731 1,731 income Upper middle 6 China* EAP 40 3 5,614 5,614 income Upper middle 7 Turkey* ECA 2 1 227 520 500 1,247 income 4 3 Upper middle 8 Brazil* LCR 17 4 1 4,370 4,371 income 1 Upper middle 9 Colombia LCR 5 45 2,087 2,132 income 1 2 Upper middle 10 Peru LCR 9 980 980 income Upper middle 11 Jordan MNA 4 2 197 199 income 1 1 Upper middle 12 Lebanon MNA 5 313 313 income Lower middle 13 Bangladesh SAR 4 1 696 696 income Lower middle 14 India* SAR 1 12 4 2 2,726 2,728 income TOTAL 3 6 3 122 25 Source: Business Intelligent and World Development Indicators. Note:* are countries for field visits. The Transformational Engagement Learning Product also showcases Bogota Urban Services Project (Columbia) and Columbia Integrated Mass Transit Systems, which supported Transmilenio as being transformational. Main Feature of 14 countries Commitment Amount (M$) No. of Income Group FCS Country IFCAS IFCIS MIGA WB Total Low income  1  1        1,649    Lower middle income    5  2  20    7,346  7,368  Upper middle income    7  3  272  520  14,063  14,857  Total  1  13  5  292  520  23,057  23,874  GRAND TOTAL    100  11  486  1,305  34,077  35,878  %       46%  60%  40%  68%  67%  Source: Business Intelligent and World Development Indicators. 37 xix A preliminary review of all catalogued impact evaluations and systematic reviews from 3ie suggests an exceptionally limited rigorous impact literature on urban transport. xx The WBG’s Advisory Service Activities (ASAs) including ESWs, and Non-Lending Technical Assistance (NLTA) such as workshops, conferences, surveys, and policy advice will be analyzed as part of the portfolio review. xxi For the World Bank, this includes both core and non-core projects. 38 Attachment 9: IEG Major Evaluations relevant to the on-going Urban Transport evaluation Improving Institutional Capability and Financial Viability to Sustain Transport: An Evaluation of World Bank Group Support since 2002 (March 2013) 1. This evaluation assessed the effectiveness of World Bank Group support to countries in sustaining the provision of transport infrastructure and services and distills lessons on the factors contributing to sustained transport. Over the past decade the World Bank, IFC, and MIGA have committed about $50 billion for operations or guarantees in the transport sector; yet inadequate operations and maintenance has remained a concern. The study found potential to increase WBG focus on sustaining transport infrastructure and services in project design, including with regard to both financial arrangements and institutional capabilities. It identified a need to improve the long-run financial viability of financing for transport systems. In this regard, it found a need for greater realism in urban transport projects in forecasting costs and revenues, taking account of competition between different modes of transport. It found that support to urban transport should “include a comprehensive financial analysis of the overall urban transport system, including fare integration, tariffs and subsidies and the net impact on the poor.” It also found that reliance on “diverse funding sources” enhanced financial viability. Finally, it found a need to improve engagement to strengthen institutional capability, including a more continuous and sequential engagement in support of complex reforms, a politically aware engagement of key stakeholders, implementation of models to improve incentives and accountability of public sector-dominated systems, and strengthened M & E capabilities at the country system level. A Decade of Action in Transport: An Evaluation of World Bank Assistance to the Transport Sector, 1995–2005 (2007) 2. This evaluation assessed the Bank’s activities in transport during the period 1995– 2005 and concluded that transport must focus more attention on confronting cross-cutting issues such as traffic congestion, environmental damages, safety, efficiency, and affordability. This focus would necessitate more innovative, multi-sectoral approaches to resolve these complex and urgent country and global concerns. The Bank may have to reconsider its priorities to fully address these challenging social, political, and environmental issues and shift resources to ensuring efficient multimodal transport, improved rural linkages, and better urban transport. Increased support to urban transport would provide opportunities to explore reducing long-term energy demand through traffic management, traffic pricing, limits on the use of private automobiles, and greater support for mass transit systems and public transport in general. The Bank may make increased use of funding sources such as the Global Environment Facility, the United Nations Environment Program, and carbon finance initiatives in future years to tackle some of these important developments. World Bank Group Support to Public-Private Partnerships: Lessons from Experience in Client Countries, FY02–12 (2013) 3. This evaluation finds that the World Bank Group has made a significant contribution to capacity building for PPPs, but a lack of local skills and resources for the preparation of a PPP pipeline and bankable PPP projects poses a serious limitation across most World Bank- 39 supported countries. Most of the Bank’s upstream work aims at sector reform, which, however, failed in almost half of the cases because of the complexity and political implications of the reform processes and that advice on how to manage fiscal implications from PPPs is rarely given. To further improve the World Bank Group’s PPP ambitions, the report recommended to translate the World Bank Group’s strategic PPP intentions into an operational framework; better assist governments in making strategic decisions with regard to the level and nature of private sector participation and assessing fiscal implications; identify avenues to increase IFC investments in PPPs located in countries and markets; and define principles for the monitoring of PPPs over the long run to capture all vital performance aspects of PPPs, including – where relevant – user aspects. Phase I: An Evaluation of World Bank Win-Win Energy Policy Reforms (2009) Phase II: Climate Change and WBG: The Challenge of Low-Carbon Development (2010) Phase III: Adapting to Climate Change: Assessing the World Bank Group Experience (2013) 4. IEG Three-Phase Evaluation on Climate Change studied the World Bank Group’s efforts for dealing with climate change. The Phase I report assessed the World Bank’s experience with key win- win policies in the energy sector— policies that combine gains at the country level with globally beneficial greenhouse gas (GHG) reductions. It examined removal of energy subsidies and promotion of end-user energy efficiency recognizing that urban management is among the areas where cross-sectoral collaboration is essential to promoting win-win policies and programs to address climate change issues. The Phase II report (Chapter 4) specifically discussed the challenge of promoting low-emissions urban transportation that was illustrated through a detailed examination of bus rapid transit, which ties together the issues of modal shift, fuel shift, and land use that are central to a city’s transport footprint. The report concluded that BRTSs could be a contributing component in the construction of efficient cities with low carbon footprints, if they are able to retain their share of passenger trips; larger, sustainable long-run gains would require demand-side management of traffic and rational land use planning. The Phase III study finds that long- lived, inflexible infrastructure projects are often subject to climate risk, but the Bank Group lacks procedures for identifying and mitigating these risks, and climate models have proved less useful than hoped for in identifying adaptation options, suggesting the need for more attention to decision making under extreme uncertainty. Improving Municipal Management for Cities to Succeed (2009) 5. This IEG evaluation focused on three dimensions of municipal management— planning, finance, and service provision. The service provision dimension referred to the capacity of a municipality to manage the services required by city residents and business people through the effective prioritization of investments, management of competitive procurement, and the ability to sustain services through O&M. The report concluded that improved municipal management became increasingly challenging as cities grow, costs increase, and service expectations rise. It also found that few municipal development projects focused on assisting the poor and improving the lives of the poor, and evidence of actual results of better access to services obtained was thin. It recommended that municipal development projects need to give much more attention to poverty reduction in their 40 objectives, showing how the poor would benefit from municipal investments and services improved through stronger municipal management. Environmental Sustainability: An Evaluation of World Bank Group Support (2008) 6. This IEG report assessed the Bank Group’s support for environmental sustainability—in both the public and private sectors. It identified several crucial constraints that need to be addressed, including insufficient government commitment to environmental goals and weak institutional capacity to deal with them. The report recognized that to effectively address environmental problems, there is a need for more strategic and coordinated approaches across several sectors in a given locality—such as energy, industry, transport, and land use planning in the case of urban air pollution. Gender and Development: An Evaluation of World Bank Support (2010) 7. The evaluation reviewed how the Bank implemented its gender policy and to what extent it supported the reduction of gender disparity and enhanced women’s empowerment. Outcomes of Bank support were assessed around three key domains: enhanced human capital, increased access to economic assets and opportunities, and enhanced voice of men and women in development. It found that that to be effective, gender integration needs to address strategic issues (institutional and policy reform) that would help facilitate and sustain gender and development outcomes in the field. The lack of a results framework reduces the policy’s relevance, leaving unclear precisely how the Bank aims to translate the goal into results. Recommended actions include redoubling efforts to institutionalize the accountability framework, develop the monitoring and results framework to facilitate consistent adoption of an outcome approach to gender integration in the Bank’s work, and broadening the requirement for gender integration at the project level. Global Program Review of the WBG’s Partnership with the Global Environment Facility (GEF) (2013) 8. The report acknowledges that the mandates and strategies of the World Bank Group and the GEF have been and remain highly compatible and mutually relevant. The Bank Group considers the GEF as a crucial contributor to innovative and risk-sharing approaches, and the GEF perceives the Bank as having a key comparative advantage in leveraging GEF funding to generate global environmental benefits in large projects. The partnership arrangement require however further recalibrating and specifying partnership objectives to mutually agreed levels, establishing procedures for implementation that honor respective institutional and operational strengths and constraints. The areas include resource allocation, project-cycle management and administrative fees, blending and innovation, adapting the World Bank Group-GEF partnership to evolving realities. 9. The team also notes that the relevance of the FY17 IEG Evaluation of WBG’s support for management of air and water pollution (Environmental Management of Air and Water Resources). This evaluation will focus on local pollution issues impairing economic growth and effecting the lives of the poor. Given The evaluation will address both upstream / policy issues related to strengthening regulatory frameworks, institutional capacity and environmental strategies, as well as results at the project level (investments, guarantees and 41 advisory services). The evaluation will also look at the WBG’s role in partnerships related to environmental management, in particular to air and water pollution, and will assess their effectiveness, to the extent results data are available. Given the complementarity of the two evaluations (and the importance of mass transit and NMT as pollution control strategies), the two teams will confer closely and regularly. IEG Relevant Learning Notes Learning Note: Additional Financing for Transport and Information and Communication Technology (ICT) 10. This IEG learning note assessed the performance of the Additional Financing (AF) operations of the WBG of the Transport and Information and Communication Technology (ICT) Global Practice to draw lessons from their implementation experience. The main findings included the fact that even the well performing additional financing projects are facing challenges achieving their institutional objectives, there was no strong correlation between the number or the timing of the AF and the final outcome of the project, and not revising PDOs or project design in the face of implementation challenges can represent a missed opportunity especially at the time of AF when such changes can be more easily processed. Learning Note: Making Roads Safer: Learning from the World Bank's Experience (2014) 10. This study was created by IEG in cooperation with transport operational staff and the Bank’s Global Road Safety Facility (GRSF), with an objective to provide useful knowledge to Bank operational staff involved with road safety, to support Bank and client countries in fine tuning their road safety strategies and practices, and to support the acceleration of the Bank’s operational road safety agenda. The main findings include: (i) the funding gap for road safety remains; (ii) knowledge about road safety in the developing world needs to be enhanced; (iii) strong government commitment to improving road safety is a key to success; (iv) sustainable road safety improvements take time; (v) there is room to improve the design and implementation of road projects; and (vi) The Global Road Safety Facility greatly supported the Bank’s progress in road safety. 42 Attachment 10: Proposed list of Project Performance Assessment Reports • Morocco Urban Transport Sector DPL (P115659); • Senegal Urban Mobility Improvement Project (P055472); • Nigeria Lagos Urban Transport Project (P102029); • India Mumbai Urban Transport Project (P050688); • Mexico Framework for Green Growth Development Policy Loan (P115608); • Argentina GEF Sustainable and Transport and Air Quality Project (P114008); • China Second Tianjin Urban Development and Environment Project (P040599); • China Chongqing Small Cities Infrastructure Improvement Project (P081161); • Indonesia Second Urban Poverty Project (P072852); and, • Indonesia Third Urban Poverty Project (P084583) 43