96834 Approach Paper Accelerating Growth through Skills and Knowledge: An Evaluation of the World Bank Group’s Support for Higher Education May 14, 2015 Background and Context 1. The World Bank Group’s support for higher education is at a crossroads. As developing countries have expanded basic education systems and transition into the knowledge economy, the World Bank Group is facing increased demand for support of higher education. Despite agreement on the contribution of higher education to development, there is no consensus about the degree and type of support the Bank Group should provide. There are also concerns that the quality of higher education offered in many countries does not sufficiently address the challenges of modern global economies. While the Bank Group has been a pioneer in providing policy support for higher education, its higher education portfolio has not been evaluated for at least a decade. 2. This evaluation is a component of the Inclusive Growth Strategic Engagement Area of the Independent Evaluation Group (IEG). Higher education lies at the nexus of growth, jobs, and competiveness. It has the potential to create new and better quality employment as well as to serve as a catalyst for economic transformation. Higher education also contributes to the post-2015 Development Agenda and the Bank Group’s twin goals. The development of a country’s higher education system can have a positive impact in many other areas, including social equity, political stability, and regional development. For the World Bank, it is important to understand what approaches and interventions have been effective in promoting higher education as a public good and in improving access for excluded groups. Likewise, the International Finance Corporation (IFC) can benefit from an examination of how its support for higher education might better contribute to its development objectives. Both organizations can benefit from understanding their complementary functions in this context. Higher Education and Development: A Brief Survey 3. The higher education system plays several important roles by educating professionals, supporting research, and contributing to social capital. Higher education is multifaceted, often combining research, teaching, and regional development into a single institute. It has the potential to generate positive externalities that benefit the economy and other aspects of society (Bloom, Hartley, and Rosovsky 2006; McMahon 2009). It builds on secondary education, with learning activities in specialized fields of education that often complement research. Higher education addresses learning at a high level of complexity and specialization (UNESCO Institute of Statistics 2015). 1 Box 1. The World Bank Group’s Definition of Higher Education In classifying interventions, the World Bank Group defines its support for higher or tertiary education under the Sector Definition Guidelines of Operations Policy and Country Services as: “…efforts to improve the teaching and research capacity of degree-granting educational institutions that support students beyond the secondary school level, specifically colleges, universities, graduate schools, and professional schools.” This definition focuses on formal post-secondary educational institutions and generally excludes technical and vocational education programs that do not grant degrees and institutions focused exclusively on research. The Bank Group definition appears to reflect a dual objective of improving the teaching and research capacity of higher education institutions. Broadly speaking, this corresponds to levels 5, 6, 7, and 8 of the International Standard Classification of Education developed by the United Nations Educational, Scientific, and Cultural Organization (UNESCO): short-cycle tertiary education, bachelor’s degree or equivalent, master’s degree or equivalent, and doctoral degree or equivalent. Higher or tertiary education differs from training; the latter focuses on imparting a specific set of skills. Tertiary education is the official term as defined in the UNESCO classification. However, in practice, many organizations, including the World Bank and UNESCO, use the term higher education. Despite some technical differences, the two terms are often used interchangeably. In this evaluation, both terms are used to describe Bank Group practices. Source: UNESCO Institute of Statistics 2015; Campbell and Carayannis 2013. 4. A key function of higher education is to educate professionals, knowledge workers, and skilled civil servants. While higher education institutions perform many functions, they primarily offer specialized education for adults in a wide range of fields, ranging from the humanities to medicine and science (UNESCO 1998). The number of higher education students has grown substantially from 68 million (46 percent from developing countries) in 1990 to 196 million (68 percent from developing countries) in 2012. Globally, it appears that the social sciences, business, and law are the most popular programs. Women tend to be more highly represented in education and health fields and men in engineering and scientific fields. 5. Higher education also has positive externalities that contribute to economic growth by promoting technological potential. The Commission for Growth and Development argues that balanced economic growth requires local innovation as well as the capacity to adapt technology and know-how from the rest of the world (Commission on Growth and Development 2008). Adapting foreign technology relies on local research capacity (Todtling and Trippl 2005; Dolreux and Parto 2005). Likewise, technologies cannot be adapted to local conditions unless the workforce is sufficiently educated (Lucas 1988; Romer 1990). To bring in foreign knowledge workers is an expensive undertaking and not a sustainable strategy for most developing countries. Universities play a major role as local hubs for research by supplying knowledge, technology, and innovations to many sectors (Altbach, Reisberg, and Rumbley 2009). Overall, investing in higher education has led to greater economic growth, partly through its contribution to sector and national productivity as well as through its impact on labor productivity (Bloom 2 and others 2013; Bloom, Hartley, and Rosovsky 2006). Studies have shown these results in countries as diverse as Cameroon (Fonkeng and Ntembe 2009), China (Wang and Yao 2003; Whalley and Zhao 2010), and Pakistan (Stengos and Aurangzeb 2008). 6. Higher education supports private sector development in a number of ways. As industries move up the “value chain,” they require a more educated and productive workforce. By providing better educated workers, higher education can contribute to increased national productivity and economic growth (Heckman and Ji 2012; Jung 2011). In many cases, firms work with universities to identify areas where greater skills are needed. In addition to the direct, two-way relationship between firms and universities in producing and hiring employees, universities often collaborate with the private sector to promote innovation. This can take the form of contracted research, where a firm engages a university for a particular research project. Moreover, universities provide “basic research” to support industries with projects that might benefit more than one firm. 7. Employers are willing to pay more in exchange for the higher productivity of a better educated worker. In the past, many estimates showed a pattern of diminishing returns across education levels, implying lower rates of return at the tertiary level (Psacharopoulos and Patrinos 2004; Psacharopoulos 1994). However, recent literature has found evidence that the marginal rate of return for earnings has been increasing more quickly for more educated groups (Montenegro and Patrinos 2014). This trend have been seen in a wide range of countries and regions, including India, Pakistan, the Philippines (Schady 2003), the Pacific Islands, and Vietnam (Fasih and others 2012). For countries in Sub-Saharan Africa (Côte d’Ivoire, Ghana, Kenya, and South Africa), returns for schooling are highest at the higher education level, with a decreased rate at lower school levels (Schultz 2004). 8. There are other important nonmonetary benefits associated with higher education. Increased earnings do not reflect the full value of a higher education. In India, several studies have shown that university graduates have greater knowledge about health risks (Agrawal and others 2011; Sansone and others 2012). In Sudan, evidence shows that while primary education is insufficient to induce a large change in fertility, higher education has a significant impact on family composition (Ahmed 2010). In Eritrea, university access led to greater career and marriage choice (Müller 2004). In Pakistan (Shafiq 2010), university study had a positive effect on women’s independence and awareness of rights (Malik and Courtney 2011). More broadly, a university education is commonly understood as a vital source of social mobility. 9. In addition to the benefits to individuals, higher education can make important contributions to the public and social sectors. By supplying qualified graduates and housing policy institutes and NGOs, higher education institutions can address the public sector demand for civil servants and for other professionals who serve the public interest. Higher education also has a positive effect on civic participation and governance in low-income countries (Dee 2004; Shafiq 2010) and on political stability in Africa (Gyimah-Brempong, Paddison, and Mitiku 2006). Through Third Mission activities that complement their research and teaching functions, universities can make a variety of social contributions through community service, outreach, and technology transfer. 3 10. Higher education can play an important role in promoting regional and urban development. Universities can serve as “development universities” that directly engage with the community to improve productivity and efficiency (Oketch, McCowan, and Schendel 2014). Working in partnership with local industries—and often through regional campuses—higher education institutions can help to harness the competitive advantage of a particular region and prepare graduates for jobs in applicable fields (Fetters, Green, and Rice 2010). As vital components of a regional ecosystem and as centers for innovation, these institutions can transform cities in aspects that include quality of life, infrastructure, and culture. 11. Higher education institutions in both developed and developing nations have steadily increased their efforts toward internationalization, online education, and other new modes of delivery. These trends are likely to continue in the face of ongoing growth in overall higher education enrollments, globalization, and economic development. In 2012, at least 4 million students went abroad for study, up from 2 million in 2000 and representing 1.8 percent of all higher education enrollments (UNESCO Institute of Statistics 2014). While institutions in high income-countries remain a common destination for students and faculty, the earlier, one- directional South–North flows of students, scholars, and revenue have been balanced by greater North–South exchange and South–South engagement (Chao 2014). There is strong interest in expanding online and distance education as well as other non-traditional modes of instruction, particularly those that are scalable and can be delivered at lower cost. Massive open online courses (MOOCs), which are offered by both established universities and for-profit companies, have enrolled millions of students worldwide. 12. For many countries, universities are viewed as a source of prestige. Governments often support higher education to promote national pride, pushing for the development of a “world class university” beyond a country’s development needs. This trend is often reflected in efforts to improve the international rankings of a country’s flagship universities (Rigoglioso 2014; Ramirez and Tiplic 2014). These efforts can alter the trajectory of higher education systems in several ways. First, substantial resources may be diverted to elite research and teaching institutions. Second, students—often from affluent backgrounds—may, in a similar manner, seek degrees from top universities to acquire prestige rather than marketable skills (Usher 2012). Third, pursuing prestige and reputation has led to increased competition among higher education institutions, both locally and globally (The Economist 2014). With standards for quality (often expressed through rankings) becoming internationalized, countries may strive to conform their systems to homogenous global standards rather than to local and national needs. 13. Higher education institutions vary widely in financing, function, and form. Institutions that closely integrate teaching with research are common in British and U.S. universities, which have been widely held as models for structuring higher education (Salmi 2009; Marginson 2006). It is often impossible to separate the different functions of higher education institutions as the varied activities they engage in tend to reinforce each other. Ownership and funding in higher education varies, with public and private institutions supported by combinations of tuition revenues, public subsidies, and institutional endowments. In many countries, leading universities have expanded their ability to generate revenue from government, the private sector, and philanthropic organizations to support the production of academic research and other knowledge-based activities. Many higher education systems have promoted 4 partnerships with the private sector locally and internationally as well as with foreign universities (Rhoads 2011). Private colleges and universities offer the same fields of study as public institutions but often focus exclusively on teaching (Altbach, Reisberg, and Rumbley 2009). 14. While the organization of higher education systems vary greatly from country to country, many developing countries face a number of common challenges. There is rapidly growing demand for post-secondary education as a result of the substantial expansion in primary and secondary education. However there is also a great deal of inequality, as students from poorer households often lack the same access to quality education as students from wealthier backgrounds. Access and results often differ for male and female students. Likewise, many students with disabilities and from ethnic minorities face serious issues with access. As higher education systems expand, they face increased challenges to sustain the quality of teaching and academic research. Low quality in higher education threatens the employability of graduates. Attachment 2 presents a more detailed discussion of these challenges. The World Bank Group and Higher Education 15. The World Bank Group’s support for higher education began in the 1960s as it sought to expand its development impact. Initially, the World Bank focused on adult education, including support for universities and vocational education. As part of efforts to improve economic productivity, there was a gradual shift toward higher education. Later, concerns about poverty and excluded populations led to a greater focus on basic education (Kapur, Lewis, and Webb 1997). 16. In 1980s and 1990s, the World Bank shifted its focus away from higher education. This shift reflected concerns that the higher education system was regressive and that subsidies primarily benefited upper income groups. The World Bank’s 1999 education strategy identified higher education as a secondary priority (World Bank 1999), but it also recognized that the demand for higher education was often quite high (World Bank 1995). The World Bank’s approach may have led to the “neglect” of higher education, particularly in low-income countries (Bloom and others 2013; Kent 2005). This trend may have also affected other areas of education, such as technical and vocational education and secondary education. 17. By the 2000s, the World Bank Group reevaluated its approach to higher education. As country demand for higher education increased, the Bank Group changed its approach. A policy paper on higher education recommended that each country establish its own priorities (Task Force on Higher Education and Society 2000). This was incorporated in the education strategy update, calling for “greater awareness of the role of tertiary education and lifelong learning in promoting knowledge-driven economic growth” (World Bank 2005). Higher education was explicitly included in the 2009 Education Strategy, and most World Bank Regions have developed strategic documents for higher education. The IFC developed its Strategic Directions for Education (focusing largely on higher education) with the business objectives of improving financial viability and mobilizing private resources. Its development objectives are promoting education innovation, strengthening the education system, promoting opportunities, and bridging the digital divide (IFC 2001). 5 18. Other development partners offer a variety of support to higher education. A few development partners like Japan and the United States provide direct support for higher education as have several regional development banks, such as the Asian Development Bank. Most development partners do not directly finance higher education. However, many governments, foreign universities, and non-state actors promote university-to-university collaboration and offer scholarships. This support has been influential in developing new research agendas and academic talent. 19. The World Bank Group has a large and diverse investment and knowledge portfolio in all Regions and in both middle- and low-income countries. The combined Bank Group investment portfolio from 2003 to 2014 includes 160 higher education-related projects totaling $10.9 billion. Table 1 presents a summary of the public sector portfolio. These operations are well-distributed in all Regions and include support to low-income countries such as Bangladesh and Tanzania, middle-income countries such as Mexico and Vietnam, and fragile and conflict- affected states such as Afghanistan and the West Bank and Gaza. The IFC’s private sector financing (see table 2) was largely concentrated in middle-income countries in Europe and Central Asia and Latin America and the Caribbean. The IFC has supported global education providers for their operations in developing countries, creating centers of excellence and providing employment opportunities for professionals. Table 1. World Bank Higher Education Investment Portfolio, 2003–2014 Amount Number of Projects (US$, millions) Higher education projects IBRD IDA Total IBRD IDA Total 46 85 131 5,619.0 4,707.8 10,326.8 Sector board ED 28 59 87 2,502.5 3,843.4 6,345.9 Non-ED 17 27 44 3,116.5 864.4 3,980.9 Project status Active 19 40 59 1,149.3 2,594.9 3,744.2 Closed 27 45 72 4,469.7 2,112.9 6,582.6 Lending instrument type Investment 36 76 112 2,680.9 4,279.5 6,960.4 DPO 10 9 19 2,938.2 428.3 3,366.5 Region AFR 2 38 40 15.0 1,848.6 1,863.6 EAP 9 12 21 407.4 826.6 1,234.0 ECA 14 11 25 2,262.8 257.3 2,520.1 LAC 15 2 17 2,176.4 14.2 2,190.6 MNA 6 2 8 757.4 19.0 776.4 SAR 0 20 20 0.0 1,742.1 1,742.1 Source: World Bank Group and IEG databases. Note: AFR = Africa; DPO=Development Policy Operation; EAP = East Asia and Pacific; ECA = Europe and Central Asia; ED = Education; IBRD = International Bank for Reconstruction and development; IDA = International Development Association; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. 6 Table 2. IFC Higher Education Investment Portfolio, 2003–2014 Amount Number of Projects (US$, millions) Higher education investment projects Equity Loan Mixed Total Equity Loan 8 20 1 29 178.7 359.0 Project status Active 8 17 0 25 177.1 312.5 Closed 0 3 1 4 0.0 46.5 Region AFR 0 4 0 4 0.0 18.4 EAP 0 1 1 2 1.6 23.0 ECA 0 4 0 4 0.0 78.0 LAC 4 11 0 15 54.0 239.6 MNA 2 0 0 2 18.1 0.0 SAR 0 0 0 0 0.0 0.0 World 2 0 0 2 105.0 0.0 Source: World Bank Group and IEG databases. Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. 20. Public sector projects support both higher education institutions and students. Based on an initial review of project objectives, World Bank public sector investments appear to focus primarily on improving the quality of teaching, research, and higher education governance. An important subset of operations focus on improving access and equity by strengthening student credit, providing scholarships, and using other relevant instruments. The World Bank typically promotes university autonomy with interventions to strengthen governance, capacity, and quality assurance. 21. The World Bank Group has a large and diverse portfolio of knowledge products at the global, regional, and country levels. The Bank Group supports both regional projects and country-based projects. Some of the regional projects focus on a group of countries such as the Caribbean or the European Union accession states. Table 3 presents the distribution of Bank Group knowledge products. Many of these knowledge products are diagnoses of higher education that often shape World Bank strategy. The IFC’s support has largely focused on the Africa and the Middle East and North Africa regions. Its advisory services in higher education are quite diverse, ranging from support, to developing university-based commercial activities, to results monitoring, to studies on developing student loan markets. In addition to direct financing, the IFC has also been active in promoting public-private partnerships in the education sector. In addition, the Bank Group has provided fee-based services in the area of higher education, either as self-standing operations or in coordination with Bank-financed knowledge projects. For example, in Kazakhstan, the World Bank has a partnership program to support the development of the higher education system. Likewise, in Chile, the World Bank worked with the government to review higher education finance. 7 Table 3. Number of World Bank Group Higher Education Knowledge Projects and Advisory Services, 2003–2014 Regional and Subregional Level Country Level Public Sector Private Sector Public Sector Private Sector Area (World Bank) (IFC) (World Bank) (IFC) Total AFR 8 3 21 7 39 EAP 0 0 10 0 10 ECA 9 0 12 0 21 LCR 3 0 7 2 12 MNA 2 0 7 5 14 SAR 2 0 7 2 11 Global 5 0 0 0 5 Total 24 3 64 16 107 Source: World Bank Group and IEG databases. Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; IFC = International Finance Corporation; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. Purpose, Objectives, and Audience PURPOSE 22. This evaluation aims to increase knowledge about the World Bank Group’s approach to higher education. While there is agreement over the contribution higher education can make to development, there is no consensus about the degree to which the Bank Group should support higher education. Crucial questions remain about whether the Bank Group should be involved in higher education, how it might leverage its competitive advantage where it is engaged, and how its support for higher education might best reflect the World Bank Group Strategy. The information gathered by the evaluation may serve as a resource for addressing these types of questions. OBJECTIVES 23. The evaluation aims to understand the design of the Bank Group’s support for higher education and its contribution to the World Bank Group’s objectives. The evaluation seeks to provide clarity on the complementary and sometimes competing objectives for higher education that inform the Bank Group’s support and to identify areas where a coherent vision is lacking. It will attempt to discern the underlying rationale behind the Bank Group’s support for higher education, what strategies have worked to address the challenges for higher education in developing countries, and how it might improve its capacity to learn from its experience with higher education. This includes the range of strategies pursued in interventions from improving teaching and research, to addressing governance issues, to promoting autonomy, to bolstering capacity in quality assurance—interventions that are expected to support inclusive economic growth, jobs, and competitiveness. It will examine how the Bank Group combines these types of 8 interventions at the country level. The evaluation will identify lessons based on the Bank Group’s recent experiences that can inform future support for higher education. Box 2. The Evaluation and the Inclusive Growth Strategic Engagement Area The Independent Evaluation Group (IEG) organizes its work plan around three strategic engagement areas (SEAs): (i) Sustained Services for the Poor; (ii) Equity and Sustainability of Natural Capital; and (iii) Inclusive Growth. The evaluation fits in the third SEA as higher education plays an important role in generating the resources for growth, providing pathways out of poverty, and improving living standards. In addition, higher education contributes to jobs that are important for development and are transformational in people’s lives and productive work. This evaluation will be carried out in parallel with the IEG’s evaluation on jobs and competiveness, which also falls under the Inclusive Growth SEA. The two evaluations will work together to understand the role of Bank Group support for higher education in promoting competitiveness at both the country and operations levels. The higher education evaluation is likely to focus on Bank Group support for information and communication technology and the role of higher education in supporting these investments. The two evaluation teams will study the possibility of a joint country case study and will coordinate in the design of evaluation instruments. AUDIENCE 24. The evaluation will be of interest to internal stakeholders as they address questions about the role of higher education in development. The evaluation should be of interest to the Board of Executive Directors, offering a report that can help it better understand the Bank Group’s rationale for supporting higher education and the effects of its support on a broad range of pressing public, private, and social sector development themes. The Education Global Practice can benefit from an evaluation of its activities in light of the new Bank Group strategy and other changes in the organization. The evaluation may provide insight and guidance to the IFC as it develops strategies for supporting private sector higher education institutions. 25. The evaluation also has an external audience. The evaluation will be of interest to stakeholders who want to understand the Bank Group’s approach and to engage it on issues involving higher education. This audience includes governments and policy makers, universities and the academic community in developed and developing countries, development partners and international organizations, and nongovernmental organizations (NGOs). Although the evaluation will focus on the Bank Group and its support, it is likely to draw lessons that are useful to development partners. Many donors face similar issues in defining their support for higher education and can benefit from an evaluative report on the Bank Group’s experience. These include governments, the Bank Group’s trust fund contributors, and peer financial institutions that are engaged in higher education or that seek to reassess their public and private sector involvement. 9 Evaluation Questions 26. The evaluation’s overarching question is: “How has the World Bank Group’s support to higher education contributed to its twin goals of poverty reduction and shared prosperity?” To address this subject, the evaluation is divided into three questions and 13 subquestions. Each of the evaluation questions was designed with three criteria: (i) the question should serve the evaluation’s purposes and objectives; (ii) a viable methodology should be available to answer it; and (iii) the question should not be merely descriptive. While the evaluation does have an accountability function, it will not rate the efficacy or relevance of the Bank Group’s overall higher education program. 27. The evaluation will be both prospective and retrospective in its analysis. While the evaluation will identify the channels through which higher education contributes to the Bank Group’s twin goals, it will not hold the Bank Group accountable for achieving goals that were articulated after an operation was approved. This examination will help deepen the understanding of how higher education can best contribute in the future. 28. The first evaluation question asks: “Is the World Bank Group’s support for higher education consistent and well-articulated?” This question focuses on the logic behind the Bank Group’s strategies in higher education. The evaluation question will aim to understand whether its approach to higher education is internally coherent and in line with established policy approaches. Likewise, it will identify how the Bank Group generates and applies evidence and lessons, and how the Bank Group works in partnership with other actors. This question is divided into four subquestions:  How has the Bank Group incorporated higher education in its strategic documents?  How does it coordinate its support for higher education internally within the Bank Group?  How does it coordinate its support for higher education with external development partners and nongovernment actors?  How does it conceptualize higher education and incorporate local context into the design of its operations? 29. The second evaluation question asks: “How has World Bank Group support contributed to higher education systems?” This question focuses on the linkages between the design of the Bank Group’s operations and outcomes affecting higher education systems. The evaluation will analyze its support that contributes to the reform of higher education institutions and systems. The evaluation addresses this line of inquiry through four subquestions:  How has the Bank Group contributed to changes in the financial sustainability and management of higher education systems?  How has its support strengthened the connection between higher education and both the public and private sectors?  How has it supported regulation and quality assurance in public and private universities? 10  How has its support contributed to internal efficiency in higher education? 30. The third evaluation question asks: “How has the World Bank Group’s support for higher education contributed to social and economic outcomes? This question focuses on the linkages between the design of the Bank Group’s operations and outcomes affecting society and the economy. The question will identify how the Bank Group’s support has addressed vulnerable groups and improved opportunities for women. The evaluation question has five subquestions:  How has Bank Group support improved access and equity for lower income households?  How has its support addressed gender and other issues affecting traditionally excluded groups in higher education?  How has its support contributed to external efficiency through developing skills and improving the employability of graduates?  How has its support contributed to external efficiency through private sector development and increased industry competiveness?  How has its support contributed to the quality of research and its relevance to local development challenges? Evaluation Design UNIVERSE, SCOPE, AND LIMITATIONS 31. The evaluation will be limited in scope and focuses on the World Bank Group’s role in higher education. As outlined in box 1, the evaluation will focus on higher education and not on other types of post-basic education. While other post-secondary education interventions are important, they are fundamentally different from higher education in the objectives and the organization of the institutions involved. This limitation in scope does not imply that other areas are less important or do not complement higher education. When compatible with its scope, the evaluation questions, and budgetary considerations, the evaluation will explore connections between higher education and other areas of the education sector as well as other sectors. The IEG Youth Employment Evaluation provides insight on alternative forms of post-secondary education (IEG 2013) as does IEG’s ongoing Jobs and Competitiveness Evaluation. There are several interventions that the evaluation will not analyze, including:  Other types of post-basic and post-secondary education, including nondegree technical and vocational training programs at the secondary and post-secondary level; and  Non-degree adult education such as “second chance” learning or lifelong learning initiatives. 32. In addition, the evaluation will not focus on higher level objectives or impacts that are beyond the immediate influence of the Bank group’s operations. This reflects the difficulty in establishing causality as well as the paucity of data. Specifically, the evaluation will not be able to measure directly: 11  Impacts on broad education outcomes, such as the average level of education; and  Direct impacts on broad economic outcomes, such as employment rates, productivity, poverty reduction, and economic growth. 33. The evaluation will primarily use the self-classification of operations to define the evaluation universe. For public sector projects, the portfolio includes all projects that are classified to the tertiary education (ET) sector or to the education for the knowledge economy (66) theme. The portfolio review inspected all projects classified as “ET” of less than 20 percent. The evaluation classified operations that focus primarily on strengthening the higher education system as core higher education projects to distinguish them from operations where higher education is part of a larger project. This classification includes standalone higher education projects as well as projects that have a substantial focus on the higher education sector. The private sector portfolio includes all projects classified as colleges, universities, and professional schools. This was followed by an inspection to ensure appropriateness for inclusion in the portfolio. A similar approach identified knowledge products and advisory services. The evaluation universe broadly corresponds to categories 5, 6, 7, and 8 of the International Standard Classification of Education (UNESCO Institute of Statistics 2015). It will include teacher training carried out through a degree program as well diploma programs offered at community colleges. The team will confirm the portfolio with the Education Global Practice. 34. The primary unit of inquiry is the higher education country programs. At the country level, the Bank Group offers a variety of services including public and private sector investments, knowledge products, and policy dialogue. Further analysis is expected to uncover why and how these drivers are able to yield results in each country’s circumstances and to what extent these experiences can be replicated in other country situations. APPROACH AND METHODOLOGY 35. The evaluation will utilize a variety of methods to develop an understanding of the World Bank Group’s support for higher education. The evaluation will triangulate both qualitative and quantitative data to understand how the Bank Group’s support for higher education works. Figure 1 presents a conceptual model for Bank Group support and the evaluation’s approach. 12 Figure 1. The Evaluation Framework for Higher Education 36. The evaluation will provide qualitative answers to the evaluation questions in order to develop a better understanding of the Bank Group’s contribution to higher education. The evaluation will not attempt to rate or grade the Bank Group’s performance against any particular question. The evaluation will report on existing self-evaluation and IEG ratings of Bank Group operations. The methodological tools are outlined in greater detail in attachment 2. They include:  Consultation with country higher education stakeholders. The evaluation will hold consultation with higher education leaders at the country level to identify key issues in the sector. This might include government officials, university administrators and staff, students, private sector actors, and NGOs. In addition, the evaluation will establish a Sounding Board, composed of higher education experts and practitioners, to provide feedback on the evaluation and its relevance. 13  Portfolio review. The portfolio review will examine project documents and evaluations— Implementation Completion and Results Reports (ICRs), ICR Reviews, and Project Performance Assessment Reports (PPARs). This will include both a description of the portfolio and analysis of individual operations as well as a review of the lessons reported by the Bank Group in both project and evaluation documents.  Review of the Bank Group’s strategies. The evaluation will analyze corporate, sector, and country strategies as well as associated documents, such as Systematic Country Diagnoses and Country Economic Memoranda.  Review of the Bank Group’s knowledge work. The evaluation will review a range of the Bank Group’s knowledge products, including non-lending technical assistance, economic and sector work, sector analyses, advisory services, economic analysis, and impact evaluations as well as policy dialogue that supports higher education.  Review of academic and policy literature. The evaluation will review the existing academic and policy literature to better understand current thinking about the sector.  Interview with key informants. The evaluation will interview Bank staff, development partners, and higher education specialists to discuss the Bank Group’s strategy and performance. The evaluation will maintain an open line of communication with the Bank Group’s tertiary education community of practice, the Core Higher Education Group (COREHEG).  Country case studies. The evaluation will draw on evidence gathered from field-based country case studies. This step will include collection of data as well as interviews with government officials and local stakeholders. 37. The evaluation will support some complementary knowledge work on key issues and on “what works.” The primary function of the evaluation is to understand the Bank Group’s role in supporting higher education. To complement this, the evaluation will support the preparation of several background papers on key issues in higher education, such as the Third Mission of universities of supporting the wider community or the development of new technologies. This knowledge work serves two purposes. First, the papers will identify common themes, practices, and policies in higher education that may provide a reference point for understanding the Bank Group’s strategies and operations. Second, the background papers will serve to organize and share knowledge for higher education policy makers. FIRST STEPS: SETTING THE STAGE 38. The evaluation’s initial work will focus on describing the World Bank Group’s higher education investment and knowledge portfolios and on understanding pressing global issues. Although the evaluation’s primary focus is on analysis, it will include a detailed description of what the Bank Group supports and the impact of this support. This will be built around an ongoing portfolio review, which will support the analysis of all of the evaluation questions. In addition, the evaluation will analyze country and global data to characterize the higher education sector in quantitative terms. The portfolio review will look at both the design and results of higher education operations and will classify the lessons reported in project documents and evaluations. 14 39. The evaluation will hold consultations and review literature to understand higher education priorities. This process will support efforts to ensure that evaluation benefits from a diverse set of views. Opinions vary on the major issues facing higher education and future directions for the sector. Through consultations with stakeholders and a review of the academic and policy literature, the evaluation will prepare an analysis of the state of thinking on higher education—in country contexts, with reference to Bank Group involvement, and more broadly. This will provide a benchmark or normative framework to which the evaluation can compare the objectives and design of the Bank Group’s interventions. FIRST EVALUATION QUESTION: IS THE WORLD BANK GROUP’S SUPPORT FOR HIGHER EDUCATION CONSISTENT AND WELL-ARTICULATED? 40. The evaluation will analyze how the World Bank Group views higher education and how it designs its operations. The evaluation will use the review of Bank Group strategies to understand the logic model for higher education at the corporate, regional, and country levels. Then, applying the normative framework as outlined, the evaluation will analyze the Bank group’s logic model compared to views of stakeholders. This will lead to a qualitative discussion of the relevance of the Bank Group’s higher education strategy. A similar approach will be used to evaluate the design of the Bank Group’s higher education operations. The evaluation will use the portfolio review to classify and analyze the design of investment operations and the review of knowledge work for technical support. In both cases, the country case studies will provide detailed data to complement the analyses of strategies and portfolios. Interviews with key informants will also provide an understanding of all four of the subquestions. The interviews with a diverse set of key informants, along with the country case studies, are important for understanding how the Bank Group coordinates internally and with other stakeholders. Together, this evidence will be useful to identify the Bank Group’s logic model for higher education. SECOND EVALUATION QUESTION: HOW HAS WORLD BANK GROUP SUPPORT CONTRIBUTED TO HIGHER EDUCATION SYSTEMS? 41. The evaluation will examine available data to understand the impact of World Bank Group support on higher education systems. The portfolio analysis and the review of knowledge work will provide data on the performance of the Bank Group’s operations in transforming higher education systems. The country case studies will provide additional data in the selected countries and will include specific questions tailored to evaluate the Bank Group’s contribution to higher education systems. Interviews with key informants will provide corroborating information to understand results and the Bank Group’s value added. THIRD EVALUATION QUESTION: HOW HAS THE WORLD BANK GROUP’S SUPPORT FOR HIGHER EDUCATION CONTRIBUTED TO SOCIAL AND ECONOMIC OUTCOMES? 42. The evaluation will examine available data to understand how the World Bank Group’s support for higher education has contributed to social and economic outcomes. The portfolio analysis will provide data on the performance of operations on higher level outcomes. The inquiry will require the application of the Bank Group’s logic model in the interpretation of social and economic outcomes. The country case studies will provide additional 15 data and will include specific questions tailored to the country and the higher education system contexts when evaluating the Bank group’s contributions. Interviews with key informants will provide corroborating information to understand results and value added. 43. Secondary data will play a major role in providing evaluation results. The evaluation will not produce any primary quantitative data and will instead rely on secondary data, including those from the Bank Group’s results frameworks. Particularly in the country cases, the evaluation may also benefit from data from other sources, such as governments or universities. It will provide data disaggregated by gender and other socioeconomic indicators when possible. Selection of Country Cases 44. Evidence gathered from the field plays a central role and will be vital to answering all of the evaluation questions. The country case studies will provide in-depth analysis by collecting information needed from a range of countries where the Bank Group is active in higher education. A purposeful country selection will ensure that the case studies cover the diversity of support that the Bank Group provides as well as the diversity of country contexts where it supports higher education. The unit of observation is the Bank Group’s country program in higher education—that is, the portfolio of public and private investments in the country, the Bank Group’s knowledge work, and its advisory services. The countries selected are not intended to be a random sample of beneficiary countries or of Bank Group interventions. 45. The countries are divided using two primary criteria. The primary criteria are designed to ensure a mix of different types of country interventions. The evaluation will organize the countries where the Bank Group is active. The criteria include:  The public and private sector mix of World Bank Group support for higher education. Support is provided through both the public sector window—primarily from the International Bank for Reconstruction and Development (IBRD) and the International Development Association (IDA)—and the private sector window primarily from IFC. The first category consists of countries where the Bank Group has only provided support through its public sector window (IBRD and IDA). The second category is for countries that receive IFC support, either on its own or in conjunction with public sector support. For a country to be included, it must have either a core higher education World Bank project or an IFC project.  The continuity of the World Bank Group’s support for higher education. In some countries, the Bank Group has long provided support for tertiary education, and the investment represents part of a long-term engagement with the sector. In other cases, this support is more recent. For purposes of the evaluation, the cut-off date is 2005. When the World Bank has an engagement in higher education from 1995 to 2005, the Bank Group country portfolio will be classified as having a continuous engagement. Otherwise, it will be classified as having a recent engagement in higher education. 46. The final categorization of countries will be finalized as part of the portfolio review and will be confirmed with the Bank Group’s Education Global Practice. As part of the portfolio review, the evaluation team will confirm the correct classification of each country, 16 including a determination of whether the World Bank’s public sector support is sufficiently important to be counted as public sector support. Table 4 outlines the initial division of countries. Table 4. Initial Classification of Countries by World Bank Group Support Countries with Core Higher Education Operations INITIAL CLASSIFICATION RECENT BANK GROUP CONTINUOUS BANK GROUP ENGAGEMENT ENGAGEMENT WORLD BANK SUPPORT Afghanistan Burkina Faso ONLY Armenia Ethiopia Bangladesh Indonesia Cambodia Mauritania Costa Rica Mozambique Guyana Nepal Malawi Senegal Maldives Sri Lanka Montenegro Uganda Tanzania Vietnam West Bank & Gaza Yemen, Rep. IFC SUPPORT ONLY OR BOTH Antigua and Barbuda Brazil Botswana Chile Kenya China Namibia Colombia Nigeria Dominican Republic Peru Egypt South Africa Ghana Turkey India Jordan Mexico Morocco Pakistan Trinidad and Tobago Tunisia Note: The following countries have World Bank Group higher operations but were excluded because they did not have a core investment project: Albania, Argentina, Bhutan, Bosnia and Herzegovina, Bulgaria, the Democratic Republic of Congo, Croatia, Djibouti, The Gambia, Georgia, Jamaica, Kazakhstan, Kosovo, Lao People’s Democratic Republic, Lebanon, Lesotho, Liberia, the former Yugoslav Republic of Macedonia, Madagascar, Mali, Philippines, Poland, Tuvalu, Ukraine, and Uruguay. IFC = International Finance Corporation. Source: IEG. 47. To ensure diversity among the choice of countries, the evaluation will use two secondary criteria based on country characteristics. These secondary characteristics will help ensure the evaluation looks at the different types of countries that the Bank Group supports: 17  Enrollment in higher education. The evaluation will group countries based on the size of the higher education sector, measured by the gross enrollment rate for tertiary education. To the extent possible, this grouping will focus on enrollment at the beginning of the evaluation period using available data from the Bank Group’s education statistics database (EdStats). When data for this period are not available, the evaluation will use data from independent sources of data and other years.  Share of private higher education. The evaluation will also group countries according to the share that private higher education contributes to total higher education enrollment. Given the scarcity of data, the indicator uses all available data from 2000 to 2014. The primary source of data is EdStats, with other sources used to supplement missing data. Quality Assurance Process 48. The draft report will be reviewed by both internal and external experts in higher education to ensure the accuracy in the findings and messages. The report will be prepared under the direction of Mark Sundberg, Manager IEGPS and Nicholas York, Director IEGCC. It will follow the IEG quality assurance process. The peer reviewers for the Approach Paper are Philip Altbach (Research Professor and Director of the Center for International Higher Education; Lynch School of Education, Boston College); Richard Hopper (President, Kennebec Valley Community College and Former Senior Education Specialist, World Bank); Ravi Kanbur (T.H. Lee Professor of World Affairs, International Professor of Applied Economics, and Professor of Economics, Cornell University); and Fernando Reimers (Ford Foundation Professor of International Education, Harvard University). 49. The evaluation will also have a Sounding Board that will provide feedback during critical stages of the process. In addition to the Approach Paper peer reviewers, the evaluation will convene panels of experts from a variety of backgrounds and regions. These will be organized as a Sounding Board that will meet in person and virtually to provide feedback on the relevance and quality of the evaluation. Expected Outputs and Dissemination 50. The primary output will be an evaluation report that will be submitted to the Committee on Development Effectiveness. The evaluation team will engage with COREHEG and focus groups to explore preliminary findings and messages from the evaluation. The evaluation report will be 30 to 40 pages, in addition to annexes. Some or all of the background papers that are prepared in connection to the evaluation may also be disseminated separately. 51. Dissemination will be integrated into the evaluation. The team will develop a dissemination plan while the evaluation is being conducted. This will ensure that it can be widely disseminated at existing forums and events upon completion. The team also will use social media to solicit inputs and engage in dialogue over themes explored in the evaluation. It is expected that there will be large interest in the dissemination of the findings among internal and external stakeholders. Appropriate outreach venues will be identified, including the Bank group’s annual meetings and academic and policy workshops and conferences. Dissemination of the findings to 18 Bank staff will be conducted through workshops and learning events and externally through seminars and conferences. Resources 52. The evaluation will be undertaken in the 2015 and 2016 fiscal years, with the CODE discussions held in the fourth quarter of the 2016 fiscal year. The evaluation team will be led by Erik Bloom (Senior Economist), who will be responsible for supervising the evaluation as well as drafting the final report. The team will include Susan Caceres, Ann Flanagan, and Viktoriya Yevsyeyeva, supported by several consultants including Karol Acon, Robert Alcala, Shang Gao, Nazumi Takeda, Mercedes Vellez, Ayenachew Aseffa Woldegiyorgis, Robert Yin, and Simone Zhang. Other staff and consultants will be included as needed. References Agrawal, J., P. Murthy, M. Philip, S. Mehrotra, K. Thennarasu, J. John, N. Girish, V. Thippeswamy, and M. Isaac. 2011. “Socio-Demographic Correlates of Subjective Wellbeing in Urban India.” Social Indicators Research 101 (3): 419–434. Ahmed, H. 2010. “Non-Market Returns to Women Education in Sudan: Case of Fertility.” Journal of Comparative Family Studies 41 (5): 783–798. Altbach, P., L. Reisberg, and L. Rumbley. 2009. Trends in Global Higher Education: Tracking an Academic Revolution. Paris, France: UNESCO. Andresen, C., T. Bunda, and M. Walter. 2008. “Indigenous Higher Education: The Role of Universities in Releasing the Potentail.” Australian Journal of Indigenous Education 1–8. Barone, C. 2011. “Some Things Never Change: Gender Segregation in Higher Education across Eight Nations and Three Decades.” Sociology of Education 157–176. Barro, R., and J. Lee. 2013. “A New Data Set of Educational Attainment in the World, 1950– 2010.” Journal of Development Economics 184–198. Bloom, D., D. Canning, K. Chan, and D. Luca. 2013. Higher Education and Economic Growth in Africa. Cambridge, Mass.: Harvard University. Bloom, D., M. Hartley, and H. Rosovsky. 2006. “Beyond Private Gain: The Public Benefits of Higher Education.” In International Handbook of Higher Education, edited by J. Forest and P. Altbach, chapter 15. Dordrecht, Netherlands: Springer. Campbell, D., and E. Carayannis. 2013. “Conceptual Definition of Two Key Terms: Governance and Higher Education.” In Epistemic Governance in Higher Education, by D. Campbell and E. Caravannis, 3–11. New York: Springer. Chao, R. 2014. “Idealism and Utilitarianism in Internationalization of Higher Education.” International Higher Education 3–4. Commission on Growth and Development. 2008. The Growth Report. Washington, DC: World Bank. Dee, T. 2004. “Are There Civic Returns to Education?” Journal of Public Economics 1697– 1720. Dolreux, D., and S. Parto. 2005. “Regional Innovation Systems: Current Discourse and Unresolved Issues.” Technology in Society 133–153. 19 Economist Intelligence Unit. 2014. High University Employment, Low Graduate Employment: Analysing the Paradox in Afghanistan, India, Nepal, Pakistan and Sri Lanka. London, United Kingdom: British Council. Education for Employment. 2012. Education for Employment: Realizing Arab Youth Potential. Washington, DC: International Finance Corporation. Evans, P., and S. Ebersold. 2012. “Achieving Equity in Secondary and Tertiary Education for Students with Disabilities and Learning Difficulties. “ In Lessons in Education Equality, by J. Heyman and A. Cassola, 78–97. Oxford, United Kingdom: Oxford University Press. Fasih, T., G. Kingdon, H. Patrinos, C. Sakellariou, and M. Soderbom. 2012. Heterogenous Returns to Education in the Labor Market. Washington, DC: World Bank. Fetters, M., P. Green, and M. Rice. 2010. The Development of Universtiy-Based Entrepreneruship Ecosystems: Global Practices. Cheltenham, United Kingdom: Edward Elgar Publishing. Filmer, D. 2003. The Incidence of Public Expenditures on Health and Education. Washington, DC: World Bank. Fonkeng, G., and A. Ntembe. 2009. “Higher Education and Economic Development in Africa: The Case of Cameroon.” Educational Research and Review 4 (5): 231–246. Ford Foundation. 2008. Pathways to Higher Education. New York: Ford Foundation. Guloba, M., N. Magidu, and J. Wokadala. 2010. Public Spending in the Education Sector in Uganda: Evidence from Benefit Incidence Analysis. Kampala, Uganda: Economic Policy Research Center. Gyimah-Brempong, K., O. Paddison, and W. Mitiku. 2006. “Higher Education and Economic Growth in Africa.” Journal of Development Studies 42 (3): 509–529. Heckman, J., and Y. Ji. 2012. "Human Capital, Economic Growth, and Inequality in China." NBER Working Paper No. 18100, National Bureau of Economic Research, Cambridge, Mass. Hénard, Fabrice, and Alexander Mitterle. 2009. Governance and Quality Guidance in Higher Education. Paris, France: Organisation for Economic Co-operation and Development. IEG (Independent Evaluation Group). 2013. Youth Employment Programs: An Evaluation of World Bank and International Finance Corporation Support. Washington, D.C.: World Bank. doi: 10.1596/978-0-8213-9794-7. License: Creative Commons Attribution CC BY 3.0. IFC (International Finance Coporation). 2001. Investing in Private Education: IFC’s Strategic Directions. Washington, DC: IFC. Jung, C.S. 2011. “Higher Education Development in Korea: Western University Ideas, Confucian Tradition, and Economic Development.” Higher Education 59–72. Kapur, D., J. Lewis, and R. Webb. 1997. The World Bank, Its First Fifty Years. Washington, DC: Brookings. Kent, R. 2005. “Two Positions in the International Debate about Higher Education.” Washington, DC: Latin American Studies Association. Lebeau, Y., R. Stumpf, R. Brown, M. Abrahão, S. Lucchesi, and M. Kwiek. 2012. “Who Shall Pay for the public Good? Comparative Trends in the Funding Crisis of Public Higher Education.” Compare: A Journal of Comparative and International Education 137–157. Lucas, R. 1988. “On the Mechanics of Economic Development.” Journal of Monetary Economics 3–42. 20 Malik, S., and K. Courtney. 2011. “Higher Education and Women’s Empowerment in Pakistan.” Gender and Education 23 (1): 29–45. Marginson, S. 2006. “The Anglo-American University at its Global High Tide.” Minerva: A Review of Science, Learning, and Policy 65–77. McMahon, W. 2009. Higher Learning, Greater Good: The Private and Social Benefits of Higher Education. Baltimore, Md.: Johns Hopkins University Press. Montenegro, C., and H. Patrinos. 2014. Comparable Estimates of Returns to Schooling Around the World. Policy Research Working Paper No. WPS 7020, World Bank Group, Washington, DC. Müller, T. 2004. “‘Now I Am Free—Education and Human Resource Development in Eritrea: Contradictions in the Lives of Eritrean Women in Higher Education.” Compare: A Journal of Comparative and International Education 34 (2): 215–229. Oketch, M., T. McCowan, and R. Schendel. 2014. The Impact of Tertiary Education on Development: A Rigorous Literature Review. London, United Kingdom: U.K. Department for International Development. Psacharopoulos, G.. 1994. “Returns to Investment in Education.” World Development 1325– 1343. Psacharopoulos, G., and H. Patrinos. 2004. “Returns to Investment in Education: A Further Update.” Education Economics 12 (2): 111–134. Ramirez, F., and D. Tiplic. 2014. “In Pursuit of Excellence? Discursive Patterns in European Higher Education Research.” Higher Education 439–455. Rhoads, R. 2011. “The U.S. Research University as a Global Model: Some Fundamental Problems to Consider.” InterActions: UCLA Journal of Education and Information Studies. http://escholarship.org/uc/item/8b91s24r. Rigoglioso, M. 2014. “The Demand for ‘World-Class Universities’.” June 23. https://ed.stanford.edu/news/demand-world-class-universities-what-driving-race- top?newsletter=true. Romer, P. 1990. “Endogenous Technical Change.” Journal of Political Economy 71–102. Salmi, J. 2009. The Challenge of Establishing World-Class Universities. Washington, DC: World Bank. Sansone, G., L. Raute, G. Fong, M. Pednekar, A. Quah, M. Bansal-Travers, P. Gupta, and D. Sinha. 2012. “Knowledge of Health Effects and Intentions to Quit among Smokers in India: Findings from the Tobacco Control Policy (TCP) India Pilot Survey.” International Journal of Environmental Research and Public Health 9 (2): 564–578. Schady, N. 2003. “Convexity and Sheepskin Effects in the Human Capital Function: Recent Evidence for Filipino Men.” Oxford Bulletin of Economics and Statistics 171–196. Schultz, T. 2004. “Social Value of Research and Technical Skills: Does It Justify Investment in Higher Education for Development?” Journal of Higher Education in Africa 2 (1): 92– 134. Shafiq, M. 2010. “Do Education and Income Affect Support for Democracy in Muslim Countries? Evidence from the Pew Global Attitudes Project.” Economics of Education Review 29 (3): 461–469. Shierholz, H., A. Davis, and W. Kimball. 2014. The Class of 2014: The Weak Economy Is Idling Too Many Young Graduates. Washington, DC: Economic Policy Institute. 21 Stengos, T., and Aurangzeb, A. 2008. “An Empirical Investigation of the Relationship between Education and Growth in Pakistan.” International Economic Journal 22 (3): 345–359. Task Force on Higher Education and Society. 2000. Higher Education in Developing Countries. Washington, DC: World Bank. The Economist. 2014. “Building a Global University Brand.” The Economist, January 30. Todtling, F., and M. Trippl. 2005. “One size Fits All? Towards a Differentiated Regional Innovation Policy Approach.” Research Policy 1203–1219. UNESCO (United Nations Educational, Scientific, and Cultural Organization). 1998. World Declaration on Higher Education for the Twenty-First Century: Vision and Action. Paris, France: UNESCO. UNESCO Institute of Statistics. 2014. “Global Flow of Tertiary-Level Students.” http://www.uis.unesco.org/Education/Pages/international-student-flow-viz.aspx. ______. 2015. “Glossary.” UNESCO Institute of Statistics. http://www.uis.unesco.org/Pages/Glossary.aspx. Usher, A. 2012. “The Limits of Vocational Education in Developing Countries.” Higher Education Strategy Associates. Accessed 2015. http://higheredstrategy.com/the-limits-of- vocational-education-in-developing-countries/. Wang, Y., and Y. Yao. 2003. “Sources of China’s Economic Growth, 1952–99: Incorporating Human Capital Accumulation.” China Economic Review (World Bank) 32–52. Whalley, J., and X. Zhao. 2010. The Contribution of Human Capital to China’s Economic Growth. Cambridge, Mass.: National Bureau of Economic Research. World Bank. 1995. Priorities and Strategies for Education. Washington, DC: World Bank. ______. 1999. Education Sector Strategy. Washington, DC: World Bank. ______. 2005. Education Sector Strategy Update. Washington, DC: World Bank. World Health Organization. 2011. “World Report on Disability, 2011.” http://whqlibdoc.who.int/hq/2011/WHO_NMH_VIP_11.01_eng.pdf?ua=1. 22 Attachment 1 Evaluation Design Matrix Evaluation Method Approach and Limitations consultations Stakeholder Portfolio analysis strategies Review of Bank Group knowledge work Review of Bank Group informants Interview with key Country case studies Evaluation Questions Evaluation Question 1: Is the World Bank Group’s support for higher education consistent and well-articulated? How has the Bank Group incorporated higher education in     Evaluation question focuses on the its strategic documents? conceptual design of Bank Group How does it coordinate support for higher education      support for higher education internally within the Bank Group?  The evaluation will use qualitative evidence to develop a framework that How does it coordinate support for higher education with      will be used to assess the Bank external development partners and non-government actors? Group’s designs How does it conceptualize higher education and      incorporate local context into the design of its operations? Evaluation Question 2: How has World Bank Group support contributed to higher education systems? How has the Bank Group contributed to changes in the    Results will use qualitative data from financial sustainability and management of higher projects education systems?  The field-based country studies will be How has its support strengthened the connection between    an important source of quantitative  data to complement qualitative data higher education and both the public and private sectors? 23 Evaluation Method Approach and Limitations consultations Stakeholder Portfolio analysis strategies Review of Bank Group knowledge work Review of Bank Group informants Interview with key Country case studies Evaluation Questions How has it supported regulation and quality assurance in    public and private universities? How has its support contributed to internal efficiency in   higher education? Evaluation Question 3: How has the World Bank Group’s support for higher education contributed to social and economic outcomes? How has Bank Group support improved access and equity     Focus is on outcomes that are directly for lower income households? influenced by the higher education sector How has its support addressed gender and other issues  Some outcomes can be measured    affecting traditionally excluded groups in higher quantitatively while other can only be education? evaluated qualitatively How has its support contributed to external efficiency through developing skills and improving the employability    of graduates? How has its support contributed to external efficiency through private sector development and increased industry    competiveness? How has its support contributed to the quality of research    and its relevance to local development challenges? 24 Attachment 2 Challenges to Higher Education in Developing Countries Many countries face a demographic and education transition that is expanding the number of potential students. Table 2A.1 shows the increase in the gross enrollment rate in lower secondary education and in higher education by Bank Group Region. In most countries, the coverage of secondary education has grown substantially over the past two decades. At the same time, a higher proportion of students is transitioning from secondary to higher education. The growing demand for higher education is leading to political challenges in many developing countries. Rapid growth in enrollment has added pressure on the higher education system to ensure equitable access and quality across a growing number of providers. This situation has sparked controversy as countries seek to address the cost of higher education and attempt to divide this cost between students and the government (Lebeau and others 2012). Table 2A.1. Gross Enrollment Rate for Lower Secondary and Higher Education, by Region Region Gross Enrollment Rate 1995 2000 2005 2010 Africa Lower secondary N.A. 30.6 38.6 47.6 Lower secondary 69.3 73.4 84.7 93.8 East Asia and Pacific Higher education 6.9 11.4 19.2 24.3 Lower secondary 88.0 89.7 95.3 97.8 Europe and Central Asia Higher education 26.4 30.0 40.1 47.3 Lower secondary N.A. 97.9 99.4 100.2 Latin America and the Caribbean Higher education 16.6 21.9 29.8 39.7 Lower secondary 72.4 82.3 88.0 91.7 Middle East and North Africa Higher education 15.7 20.0 23.3 31.0 Lower secondary 59.8 58.1 67.6 76.2 South Asia Higher education 5.1 8.2 9.6 16.0 Source: World Bank EdStats database. Note: N.A. = not available. The rapid growth of demand for higher education presents challenges for delivering quality in teaching and learning. Student populations are expanding, often entering the higher education system with lower levels of academic preparation and a more diverse set of academic and nonacademic needs. With class sizes and teaching loads increasing, many universities have supplemented or replaced full-time academic staff with lower-cost part-time faculty, or they have hired staff with lower academic qualifications. Private universities, including for-profit higher education institutions, have helped to meet some of the increased demand, but they often pose quality concerns (Altbach, Reisberg, and Rumbley 2009; Economist Intelligence Unit 2014). Many universities also lack infrastructure to provide quality learning environments as they attempt to accommodate growing enrollments. 25 There is concern about the equity of public subsidies for higher education as the wealthier segments of society tend to have better access to a university education. In most low- and middle-income countries, few individuals from poor households graduate from upper secondary school. Poorer households often have no physical access to the secondary level or have access only to low quality schools. The pattern in figure 2A.1 shows that the enrollment rate for secondary education is usually substantially higher for the richest quintile than for the poorest quintile. At the same time, wealthier households have access to the best quality schools. On a per capita basis, public subsidies for higher education students are higher than subsidies in other levels of education. Since university students tend to come from wealthier households, public subsidies tend to be regressive. For example, in Uganda in 2005, the subsidy per university student was around $175, compared to $35 per secondary student and $21 per primary student; and 51 percent of students came from the highest decile, compared to 12 percent from the lowest five expenditure deciles (Guloba, Magidu, and Wokadala 2010). These results are typical for low-income countries. The same bias exists in middle income countries, although it is generally not as severe. For example, in 2001 the poorest quintile in Azerbaijan and Costa Rica received 12 percent of public subsidies for higher education (Filmer 2003). Figure 2A.1. Upper Secondary Enrollment Rate for the First and Fifth Quintiles Source: World Bank EdStats database. Note: Estimates are drawn from the Demographic and Health Surveys (DHS) and the Multipurpose Indicator Cluster Surveys (MICS). Access to higher education can also be skewed by gender and ethnicity. While traditionally, higher education institutions have been populated largely by men, access for women has been expanding rapidly, though unevenly, across regions. The proportion of women in higher education tends to be lower in poorer countries. Gender equality in higher education tends to follow gender equality in secondary education; while in some regions women are overrepresented in higher education (Barone 2011; Barro and Lee 2013). Ethnic minorities are often underrepresented in higher education as the result of a variety of factors, such as inadequate academic preparation, inability to afford higher education, and discriminatory admissions processes. Many countries have introduced policies to address this disparity, with varying degrees of success (Ford Foundation 2008; Andresen, Bunda, and Walter 2008). 26 Students with disabilities often face significant challenges in accessing higher education. Relatively little is known about access for students with disabilities in developing countries, but the lack of physical accommodations, transition programs to prepare them for entry into higher education, inclusive instruction, and academic support remain significant barriers (Evans and Ebersold 2012; World Health Organization 2011). The lack of proper accommodations and support can lead to poor retention, educational achievement, and employability. Increasing the employability of graduates is another challenge for developing countries. In many countries, unemployment is high even among graduates of post-secondary institutions. While the expansion of access to higher education requires corresponding macroeconomic development to ensure productive work for graduates (Bloom and others 2013; Shierholz, Davis, and Kimball 2014), higher education systems often do not adequately address the needs of the labor market (Education for Employment 2012). A discrepancy is found between the kinds of skills demanded by labor markets and the number of graduates with adequate skills, including soft skills and technical skills in key disciplines (Economist Intelligence Unit 2014). This gap results from the low quality of education and the imbalance between skilled labor supply and demand, with some fields facing shortages while others have too many graduates. Most higher education systems in developing countries have insufficient capacity in the areas of basic and applied research. New pressures on university systems in developing countries further threaten their capacity to support research. These pressures include reduced public subsidies for higher education accompanied by rapidly growing demand for enrollment in higher education. Higher education institutions increasingly rely on tuition revenue from these growing enrollments. However, in the process, many of these institutions have lowered academic standards and diluted teaching labor while also shifting their operations away from conducting research and educating the next generation of researchers (Altbach, Reisberg, and Rumbley 2009). Many higher education systems face serious challenges with internal efficiency. In the face of rapidly increasing enrollments, student progress toward successful degree completion has become increasingly strained. Many countries have seen an increase in enrollment without a subsequent increase in graduates. For example, between 2003 and 2013, the number of higher education students more than doubled while the number of graduates only increased by 50 percent. While data on these trends remain sparse, few developing countries appear to have experienced a significant increase in the graduation rate during the past decade. Many higher education institutions lack the capacity to address many of the critical variables affecting internal efficiency, such as student retention, transition programs, student support, and time to degree completion. Expansion of the higher education sector is putting pressure on the governance systems for higher education. Governance systems may allow institutions a high degree of autonomy, or they may be linked to a quality assurance system that imposes greater structure and oversight of institutions. Both models of governance face challenges of leadership, strategic direction, and financial management. These challenges are enhanced by rapidly increasing demand; reduced public subsidies; proliferation of new types of institutions; new forms of delivery that include competitors to traditional colleges and universities; more diverse student populations; 27 internationalization; and pressures on institutions to leverage their knowledge production (Henard and Mitterle 2009). Promoting institutional autonomy is not sufficient to allow for rapid, relevant, and sustainable adjustment to the changing landscape (Salmi 2009). Similarly, a more structured and monitored governance system cannot ensure efficiency and accountability without improving capacity in accreditation and quality assurance. Most countries lack mechanisms to provide effective quality assurance. Quality assurance provides a means of ensuring that the higher education system responds in a timely fashion to the changing needs of society and the economy. Quality assurance has increasingly relied on data and indicators that demonstrate “outcomes” of higher education and that comply with internationalized standards (Altbach, Reisberg, and Rumbley 2009). In many countries, the capacity to carry out these processes is not well established, rigorous, or sufficiently funded. 28 Attachment 3 Evaluation Methodology The evaluation will utilize a variety of methods to develop an understanding of the Bank Group’s support for higher education. Evidence will be triangulated and synthesized to answer the respective evaluation questions. The goal of this approach is to understand the Bank Group’s support for higher education rather than to provide quantitative grades or ratings of its support. FIRST STEPS: SETTING THE STAGE Although the evaluation’s primary focus is on analysis, it will include a detailed description of what the Bank Group supports and the impact of this support. This will be built around an ongoing portfolio review. The portfolio review will look at both the design and results of the Bank Group’s higher education operations and will classify the lessons reported in project documents and evaluations. Through consultations with stakeholders and a review of the academic and policy literature, the evaluation team will prepare an analysis of the state of thinking on higher education in country contexts with reference to the Bank Group’s involvement and more broadly. This will provide a benchmark or normative framework to which the evaluation can compare the objectives and design of the Bank Group’s interventions. The First Evaluation Question: Is the World Bank Group support for higher education consistent and well-articulated? The evaluation will use the review of the Bank Group strategies to understand the logic model for higher education at the corporate, Regional, and country level. Then, using the normative framework, the evaluation team will analyze the Bank Group’s logic model compared with the views of stakeholders. This will lead to a qualitative discussion of the relevance of the Bank Group’s higher education strategy. A similar approach will be used to evaluate the design of higher education operations. The evaluation will use the portfolio review to classify and analyze the design of investment operations and the review of knowledge work for technical support. In both cases, the country case studies will provide detailed data to complement the analyses of strategies and portfolios. Interviews with key informants will also provide an understanding of all three of the subquestions. The interviews with key informants, along with the country case studies, are especially important for understanding how the Bank Group coordinates internally and with other stakeholders. Together, this evidence will be useful to identify its logic model for higher education. The Second Evaluation Question: How has World Bank Group support contributed to higher education systems? The portfolio analysis and the review of the Bank Group’s knowledge work will provide data on the performance of operations in transforming higher education systems. The country case studies will provide additional data in the selected countries and will include specific questions tailored to evaluate the Bank Group’s contribution to strengthening the higher education system. Interviews with key informants will provide corroborating information to understand results and value added. 29 The Third Evaluation Question: How has the World Bank Group’s support for higher education contributed to social and economic outcomes? The portfolio analysis will provide data on the performance of Bank Group operations on higher level outcomes. The inquiry will require the application of the Bank Group’s logic model in the interpretation of social and economic outcomes. The country case studies will provide additional data and will include specific questions tailored to the country and higher education system context when evaluating the Bank Group’s contributions. Interviews with key informants will provide corroborating information to understand results and value added. CONSULTATION WITH COUNTRY AND REGIONAL HIGHER EDUCATION STAKEHOLDERS The evaluation team will hold consultation with higher education leaders at the country and regional levels to identify key issues in the sector. This might include government officials, university administrators and staff, students, the private sector, and NGOs. The evaluation will also explore options to dialogue with students and university staff using social media or similar approaches. The consultations will be used to identify received understanding of the different options to support efforts to increase the coverage, quality, and relevance of higher education. The evaluation team will work with the Bank Group and external experts to identify policy and academic leaders and will consider a number of different mean to hold the consultations, including online and through interviews, among others. PORTFOLIO REVIEW The portfolio review will examine project documents and evaluations (i.e., ICRs, ICR Reviews, and PPARs). The evaluation will develop instruments to code areas of interest from the portfolio to answer particular evaluation questions. This will include information about project design from Project Appraisal Documents (PADs) as well as information about project execution and results from ICRs, ICR Reviews, and PPARs. The initial screening was based on data from the World Bank Business Warehouse, which was downloaded on December 1, 2014. The higher education projects included in this evaluation were identified based on the following steps: (i) the basic data for all 3,734 projects approved between FY03 and FY14 was downloaded from Business Warehouse on December 1, 2014; (ii) 263 projects mapped to the sector code ET (tertiary education) and the theme code 66 (education for the knowledge economy) were extracted; and (iii) project activities for these 263 projects were examined through the quick PAD review to refine the selection of higher education projects based on the following criteria: 1. The project is considered a “higher education project” and included in the evaluation if it either:  Includes the activities that support higher education institutions, which offer a degree program (e.g., diploma or bachelor’s, master’s or doctoral degree). This may include quality enhancement, institutional strengthening, capacity development, enhancement of facilities, system reform, and policy reform of higher education. 30  Targets students in higher education institutions as main beneficiaries of project activities. 2. A project that is classified as ET is excluded if its focus differs from the evaluation’s objectives. This includes projects that:  Primarily support non-degree programs at the higher education level, including short- term training and certificate programs.  Interact with higher education only through the services provided by higher education institutions to the project. 3. The private sector portfolio includes all projects classified as “colleges, universities, and professional schools,” followed by an inspection to ensure appropriateness. A similar approach is used to identify relevant knowledge products and advisory service products. In addition, the evaluation will divide higher education operation into two categories. The first, core higher education operations, refers to projects whose primary function is to support higher education. A second group of operations, other higher education operations, are projects that include a higher education intervention or objective as part of a larger project. This could include a health sector project that includes an effort to strengthen a medical school along with many other health interventions. The classification was determined by the World Bank Group’s classifications. For public sector projects, projects were that were classified as 50 percent or more ET are identified as core operations. In addition, the evaluation team inspected other public sector projects and identified those projects that primarily focused on higher education. All IFC projects that focus on higher education (colleges, universities, and professional schools) are classified as core. REVIEW OF BANK GROUP STRATEGIES The evaluation will analyze corporate, sector, and country strategies as well as associated documents, such as Systematic Country Diagnoses and policy documents that aim to inform Bank Group management on higher education. The evaluation will develop instruments to code areas of interest from the countries to answer particular evaluation questions to understand how the Bank Group incorporates higher education in its strategies. REVIEW OF BANK GROUP KNOWLEDGE WORK The evaluation will review a range of the Bank Group’s knowledge products, including non- lending technical assistance, economic and sector work, sector analyses, advisory services, economic analysis, and impact evaluations as well as policy dialogue that supports higher education. The evaluation will review the knowledge portfolio using the Business Portfolio for products classified as sector code ET and the theme code 66. In addition, the evaluation team will consult with the Bank Group to identify improperly coded knowledge work. The evaluation will develop an instrument based on the evaluation questions to extract necessary information for the evaluation. 31 REVIEW OF ACADEMIC AND POLICY LITERATURE An initial review shows that there are few rigorous quantitative evaluations focusing on higher education, particularly for interventions in low- and middle-income countries. Given this paucity of research, there is little room to carry out a systematic review of evidence. The evaluation will review the existing academic and policy literature to develop a database of evidence and policy approaches. The literature review will be partly guided by the results of the consultations with the country and global higher education stakeholders. INTERVIEW WITH KEY INFORMANTS In addition to interviews carried out as part of the field-based cases studies and the interaction with country and global higher education leaders, the evaluation team will meet with Bank staff and consultants and with development partners, higher education specialists, NGOs, and foundations. These interviews will provide general context and address specific evaluation questions using a qualitative instrument based on the evaluation questions. COUNTRY CASE STUDIES The country case studies will not focus on information already available in project document and project evaluation documents. An instrument will be prepared to gather information during the country visits based on the evaluation questions. This will include questions on how well the Bank Group has responded to the country’s needs in higher education, what its support focuses on, and how the Bank Group incorporates lessons and evidence in the design and implementation of projects. The country case studies will include interviews with Bank Group and government staff as well meetings with beneficiaries and the development community. 32 Attachment 4 Preliminary Portfolio Review IDENTIFICATION OF WORLD BANK GROUP SUPPORT TO HIGHER EDUCATION Table 4A.1 presents a summary of the World Bank’s lending portfolio, based on operations approved from 2003 to 2014. It includes all projects classified by the evaluation as “higher education.” In terms of number of projects, around 30 percent of education projects are in higher education. Table 4A.2 presents a summary of IFC’s private sector lending program. Table 4A.1. Distribution of World Bank Public Sector Investment Amount Number of Projects (US$, millions) Tertiary education projects IBRD IDA Total IBRD IDA Total 46 85 131 5,619.0 4,707.8 10,326.8 Sector board ED 28 59 87 2,502.5 3,843.4 6,345.9 Non-ED 17 27 44 3,116.5 864.4 3,980.9 Project status Active 19 40 59 1,149.3 2,594.9 3,744.2 Closed 27 45 72 4,469.7 2,112.9 6,582.6 Lending instrument type Investment 36 76 112 2,680.9 4,279.5 6,960.4 DPL 10 9 19 2,938.2 428.3 3,366.5 Region AFR 2 38 41 15.0 1,848.6 1,863.6 EAP 9 12 21 407.4 826.6 1,234.0 ECA 14 11 25 2,262.8 257.3 2,520.1 LAC 16 2 18 2,176.4 14.2 2,190.6 MNA 6 2 8 757.4 19.0 776.4 SAR 0 20 20 0.0 1,742.1 1,742.1 Source: IEG Data. Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; ED = Education; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. 33 Table 4A.2. Distribution of World Bank Private Sector Investment Amount Number of Projects (US$, millions) Tertiary education investment projects Equity Loan Mixed Total Equity Loan Total 8 20 1 29 178.7 359.0 537.7 Project status Active 8 17 0 25 177.1 312.5 489.6 Closed 0 3 1 4 0.0 46.5 46.5 Region AFR 0 4 0 4 0.0 18.4 18.4 EAP 0 1 1 2 1.6 23.0 24.6 ECA 0 4 0 4 0.0 78.0 78.0 LAC 4 11 0 15 54.0 239.6 293.6 MNA 2 0 0 2 18.1 0.0 18.1 SAR 0 0 0 0 0.0 0.0 0.0 World 2 0 0 2 105.0 0.0 105.0 Source: IEG Data. Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. 34 Attachment 5 List of Completed and Current PPARs Table 5A.1. Completed and Current PPARs PPAR Project Approval Closed Relevance and Country Project Title FY ID FY FY Remarks 2013 P050741 2003 2010 Sri Lanka Improving Curriculum and Relevance and institutional reform to Quality of improve employability Undergraduate of university graduates Education Project 2012 P077282 2003 2007 Chile Science for the Support for research Knowledge and scholarships at Economy Project doctoral level in science and technology sector P074138 2003 2009 Colombia Higher Education- Support for low- Improving Access income students to Project improve equity; quality improvement of doctoral programs P050658 2001 2007 India Technician Support for Education III polytechnics at Project undergraduate diploma level P072123 2003 2009 India Technical and Reforms in technical Engineering and engineering Education Quality education at HE level Improvement to improve Project competitiveness 2011 P069326 2000 2007 Jordan Higher Education Strengthening of Development governance and Project systems in HE sector P056236 2002 2009 Egypt, Arab Higher Education Legislative reform, Rep. Enhancement institutional Project restructuring, and establishment of QA and monitoring mechanisms P076183 2002 2008 Yemen, Rep. Higher Education Preparation and Learning and implementation of HE Innovation Project sector reform strategy and capacity development of ministry P001786 1999 2006 Mozambique Education Sector One component Strategic Program focuses on HE strategy for expansion and quality Note: HE = higher education; PPAR = Project Performance Assessment Report; QA = quality assurance. 35