Document of The World Bank Report No.: 97849 PROJECT PERFORMANCE ASSESSMENT REPORT VIETNAM SECOND HIGHER EDUCATION PROJECT (P079665) June 30, 2015 IEG Public Sector Evaluation Independent Evaluation Group ii Currency Equivalents (annual averages) Currency Unit = Vietnamese Dong (VND) 2008 US$1.00 VDN16.30 2009 US$1.00 VDN17.06 2010 US$1.00 VDN18.61 2011 US$1.00 VDN20.50 2012 US$1.00 VDN20.82 Abbreviations and Acronyms FM Financial Management HEP1 First Higher Education Project HEP2 Second Higher Education Project HERA Higher Education Reform Agenda ICR Implementation Completion and Results Report IDA International Development Association IEG Independent Evaluation Group IEGPS IEG Public Sector Evaluation ISR Implementation Status and Results Report MOET Ministry of Education and Training PHRD Policy and Human Resource Development PAD Project Appraisal Document PHRD Policy and Human Resource Development PPAR Project Performance Assessment Report PMU Project Management Unit TRIG Training and Research Innovation Grant Fiscal Year Government: January 1 – December 31 Director-General, Independent Evaluation : Ms. Caroline Heider Director, IEG Public Sector Evaluation : Mr. Marvin Taylor-Dormond (Acting) Manager, IEG Public Sector Evaluation : Mr. Mark Sundberg Task Manager : Mr. Erik Bloom iii Contents Principal Ratings ............................................................................................................... vii Key Staff Responsible....................................................................................................... vii Preface................................................................................................................................ ix Summary ............................................................................................................................ xi 1. Background and Context................................................................................................. 1 Socio-Economic Context ................................................................................................ 1 Education Sector ............................................................................................................. 2 Higher Education in Vietnam ......................................................................................... 2 2. Objectives, Design, and their Relevance ........................................................................ 5 The Project’s Objectives ................................................................................................. 5 The Project Design .......................................................................................................... 5 Financing......................................................................................................................... 7 Relevance of Objectives ................................................................................................. 8 Relevance for the Government’s Strategy .................................................................. 8 Relevance for the Bank’s Country Partnership Strategy .......................................... 10 Relevance of Design ..................................................................................................... 10 Technical Design ...................................................................................................... 10 Results Chain ............................................................................................................ 12 Choice of Instruments and Financing ....................................................................... 12 3. Implementation ............................................................................................................. 12 From Initiation to the Mid-Term Review ..................................................................... 12 After the Mid-Term Review ......................................................................................... 14 Safeguard Compliance .................................................................................................. 15 Fiduciary Issues ............................................................................................................ 15 4. Achievement of the Objectives ..................................................................................... 16 Objective 1: Increase the Quality of Teaching in Order to Improve the Employability of Graduates .................................................................................................................. 17 Outputs ...................................................................................................................... 19 Outcomes .................................................................................................................. 20 Objective 2: Increase the Quality of Research in Order to Improve the Relevance of Research ........................................................................................................................ 21 This report was prepared by Erik Bloom and Hoàng Thanh Hà, who assessed the project in June 2014. The report was peer reviewed by Susan Caceres and panel reviewed by Pia Schneider. Viktoriya Yevsyeyeva provided administrative support. iv Outputs ...................................................................................................................... 21 Outcomes .................................................................................................................. 22 5. Efficiency ...................................................................................................................... 23 External efficiency ........................................................................................................ 23 Internal Efficiency ........................................................................................................ 25 6. Ratings .......................................................................................................................... 25 Outcome ........................................................................................................................ 25 Risk to Development Outcome ..................................................................................... 26 Bank Performance ......................................................................................................... 28 Quality at Entry ......................................................................................................... 28 Quality of Supervision .............................................................................................. 29 Borrower Performance .................................................................................................. 30 Government Performance ......................................................................................... 30 Implementation Agency Performance ...................................................................... 31 Monitoring and Evaluation ........................................................................................... 31 Monitoring and evaluation design ............................................................................ 32 Monitoring and evaluation implementation .............................................................. 32 Monitoring and evaluation utilization ....................................................................... 32 7. Lessons .......................................................................................................................... 33 References ......................................................................................................................... 35 Annex A. Basic Data Sheet ............................................................................................... 37 Annex B. Participating Universities ................................................................................. 40 Annex C. List of Persons Met ........................................................................................... 41 Boxes Box 1.1. The Higher Education Reform Agenda ................................................................ 4 Box 1.2. The German University ........................................................................................ 5 Box 2.1. Design of the Teaching and Research Innovation Grants .................................... 7 Box 3.1. The Higher Education Development Policy Program ........................................ 14 Box 4.1. The View of Students ......................................................................................... 20 Tables Table 2.1. Activities in the Higher Education Project II with Financing from All Sources 6 Table 2.2. Source of Financing at Appraisal under the Higher Education Project II ......... 8 Table 4.1. Intermediate Indicators Associated with Both Objectives ............................... 17 v Table 4.2. Indicators Associated with Objective 1, Group 1 TRIG Universities ............. 18 Table 4.3. Indicators Associated with Objective 1, Group 2 TRIG Universities ............. 19 Table 4.4. Indicators Associated with Objective 2 ........................................................... 21 Table 4.5. Self-assessment of the Impact of TRIGs on Research ..................................... 22 Table 5.1. Summary of Assumptions in PAD................................................................... 24 Table 6.1. Summary Outcome Ratings ............................................................................. 26 Table 6.2. Total Public Spending on Education, 2005-2013 ............................................ 27 Figures Figure 1.1. Enrolment in Higher Education per 100,000 Population ................................. 3 vii Principal Ratings ICR* ICR Review* PPAR Outcome Satisfactory Satisfactory Satisfactory Risk to Development Low to Moderate Negligible to Low Moderate Outcome Bank Performance Satisfactory Moderately Satisfactory Satisfactory Borrower Satisfactory Satisfactory Moderately Satisfactory Performance * The Implementation Completion Report (ICR) is a self-evaluation by the responsible Bank department. The ICR Review is an intermediate IEGWB product that seeks to independently verify the findings of the ICR. Key Staff Responsible Division Chief/ Project Task Manager/Leader Country Director Sector Director Appraisal Jeffrey Waite Christopher J. Thomas Klaus Rohland Completion Thanh Thi Mai Luis Benveniste Victoria Kwakwa viii IEG Mission: Improving World Bank Group development results through excellence in evaluation. About this Report The Independent Evaluation Group assesses the programs and activities of the World Bank for two purposes: first, to ensure the integrity of the Bank’s self-evaluation process and to verify that the Bank’s work is producing the expected results, and second, to help develop improved directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, IEG annually assesses 20-25 percent of the Bank’s lending operations through field work. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which Executive Directors or Bank management have requested assessments; and those that are likely to generate important lessons. To prepare a Project Performance Assessment Report (PPAR), IEG staff examine project files and other documents, visit the borrowing country to discuss the operation with the government, and other in-country stakeholders, and interview Bank staff and other donor agency staff both at headquarters and in local offices as appropriate. Each PPAR is subject to internal IEG peer review, Panel review, and management approval. Once cleared internally, the PPAR is commented on by the responsible Bank department. The PPAR is also sent to the borrower for review. IEG incorporates both Bank and borrower comments as appropriate, and the borrowers' comments are attached to the document that is sent to the Bank's Board of Executive Directors. After an assessment report has been sent to the Board, it is disclosed to the public. About the IEG Rating System for Public Sector Evaluations IEG’s use of multiple evaluation methods offers both rigor and a necessary level of flexibility to adapt to lending instrument, project design, or sectoral approach. IEG evaluators all apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (additional information is available on the IEG website: http://worldbank.org/ieg). Outcome: The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently. The rating has three dimensions: relevance, efficacy, and efficiency. Relevance includes relevance of objectives and relevance of design. Relevance of objectives is the extent to which the project’s objectives are consistent with the country’s current development priorities and with current Bank country and sectoral assistance strategies and corporate goals (expressed in Poverty Reduction Strategy Papers, Country Assistance Strategies, Sector Strategy Papers, Operational Policies). Relevance of design is the extent to which the project’s design is consistent with the stated objectives. Efficacy is the extent to which the project’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Efficiency is the extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared to alternatives. The efficiency dimension generally is not applied to adjustment operations. Possible ratings for Outcome: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Risk to Development Outcome: The risk, at the time of evaluation, that development outcomes (or expected outcomes) will not be maintained (or realized). Possible ratings for Risk to Development Outcome: High, Significant, Moderate, Negligible to Low, Not Evaluable. Bank Performance: The extent to which services provided by the Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of supported activities after loan/credit closing, toward the achievement of development outcomes. The rating has two dimensions: quality at entry and quality of supervision. Possible ratings for Bank Performance: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. Borrower Performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development outcomes. The rating has two dimensions: government performance and implementing agency(ies) performance. Possible ratings for Borrower Performance: Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory. ix Preface This Project Performance Assessment Report (PPAR) assesses the Second Higher Education Project in Vietnam. This project was financed by the government of Vietnam, beneficiary universities, the World Bank’s International Development Association (IDA), and the government of Japan through its Policy and Human Resource Development trust fund. The World Bank approved its contribution on June 20, 2007 and the program closed on June 30, 2012. The World Bank’s contribution through IDA was US$59.4 million, while the government of Japan committed US$4.8 million. The program focused on improving the quality and relevance of higher education of Vietnam and was part of a long-term effort by the government to increase the contribution of higher education to development. This report was prepared by Erik Bloom, Senior Economist, IEG with the assistance of Ms. Hoàng Thanh Hà, consultant. The findings are based on an extensive review of project documents as well interviews and field visits. As much as possible, the PPAR cites publically available documents, and when appropriate, it refers to interviews and internal documents. Following standard IEG procedures, the report was sent to the government officials and agencies in Vietnam for review and comments, no comments were received. xi Summary This Project Performance Assessment Report reviews the impact of the World Bank’s Second Higher Education Project in Vietnam. The project was approved on June 20, 2007 and closed on schedule on June 30, 2012. The World Bank provided a credit for US$59.4 million, of which US$55.5 million were disbursed. The government committed US$0.8 million and participating universities US$5.5 million; total disbursement from local sources was US$5.5 million. The project also benefited from a US$4.8 million grant from Japan’s Policy and Human Resource Development trust fund, of which US$2.6 million were disbursed. The project objective was: to increase the quality of teaching and research in universities in Vietnam so as to improve the employability of graduates and the relevance of research. The project consisted of three components:  Component 1, Capacity Building for Policy Development (Appraised at US$4.8 million). This component aimed to develop policy and build capacity in in governance of higher education, higher education financing, and higher education quality.  Component 2, Capacity Building for University Teacher and Research (appraised at US$61.4 million). This component focused on strengthening teaching and research in selected universities, through the provision of Teaching and Research Innovation Grants (TRIGs).  Component 3, Project Management, Monitoring and Evaluation (appraised at US$4.3 million). This component aimed to support project management and management and evaluation, including support for project management in the areas of planning, financial management, procurement, and communication. The project’s objectives remain very relevant to both the government’s strategy towards higher education as well as the Bank’s country strategy. Vietnam is in the process of modernizing and strengthening the higher education system. It fits within the government’s strategies and supports several areas of its approach towards higher education. Higher education continues to play an important part in the World Bank’s country strategy and the project’s objectives remain relevant for the Bank. Objective 1: Increase the Quality of Teaching in Order to Improve the Employability of Graduates This achievement of this objective was rated substantial. The project contributed to the teaching capacity, leading to greater relevance and improved opportunities for students. The project supported a number of programs to improve the quality of teaching. The project supported both post-graduate training and short-term courses. Universities have seen an increase in the percentage of instructors with masters and doctoral degrees. xii During the project period, average score increased on standardized tests in both group 1 and group 2 universities. The average time of completing undergraduate studies decreased in both groups of universities. In Group 1 Universities, the number of students going overseas within six months of graduation increased from 1.7 to 2.4 percent. The percentage of students getting a job within six months increased from 68.4 to 72.4 percent. However students in Group 2 Universities saw an increase in time to get a job. Objective 2: Increase the Quality of Research in Order to Improve the Relevance of Research. The achievement of this objective is rated high. The project supported 249 research projects. The project supported the development and dissemination of research. TRIGs supported 1,162 national articles and 645 international articles, well exceeding the established target. During the same period, there was an increase in the total number of national and international articles, approximately doubling the annual production. While this cannot be wholly attributed to TRIG, its support played a major role in this increase. Science and technology has paid for a growing share of revenues in participating universities growing from 5.2 percent to 11.8 percent during the project period. Efficiency The project’s efficiency is rated as substantial. The project was executed in timely fashion and the project closed as expected. Most of the proposals for TRIGs were prepared as expected and the PMU was able to disburse resources quickly. Although the procurement was quite complicated, involving international competitive bidding for sophisticated laboratory equipment did not affect the disbursement or the schedule of implementation. The project audits did not show any serious problems with financial management. Given the general success of the research component, the estimates in the PAD probably understate the return. The PPAR estimates that the rate of return was at least 34 percent, above the opportunity cost of capital. Ratings Overall, the project is rated satisfactory. Risk to Development Outcome The risk to development outcomes is rated modest. The project was developed with strong support from the government, as a follow-on project and well aligned with the government’s strategy. The government has continued its commitment to the project’s objectives. It is likely that the government will continue with TRIG. Increasing financial autonomy also provides universities with resources to maintain their investments. Bank Performance Overall Bank performance is rated satisfactory. The design period was quite long, allowing the Bank time for detailed technical work as well as project preparation. The Bank held an identification mission in 2003. This was followed by a number of preparation missions that culminated in appraisal in March 2007. The Bank drew heavily xiii from the lessons from the previous project and worked to make the grants more efficient. The design was informed by consultation and technical work, including the preparation of technical papers. The Bank also prepared detailed economic and financial analyses. The Bank’s supervision was proactive, efficient, and supportive. It was responsive to the project’s need and worked to prevent and solve bottlenecks. This included a fast approvals (“no objection”). The Bank held at least one supervision mission per year, both in Hanoi and in the field. Both Bank task team leaders were based in Hanoi and were supported by local education, operations, and fiduciary staff. Borrower performance The overall borrower’s performance is rated as satisfactory. Central government ministries, the Ministry of Finance and the Ministry of Planning and Finance, fully supported the project. The Ministry of Education and Training was well engaged with the project. It established the PMU and ensured that it was staffed. This was a major part of the Financial Management Action Plan. The Project Management Unit provided sound management to a complex project. The PMU had a strong sense of ownership of the project. Universities indicated that it provided timely technical and fiduciary advice that was crucial in implementing the project, in particular for the complicated procurement processes. The PMU was able to implement an ambitious monitoring and evaluation system that went beyond what was indicated in the PAD. The PMU monitored the status of each university and provided “grades” as well as feedback to improve performance. Monitoring and Evaluation The project had a strong monitoring and evaluation framework. While the World Bank had a relatively underdeveloped results framework, the government had a more detailed and sophisticated results framework. Both sets of results framework had baseline and target indicators. The project was able to monitor all of the required data as expected. Universities provided a variety of information on their inputs as well as outputs and outcomes in a timely fashion. Data played an important role in decision making in the project. The project’s design included a mechanism to allocate TRIGs based on performance, which the PMU used for the second tranche of TRIG. Universities that performed well received additional resources while poorly performing universities lost resources. Lessons The key ingredients for project success often come from beyond the sphere of project itself. In no small way, the project’s close alignment with the reform contributed to its strong performance. The project was developed within the context of an extensive higher education reform that had a high level of sustained government and university support. It was also implemented in the context of a larger World Bank program for higher education as well as full support from the Bank’s Country Strategy. Although it was an investment project, HEP II supported many aspects of the reform, which, in turn, xiv strengthened the project. Many of the actions supported by the project directly feed into the reform and this is reflected in the results framework. As illustrated by the experience of the HEP II, a project can effectively build upon the implementation experience of preceding and parallel operations to enhance both the achievement of objectives. A combination of Bank support instruments can contribute to a complex reform. The Bank’s support for Vietnam’s higher education reform efforts has adopted a multi- pronged strategy, combing investment and development policy operations with knowledge work and a continuous policy dialogue. The investment operations, including HEP 1 and HEP 2 and to a lesser extent the New Model (German) University, supported both analytical inputs and investments to support the reform. In addition, they were aimed at building much needed technical and institutional capacity at the central government and university levels. In turn, the development policy series has provided substantial financial incentives to move forward more with the challenging policy reforms. It is clear from a review of the development policy activities that the support from the HEP II was crucial in carrying many elements that HERA needed to advance. Monitoring and evaluation can support the implementation of investment projects. A robust M&E system was a key feature of the HEP II design and contributed to its success. The PMU actively tracked and monitored implementation, beyond the World Bank’s results framework. This led to development of university level monitoring systems. This was a complex task given the diverse nature of TRIG sub-projects, and the large number of implementing entities. The PMU used the data to provide targeted support to the universities, identifying those that needed additional assistance. The PMU also used these indicators to adjust the grants that universities receive. Universities that were implementing the grant slowly saw a reduction in their allocation in the second tranche, which focused on better performing institutions. While a grant program may play an important role in developing autonomy, it is not sufficient to ensure autonomy. When well defined, grants in higher education can contribute to strengthen the capacity of universities to act autonomously. Competitive grant programs are common in World Bank-financed higher education projects, building on the positive experience of higher education systems in high income countries. Strengthening autonomy was a central tenant of the ongoing higher education reform, HERA. When the project was initiated, universities were centralized under the direct authority of a Ministry or local government. Interviews show that while individual researchers and research teams had some experience in writing grant proposals, university faculties and school lacked this experience. The introduction of TRIG gave faculties an opportunity to work together and to start to act with some autonomy. The evaluation shows that despite this progress, developing university autonomy is more complicated. Caroline Heider Director-General Evaluation 1 1. Background and Context 1.1 This Project Performance Assessment Report (PPAR) reviews the impact of the World Bank’s Second Higher Education Project in Vietnam. The project was approved on June 20, 2007 and closed on schedule on June 30, 2012. The World Bank, through the International Development Association (IDA), provided a credit for US$59.4 million. At appraisal, the government of Vietnam committed US$0.8 million and participating universities US$5.5 million for the project. The project also benefited from a US$4.8 million grant from Japan’s Policy and Human Resource Development (PHRD) trust fund. Socio-Economic Context 1.2 Since the start of market-based reforms twenty-five years ago, known as Doi Moi, Vietnam has seen rapid economic growth. Vietnam transformed itself from one of poorest countries in the world to a lower middle country in one generation. Overall, the per capita gross domestic product has increased from US$290 (in 2005 dollars) in 1989 to US$990 in 2012, with an average 5.3 percent growth rate. The industrial sector has seen an 8.2 average growth rate from 1991 to 2013, growing from 24 percent of the gross domestic economic in 1991 to 28 percent in 2013. The agriculture sector has seen a growth rate of 3.7 percent during the same period, as its relative size decreased from 41 percent to 18 percent of the gross domestic product. The services sector, which tends to rely most heavily on higher education graduates, grew from 35 percent to 44 percent, a growth rate of 7.0 percent (World Bank, 2011; World Bank, 2014). 1.3 Vietnam is passing through a major demographic transition. In 2010, Vietnam had a population of 87.1 million. While the fertility rate declined from 3.7 birth per woman in 1989 to 1.8 in 2011, the country is still relatively young with a quarter of the population below the age of 14 in 2011. Vietnam has a significant ethnic minority population, accounting for around 15 percent of the population, located primarily in the Central and Northern Highland regions (World Bank, 2014). 1.4 Economic growth and transformation has led to a significant reduction in the poverty rate. Using the US$1.25 per day poverty line, the percentage of poor has dropped from 64 percent in 1993 to 17 percent in 2008. Increasingly, extreme poverty is concentrated in the remote rural population, particularly among ethnic minorities. By 2008, around half of the poor were ethnic minorities (World Bank, 2012). While the income distribution remains relatively equitable, there is growing inequality in access to health and education services. 1.5 Economic growth has transformed opportunities for employment. From 1990 to 2013, the national unemployment rate has averaged 2.3 percent and 4.9 percent for youth.1 Both of these rates have remained relatively steady as has the labor participation rate, around 77 to 79 percent. Between 2000 and 2012, the share of the employment in agriculture has dropped from 65 to 47 percent, while the share in services increased from 22 to 32 percent. Women have a high participation rate in the economy of around 71 percent, compared to 81 1 Data are not available on employment rates by level of education. 2 percent for men. Women tend to be more active in agriculture and services, (about 5 percentage points more), while men are more active in industry (about 6 percentage points more), using 2013 data (World Bank, 2015). For university graduates, employment for university graduates is expected to grow by 5.9 percent between 2011 to 2020 (using a computable general equilibrium mode), compared to a 1.4 percent growth rate for all employment (Giesecke, Tran, Meagher, & Pang, 2011). Education Sector 1.6 The education system of Vietnam is divided into five general levels: preschool (ages 3 to 5); primary education (grades 1 to 5); lower secondary education (grades 6 to 9); upper secondary education (grades 10 to 12); and higher education. Most education providers are under the authority of the Ministry of Education and Training (MOET) or local governments. 1.7 In parallel with Vietnam’s fast economic and rapid poverty reduction, access to education has increased substantially. While primary education has been essentially universal for at least a decade, coverage of the secondary education is rapidly increasing. By 2013, the net enrollment for lower secondary education is 93 percent. Likewise, coverage of pre- primary has also increased substantially, from 40 percent in 2000 to 81 percent in 2012. Increasingly, low access to education is confined to the population in remote areas (largely ethnic minorities) and the unregistered urban population. Higher Education in Vietnam 1.8 Prior to Doi Moi, most institutions of higher education were organized around specific sectors or professions. These were supervised by the appropriate Ministry, such as the University of Health under the Ministry of Health. There were only a few general universities, under the Ministry of Higher and Secondary Education.2 The higher education system was largely focused on teaching and producing graduates. Most research was carried out by independent institutes and most post-graduate education was carried out overseas, largely in the Soviet Bloc (St. George, 2010). 1.9 Currently, Vietnam is in a process to modernize and strengthen the higher education system. The approach, introduced in 1993, includes greater centralization of higher education, under the MOET, growth in general universities, and an increased place for research. At the same time, the reforms formalized initiatives to introduce non-governmental institutions and tuition in public universities. The reforms also aimed to break the relationship between the university and state employment and explicitly gave graduates the responsibility to find employment (Hayden & Thiep, 2010; St. George, 2010). 1.10 In 2012, there were an estimated 2.3 million students in higher education, which corresponds to a gross enrollment of around 25 percent (World Bank, 2014). This appears to be similar to enrollment in East Asian middle income countries like China or the Philippines, however it is well below countries like Thailand or the Republic of Korea, which have gross 2 This Ministry was merged into MOET in 1992. 3 enrollment rates between 50 to 100 percent (World Bank, 2013). As can be seen in Figure 1.1, enrollment has grown quickly since 1989 and the enrollment rate has increased ten-fold. 1.11 As part of the reforms introduced in 1993, there has been a consolidation in the higher education sector. In 2007, there were 322 higher education institutions, including 139 universities. In 2004, the government designated 14 universities as “key” higher education institutions.3 Approximately 15 percent of higher education institutions were private, accounting for around 15 percent of total enrollment (Hayden and Thiep 2010). Around 60 higher education institutes are under the MOET with the rest reporting to different ministries and to local governments (World Bank 2013). Figure 1.1. Enrolment in Higher Education per 100,000 Population Source: (World Bank, 2014). 1.12 In general, higher education in Vietnam is orientated towards the job market. Using data from 2008, 33 percent of graduates are in the field of education, 27 percent are in social science, business, and law, and 20 percent are sciences, industry, and engineering. The remaining 20 percent are in agriculture, health, and humanities. A separate study estimates that science accounts for around 4 percent of university enrollment in 2004 (Sheriden, 2010). 1.13 Despite the growth in higher education enrollment, the sector still faces many challenges that the reform seeks to address. In particular, higher education suffers from a lack of quality and relevance. Traditionally, higher education focused on teaching with little role for research. It often employs out-of-date curriculum and lacks necessary equipment. Likewise, there is a lack of a quality assurance system to provide feedback to universities. 3 For the sake of simplicity and following the practice in the Project Appraisal Document, this PPAR will call the higher education institutions “universities.” 4 Faculty members often lack sufficient academic credentials and training to be effective teachers and researchers (World Bank, 2007). These limitations are associated with education financing—universities have limited capacity to charge tuition or to benefit from the research. Faculty salaries were low and there are often not enough instructors to meet the demand. Universities generally lacked the capacity and authority to control their own resources and to make strategic investment. Private universities have limited coverage. Box 1.1. The Higher Education Reform Agenda The Higher Education Reform Agenda 2006-2020 (commonly called HERA) is the government’s long-term agenda to adjust the higher education system to the changing needs of the economy. Its overall objective is to develop a higher education by 2020 that is “advanced by international standards, highly competitive, and appropriate to the socialist-orientated mechanism (Nghi 2010).” HERA has five specific objectives: 1) Completion of a national network of higher education institutions suitable for Vietnam; 2) Full development of a higher education curriculum that supports research and gives student more career options; 3) Continued rapid expansion of the higher education system; 4) An increase in the number of qualified higher education staff to improve quality and decrease the student-to-teacher ratio; and 5) More scientific and technical research in higher education institutions. The agenda proposes a number of activities, including reforming curriculum and teacher methods, increasing fiscal autonomy and granting greater autonomy, expanding physical facilities, investment in teaching and research, and encouraging private and foreign investment. The government estimated that the total cost of HERA will be US$20 billion, of which approximately US$18 billion will be for physical infrastructure. Approximately half of the total is expected from private sources, including tuition paid to both public and private institutions. Source: (World Bank 2008) (World Bank 2013). 1.14 The World Bank has provided significant support to the higher education sector. In 1998, the World Bank approved the First Higher Education Project (P004828) for US$83.3 million. In addition to the Second Higher Education Project (the subject of this evaluation), the World Bank approved a series of three development policy operations (P104694, P116353, and P116354) focusing on implementing the government’s Higher Education Reform Agenda. Currently, the Bank is financing the New Model University Project (P110693), which aims to develop an autonomous research-based university in the context of Vietnam’s higher education policy framework. 1.15 Other development partners have also provided support to higher education. The Asian Development has been active in supporting higher education in Vietnam through the New Model University (see Box 1.2), in this case in collaboration with France. It has also provided knowledge work in higher education (Sheriden, 2010). The German government has provided support and parallel financing to the World Bank’s New Model University. Japan and the United States are also supporting the development of New Model Universities. Countries provide scholarships or have university-to-university cooperation programs. 5 Box 1.2. The German University The World Bank is financing a new university, the Vietnamese-German University. This is part of the government’s New Model University program based on the German model. The University will focus on Engineering, Science, and Technology. The German government is providing technical assistance including providing German instructors and researchers. The World Bank and the government are financing the construction of infrastructure as well as the purchase of equipment. The university is currently being built in Ho Chi Minh City. In addition to the German University, the government is planning to establish a total of four or five new universities, with the expected support of Japan and the United States, among others. The Asian Development Bank is supporting the development of a model university in Hanoi (“the French University”). Source: IEG mission. 2. Objectives, Design, and their Relevance The Project’s Objectives 2.1 The legal agreement between the Government and the World Bank, known as the Development Credit Agreement, states that the Project’s objective were: The objective of the Project is to increase the quality of teaching and research in universities in Vietnam so as to improve the employability of graduates and the relevance of research. 4 2.2 The objectives can be divided into two groups. The first is to increase the quality of teaching in order to improve the employability of graduates. The second is to increase the quality of research in order to improve the relevance of research. 2.3 The project objectives make clear reference to employability and not to employment, which follows the approach taking in the government strategies. In general, employability refers to increased readiness to find and a hold employment (Yorke, 2004): a set of achievements – skills, understandings and personal attributes – that make graduates more likely to gain employment and be successful in their chosen occupations, which benefits themselves, the workforce, the community and the economy. The Project Design 2.4 Following the design in the First Higher Education Project (P004828), the project had three components. Table 2.1 outlines the activities under each component, along with the budget at the time of appraisal. 4 Virtually the same wording is used in the Project Appraisal Document, which modifies the phrase “…so as to improve….” to read “…in ways to improve…” 6 2.5 Component 1, Capacity Building for Policy Development (appraised at US$4.8 million, actual allocation US2.3 million). This component aimed to develop policy and build capacity in policy development in governance of higher education, higher education financing, and higher education quality. The component was targeted at the central level, through different departments within the Ministry. It supported the governance and regulation of the sector. The component supported the introduction of accreditation with minimum standards. This was to support consultants and studies. It also financed an information management and policy information system. Table 2.1. Activities in the Higher Education Project II with Financing from All Sources Appraisal Sub-Component Amount, Name US$ million Activities Component 1: Capacity Building for Policy Development  Technical assistance to develop policy options 1.1: Governance 2.8  Technical assistance for the regulatory framework  Support to build a management information system  Technical assistance for education finance 1.2: Financing 0.8  Technical assistance on student tuition and financial aid 1.3: Quality  Technical assistance to improve quality of students at entry 1.2 Assurance  Technical assistance to develop accreditation Component 2: Capacity Building for University Teaching and Research 2.1: Capacity  Management of TRIGs Building for 1.4  Support to develop university leadership and management Universities  Technical assistance for quality assurance  Group 1 TRIGs, for major (key) universities 2.2: TRIGs 60.0  Group 2 TRIGs, for universities in disadvantaged regions Component 3: Project Management, Monitoring and Evaluation 3.1: Management 3.2  Operational support 3.2: M&E 1.1  Monitoring of implementation and evaluation studies Total 70.5 Note: Includes US$0.8 million in physical and price contingencies, distributed among components Source: (World Bank, 2007). 2.6 Component 2, Capacity Building for University Teacher and Research (appraised at US$61.4 million, actual allocation US$58.2 million). This component focused on strengthening teaching and research in selected universities. The component received the bulk of the project’s financing through its provision of grants of Teaching and Research Innovation Grants (TRIGs). Group one TRIGs were designed to provide support for 17 universities that were designated as “key” by the government and that participated in the First Higher Education Project. Group Two provided support to five universities located in disadvantaged regions. Box 2.1 outlines the design and selection criteria for the TRIGs. The component also supported training and capacity building to allow them to manage the 7 TRIGs. In addition to financing TRIGs, this component includes support to administer and monitor the program including tracer surveys to track changing patterns in student jobs. 2.7 Component 3, Project Management, Monitoring and Evaluation (appraised at US$4.3 million, actual allocation US$2.9 million). This component supported a semi- autonomous Project Management Unit (PMU) created in the Ministry of Education and Training. It was responsible for project management in the areas of planning, financial management, procurement, and communication. The component also supported the collection and analysis of data under a Monitoring and Evaluation Coordinator. Box 2.1. Design of the Teaching and Research Innovation Grants TRIGs play a central role in achieving the project’s objectives and account for approximately 85 percent of the project’s financing. Only pre-identified universities in Group One (“key universities”) and Group Two (“universities in disadvantaged regions”) were eligible to apply. The average grant for Group One was expected to be US$3.2 million (maximum US$5.0 million) and US$1.0 million (maximum US$2.0 million) for Group Two. Grants could include resources for consultants, training, goods and equipment (excluding vehicles), and refurbishment of existing infrastructure. The IDA credit could not finance salaries. Grants would be given based on proposals that aim to improve the quality of teaching and learning through changes in curriculum, teaching methods, and research. Each proposal would include a budget, performance criteria, information on procurement and financial management capacity, as well as data on gender and ethnicity. The Assessment Panel, comprising leading academic staff from universities in Vietnam, international academic staff (for Group One universities), and representatives from the business and science communities, would review the proposals. The Panel would have the option to accept the proposal, request modifications, or reject the proposal. The Ministry’s Project Management Unit would review implementation progress through reports and site visits, which were to be confirmed by an independent monitoring study. Source: Project Appraisal Document. Financing 2.8 The project received financing from four different sources: the IDA grant; recipient Group 1 Universities; the national government; and a grant from the Policy and Human Resource Development (PHRD) trust fund. Table 2.2 outlines the financing by source. 8 Table 2.2. Source of Financing at Appraisal under the Higher Education Project II Amount, US$ Million IDA University Government PHRD Total Component Credit Counterpart Counterpart Grant Component 1: Capacity Building for 1.3 0.0 0.1 3.4 4.8 Policy Development Component 2: Capacity Building for 0.0 0.0 0.0 1.4 1.4 Universities Component 2: TRIGs for Group 1 49.5 5.5 0.0 0.0 55.0 Components 2: TRIGs for Group 2 4.5 0.0 0.5 0.0 5.0 Component 3: Project Management, 4.1 0.0 0.2 0.0 4.3 Monitoring and Evaluation Total 59.4 5.5 0.8 4.8 70.5 Note: PHRD is Policy and Human Resource Development Grant offered by the government of Japan. Source: (World Bank, 2012). 2.9 Total spending under the project was US$63.4 million, around 90 of total spending. Of the total IDA financing of US$59.4 million, a total of US$55.5 million was spent. Supervision documents suggest that part of the difference is due to underspending of grants, which also explains why the universities underspent their contribution (US$4.5 million vs. a planned amount of US$5.5 million). The rest of the cancelled amount was due to reduced spending on components 1 and 2, due to the changes in plans to build an information system, the cancellation of some activities, and cost savings. This also led to the cancellation of around 46 percent of the PHRD (US$ 2.6 million), which was designed to support capacity building activities. Relevance of Objectives The relevance of objectives is defined as the extent to which an operation’s objectives are consistent with the country’s current development priorities and with current Bank…strategies and corporate goals. 2.10 The Relevance of Objectives is rated high. 2.11 The project’s objectives remain very relevant to both the government’s strategy towards higher education as well as the Bank’s country strategy. RELEVANCE FOR THE GOVERNMENT’S STRATEGY 2.12 The project’s objectives are highly relevant to the government’s current strategy for higher education. The government’s ten year development strategy, the Sustainable Development Strategy 2011-2020 (Government of Vietnam, 2012),5 makes specific reference 5 The strategy is often called the Socio-Economic Development Strategy, 2011-2020. All quotes in English are based on non-official government translations for reference, with editing for style. 9 to the importance of increasing the role of research (particularly research in science and technology) as important tools to improve the economy’s competitiveness as well as the relevance of education for the labor market. Specifically the strategy identifies one of its objectives to: focus on educating a workforce capable of meeting the diverse requirements of technology and development levels of sectors of sectors and professions [and] produce workforce for [the development] of a knowledge economy. The strategy also proposes several solutions to meet the goals, including a pledge to: …develop a high-quality workforce, closely combine human resources and science and technology…to restructure the economy [and] transform the growth model. and …step up the comprehensive of Viet Nam’s tertiary education, expanding investments in establishing excellent universities and focus on [human resources] for the knowledge economy. 2.13 The government’s medium term strategies also proposes a strong focus on science and technology as the engine of development. The Social-Economic Development Plans for 2005-2010 and 2011-2015 identify science and technology as playing a significant role in promoting development in agriculture and forestry, industry, and service sectors. The 2011-2015 Plan states: Science and technology development is…the key [factor] for a process of fast and sustainable development. Both development plans emphasize the importance of improving the coverage and quality of higher education (Government of the Socialist Republic of Vietnam, 2006; Government of the Socialist Republic of Vietnam, 2011). 2.14 The government’s objectives are in line with the expected growing demand for education and skilled worker as the economy is transformed. This is particularly the case for Vietnam, where the coverage of higher education has traditionally been lower than many economies in the East Asia. Likewise, at the time of preparation, around 20 percent of firms identified the lack of skilled labor as major or serve bottleneck and another 20 percent identify this as a moderate constraint (World Bank, 2008). 2.15 The project supports a major portion of the government’s strategy in higher education. Likewise, as indicated in Box 2.1, the Higher Education Reform Agenda specifically includes as objectives improving the quality of research and the relevance of teaching, in addition to expanding the coverage of the higher education system. The government’s strategy emphasizes improving quality, relevance, and enrollment. The project strategically focused on the first two elements, aiming to improve workforce skills by focusing on the quality and relevance of teaching and research. 10 RELEVANCE FOR THE BANK’S COUNTRY PARTNERSHIP STRATEGY 2.16 The project remains highly relevant to the World Bank’s current strategy for Vietnam. The current Country Partnership Strategy, covering the period 2012-2016, proposes a more selective program that focuses on sectors where the Bank has clear strengths and has had a successful track-record. The Bank also intends to support programs that are likely to be sustained by the government and will have a significant institutional impact (World Bank, 2011). 2.17 Higher education is included in Competitiveness Pillar. The Country Partnership Strategy places higher education in Outcome 1.3 of the Competitiveness Pillar, supporting increased capacity for innovation and value added. In particular, the Bank’s strategy aims to respond to “weak innovation capacity and the related low skills base of the Vietnamese labor force (page 17).” It further states that “[a] central element of Vietnam’s innovation agenda is strengthening labor market skills by reforming how the education and training system is managed (page 22).” The Strategy proposes an additional project, The Third Higher Education Project, as well as analytical work in the areas of skills for the workforce. It also proposes projects in science and technology and in innovation, which are both directly related to the project’s development objectives. The Country Partnership Strategy indicates that the Bank will move away from education infrastructure investment, and towards an approach that treats education as a system. Relevance of Design The relevance of design is defined as the extent to which the project’s design (its planned activities or policy areas) are consistent with the stated objectives, including assessment of the Results Framework. The Results Framework represents the underlying project logic linking the inputs and outputs to the outcomes. 2.18 The relevance of design is rated substantial. 2.19 The project’s design includes the necessary elements to meet its objectives and project’s causal chain is sound. The project’s technical design was sound and builds on previous Bank projects in a variety of countries. TECHNICAL DESIGN 2.20 The project’s technical design is sound, incorporating building on previous Bank work in Vietnam and incorporating elements from other higher education projects. The approach used for HEP2 built on the World Bank’s experience in the higher education sector. The use of competitive grants for universities to promote improving teacher and researching was a common element in many higher education operations, including the previous investment loan in Vietnam as well as in countries as diverse as Afghanistan, Chile, and Uganda among others.6 As is often the case for higher education projects, participation in the 6 These projects are examples of a model that the World Bank often finances for higher education including: (1) Afghanistan Strengthening Higher Education (P089040), May 2005 to June 2013; (2) 11 project was pre-determined during appraisal. This was one of the lessons from the previous project, where resources were too widely disbursed. 2.21 Grants were seen as essential to promoting high quality research. This approach can be seen as “semi-competitive.” While the universities were pre-selected, they had to compete with each other for the amount of the grant. This contributes to the university autonomy, which is often seen as playing an important role in promoting research. While each university would set its own priorities, the project included both technical and fiduciary safeguards. The proposals were reviewed both internally (by the PMU and the Bank) as well as an external panel of international experts. 2.22 The project included elements to improve the overall effectiveness of universities. The project supported the reform by strengthening accreditation and quality assurance. These are important in supporting autonomy for project universities. The accreditation and quality assurance system provide important feedback to universities administrators. It also supports the reform effort itself by providing assurance to central authorities that the university is “ready” to be autonomous. This is likely to have a positive impact on both the quality of research and teaching. 2.23 The project also financed efforts to improve teaching. The TRIG grants were also aimed at improving teaching and universities could use them to finance further training and education for their faculty members. This was aimed to provide faculty members with a combination of improved technical knowledge and pedagogical skills. In addition, the research component of the TRIGs was expected to give students more hand-on experience. 2.24 The targeted support to strengthen research programs and university staff will lead to better prepared students. TRIGs provided targeted support to certain programs with the intention that both faculty and students in these programs would benefit. The selection committee helped ensure that the chosen programs were relevant to the country and consistent with the university’s “comparative advantage.” The project had no illusions about improving employability for all students; it only targeted those students in participating programs. It included both a baseline and follow-up tracking survey to identify the progress of graduates. 2.25 The project also identified five disadvantaged universities to ensure equity in the project. The project included five universities in disadvantaged regions to participate in the project. These universities would not have ordinarily been included in the project. These universities were not expected to compete with the other universities. This has a positive equity impact, which could have a positive impact on the region’s development. However, the project did not have a formal equity objective and targeting disadvantaged universities reduces resources for that could have been used in key universities. Chile Higher Education Improvement Project (P055481), November 1999 to June 2005; and (3) Uganda Millennium Science Initiative (P086513), May 2006 to June 20133. 12 RESULTS CHAIN 2.26 The Bank’s result framework was relatively weak for measuring intermediate outcomes, which play an important role in understanding the project’s contribution and areas where course correction is needed. However, the Bank’s results framework was substantially buttressed by the government’s more extensive and complementary results framework. These indicators were included in its own project document and were monitored throughout the project. This included indicators to measure both teaching and especially research quality. In addition, the government had an extensive monitoring system at the university level to track inputs and to help the project to understand progress. CHOICE OF INSTRUMENTS AND FINANCING 2.27 The Project’s choice of an investment loan was appropriate. The project requires concrete investment in equipment and training to be effective. In Vietnam, investment loans represent additional resources for the budget. Given the complicated budgetary system, an investment loan is the most effective way to ensure that resources arrive to the university. While the project supported the reform process, this was a secondary objective. The Bank provided parallel support for the higher education reform process through a separate development program operation. This division of labor was appropriate and allowed the Bank to have the “best of both worlds” in its support for higher education. 2.28 Financing was generally adequate. The project predetermined which universities were going to participate and the average amount per university was around US$2.7 million. Based on the experience from HEP 1, this seemed like a reasonable amount and in line with the university’s capacity to handle the resources and the associated fiduciary responsibilities. 3. Implementation 3.1 Overall, the project had a relatively smooth implementation. The project closed on time and disbursed over 90 percent of the committed amount. Many of the activities were cheaper than originally projected and the project moved around resources among different activities that were effective. From Initiation to the Mid-Term Review 3.2 The project had a long preparation period and the approval date was delayed several times. The project was approved on June 20, 2007, 45 months after preparation formally began. The project was declared effectiveness on March 26, 2008, which represented a delay of nearly five month from the original date of November 1, 2007. By that time, the Ministry had complied with the legal covenants related to project administration. This included establishing a project management unit and a steering committee, as well as preparing a detailed project operational manual strengthening financial management and procurement capacity. Several factors appear to have contributed to the delay, including a delay in the signing of the legal agreement on November 22, 2007 and delays in establishing 13 the financial management system. The project management unit was established before November, 2007. 3.3 The delay in effectiveness did not have much impact on project implementation. By June 30, 2008, the project had disbursed US$6.4 million against a target of US$4.6 million. Significant work was carried out before the project became effective, which contributed to the rapid start of the project. For example, by May 2008, 14 Universities had signed TRIG agreements and established TRIG administrative units, four had signed agreements, and the remaining universities were expected to sign agreements by June. Likewise, the government was beginning the procurement process at the central level to support component 1, such as preparing short lists of potential suppliers. 3.4 At the same time that HEP2 was initiating, preparation began for a Development Policy Operation (DPO) for higher education, known as the Higher Education Development Policy Program (HEDP). In large part, the DPO was developed in response to the government’s and Bank’s desire to strengthen support for the reform process (See Box 3.1). 3.5 The mid-term review confirmed that the project remained relevant for the Vietnam and that there was no need for significant adjustment in its design. The mid- term review, which was originally planned for December 2009, took place in August 2010. The mid-term’s consensus was that component 2.2 (TRIGs) was generally being implemented well. The review rated this component as moderately satisfactory, with 13 universities rated moderately satisfactory or higher and 9 rated moderately unsatisfactory or less. The project was making good progress on financing training, consulting services, and equipment and that it was likely to close as planned. The mid-term review proposed adjusting the procurement plan and following up on slower elements. 3.6 The project also saw advances in the implementation of several components. For component 1.1 (governance), the project supported the Higher Education Master Plan as well as many of the planned studies either finalized or in progress. For component 1.3 (quality assurance), which focused on accreditation, the project provided technical assistance to MOET, supporting the design and implementation of new regulations. Components 1.2 (higher education financing) supported a major study on higher education financing. 14 Box 3.1. The Higher Education Development Policy Program The Higher Education Development Policy Program was designed to complement HEP2 and to support the government’s reform agenda, HERA. The DPO series had three operations that provided financing as the government approved reforms (“prior actions”) to forward HERA. The operations were approved in June 2009, August 2010, and February 2013. The final operation of the DPO had six prior actions divided into four areas: (1) Approval of the Higher Education Law; (2) Improving governance, with circulars on university autonomy and a decree on foreign financed education institutions; (3) Enhancing financing and financial management, including annual financial reports for all higher education institutions and audits on a number of universities; and (4) Improving quality, through a circular to introduce academic credits and reforming the grading systems as well as establishing a number of university-level quality assurance levels. The previous two DPOs included prior actions supporting areas (2) to (4). A review of the prior actions shows a clear complementary between the DPOs and the Higher Education Project II. The project provided some support to implement the reform. Source: (World Bank, 2013). After the Mid-Term Review 3.7 There were no major changes after the mid-term review and the project closed as planned. The project was carried out as planned and the project closed as originally planned in June 2012. Initially the government requested additional financing to extend the project. Eventually the government and the Bank agreed to prepare a new project to build on the project’s gains. 3.8 Component 2, which provided TRIGs to universities, was generally implemented smoothly. Although the procurement under this component was generally technically complicated but by 2011, most of the procurement under this component was finalized. 3.9 The project’s design included an independent evaluation of the implementation of TRIGs. While a firm was hired to carry out this evaluation, the firm was not able to meet the terms of the contract and the contract was cancelled. The project then turned to a panel of experts to carry out an evaluation. Component 2.1, which focused on strengthening the management capacity of universities, was implemented slowly but eventually led to the development of instrument to collect feedback and information from a variety of sources, including graduates, employers, and students. In addition, the project supported a follow-up tracer survey that was completed in March 2012. This was originally planned under component 2 but it appears to have been financed under component 1.2. 3.10 The implementation of component 1 was delayed and required additional attention. While there were few problems with component 2, the project’s technical 15 assistance was both delayed and required some changes in its implementation. The publically disclosed sections of the Bank’s Implementation and Results Status Report (ISR) from 2010 to 2012 generally cited “limited progress” or delays in the different aspects of this component. By the end of the project, most of the activities in this component were finished albeit later than expected. One important exception was the development of a higher education management and policy information system (an information system to track development in the higher education system), which was cancelled due to the lack of consensus within the Ministry on the design.7 The project’s Implementation Completion and Result’s Report (ICR) provided several examples of component 1’s contributions, including inputs for the 2012 Higher Education Law, new policy for tuition policy, and improvement in the quality assurance system (World Bank, 2012). 3.11 The project ended on time and was largely disbursed (more than 90 percent). Several research activities continued after the project closed, using national resources. Likewise, many graduate students remained overseas to finalize their studies with alternative resources. Safeguard Compliance 3.12 No safeguards were triggered during appraisal. The Operational Manual included provisions for environment standards, if required by the TRIG proposals. The ICR did not report on safeguard issues. As per the TRIG regulations, the project supported refurbishment of existing facilities and financed little of no new construction. Fiduciary Issues 3.13 Financial reporting, including data from each participating university, were consolidated by the PMU. The universities had to receive clearance from the provincial treasury prior to submission. During implementation, the project faced several fiduciary issues. The project submitted all of the required financial reports, such as statements, auditor reports, and interim financial statements. However, many of the universities’ financial reports were filed late or of poor quality. The project audit reports were submitted in a timely fashion. The government refunded IDA for a variety of reasons. The government refunded the Bank to repay an advance for procured goods that were ultimately not delivered. Likewise, the government refunded IDA for some illegible expenses such as subsidies to student after the project closed. Several universities had to return unspent advances after the end of project. In total, around $1.8 million were refunded. The audit reports reflected these refunds, which were all certified by the independent auditors (“clean report”). 3.14 Although the PMU developed an internal audit system, this was not implemented because the universities preferred to submit receipts rather than develop their own financial management units. Internal ratings for the project’s financial management were rated moderately satisfactory until 2011 when the rating was lowered to moderately unsatisfactory. The rating was lowered due to problems in introducing internal audit as well as delays with some of the financial management reports. 7 The Ministry eventually established its own, more limited system with its own resources. 16 3.15 Procurement was complicated with around 750 procurement contracts, including over 100 international contracts. For the procurement of goods, the Bank would review the first contract in each university. After the first contract, prior review was only required on internationally competitive bidding contracts worth more than $300,000. A similar set of rules was introduced for consultants, with lower limits. The PMU and the Bank carried out the traditional fiduciary review of the equipment that the university purchased with the TRIGs. The procurement was largely carried out by participating universities, which proved to be a challenge for some of them. The PMU provided extensive training to the participating universities on procurement. While there were a number of delays in procurements as well as canceled biding processes, there were no major delays or issues with procurement and most of the procurement was carried out as planned before the end of the project (World Bank, 2012). In October 2010, the Bank reported that “some of these procurements [for major equipment] experience delays due to potential fraud (sic) practices of bidders.” The government carried out a number of investigations on its own that led to barring two firms. 4. Achievement of the Objectives 4.1 The evaluation divides the objectives into sub-objectives, each with one final objective statement. As presented in Section 2, these objectives are to increase the quality of teaching in order to improve the employability of graduates and to increase the quality of research in order to improve the relevance of research. In university setting, these elements are closely related. Table 4.1 presents indicators that are relevant to both objectives or provide background information, such as support for policy reforms and quality assurance as well as training and education for university staff. 4.2 The project provided support for the development of the Higher Education Master Plan and education finance policy. In addition, the project supported efforts to introduce accreditation in most of the project universities. The project also supported the training and education of a number of university staff. This will also lead to an increase in lifetime income in this academic staff. 17 Table 4.1. Intermediate Indicators Associated with Both Objectives Baseline Target Indicator Final Value Remarks Value Value Analytical work on The project supported studies Work Work higher education that were incorporated into None incorporated incorporated governance incorporated the Higher Education Master into policy into policy into policy* Plan The project supported a Analytical work Work Work number of policies on unitary developed for higher None incorporated incorporated costs and public financing education finance policy* into policy into policy mechanisms. All universities carried out Proportion of eligible self-evaluation during the 3/22 22/22 universities evaluated in 20/22 (91%) project period. Two have not (14%) (100%) an accreditation system* completed the external evaluation. Combines trainees in foreign (1,368) and (21,407) Staff participating in domestic courses. The Bank short-term training 0 10,000 22,775 informally added targets. The courses* government’s target was 21,432. Staff in foreign Ph.D. At least some of the 0 57 56 training** education is financed by the Staff in foreign master’s project, typically at least the 0 147 143 training** first year. *-From the World Bank’s results framework **-From government’s results framework Source: Project data. 4.3 While each university developed its own TRIG project based on its priorities, they all support both research and training. Project rules restrict infrastructure investments to those directly related to research and teaching (World Bank, 2007). Objective 1: Increase the Quality of Teaching in Order to Improve the Employability of Graduates 4.4 This objective is rated substantial. 4.5 The project contributed to the teaching capacity of participating universities leading to greater relevance and improved opportunities for students. In total, the project reached around 50,000 students that studied in the programs covered by the project; total university enrollment was higher. Tables 4.2 and 4.3 present relevant data associated with changes in teaching and learning as well as data from tracer studies. These indicators were not formally followed by the World Bank in its internal reports. Because group 1 universities are among the “best” universities in the country and group 2 universities are located in poor provinces, the results are presented separately. Much of the data and targets are based on the government’s results framework which the universities collected and the PMU consolidated 18 and monitored. It is important to point out that the target values in the table are for the entire university system and not just the TRIG universities. Since the Group 1 universities are among the best, many of them have already exceeded the national targets. Table 4.2. Indicators Associated with Objective 1, Group 1 TRIG Universities Baseline Target Final Indicator Remarks Value Value Value Outputs affecting quality of teaching Proportion of university teaching staff Intermediate target value 44.4 40.0 50.0 with Master Degree from 2010. Targets are for Proportion of university teaching staff all universities in the 22.7 25.0 24.6 with Doctoral Degree country New curriculum developed n.a 18 38 From government project Curriculum revised n.a 220 224 document. No textbooks were Textbooks developed n.a. 462 315 developed by Group 2 Universities Student-teacher ratio 22.6 20.0 18.2 Target value from 2020. Proportion of teachers trained in 44.4 n.a. 75.9 No target indicated teaching Outcomes associated with learning Average score in standardized tests 4.4 n.a. 6.3 Students with job after six months 68.4 n.a. 74.2 Average years required to graduate 4.6 n.a. 3.6 No target indicated Percentage of recent graduates 1.7 n.a. 2.4 pursuing studies in foreign institution Note: Unless otherwise indicated, baseline is from 2007 and target is from 2012. The targets are based on national targets from HERA. Source: Project data. 19 Table 4.3. Indicators Associated with Objective 1, Group 2 TRIG Universities Baseline Target Final Indicator Remarks Value Value Value Outputs affecting quality of teaching Proportion of university teaching staff Intermediate target value 34.5 40.0 38.8 with Master’s from 2010. Targets are for Proportion of university teaching staff all universities in the 3.4 25.0 3.2 with Doctoral Degree country New curriculum developed n.a 0 0 From government project Curriculum revised n.a 3 3 document. Student-teacher ratio 19.6 20.0 16.6 Target value from 2020 Proportion of teachers trained in n.a. No target indicated teaching Outcomes associated with learning Average score in standardized tests 5.0 n.a. 5.8 Students with job after six months 69.0 n.a. 51.5 Average years required to graduate 3.6 n.a. 3.2 No targets indicated Percentage of recent graduates 0.1 n.a. 0.4 pursuing studies in foreign institution Note: Unless otherwise indicated, baseline is from 2007 and target is from 2012. The targets are based on national targets from HERA. Source: Project data. OUTPUTS 4.6 The project supported a number of programs to improve the quality of teaching. The two main areas of support were through training—both long term post-graduate training and short-term course on teaching methods. The evidence presented in Tables 4.2 and 4.3 show an increase in the teaching capacity of participating universities. Both group 1 and group 2 universities have seen an increases in the percentage of instructors with master’s degrees. Likewise, the proportion of instructors with doctoral degrees has increased in group 1 universities. The project supported the training of 149 staff at the master’s level (111 finished by the end of the project and returned) as well as 56 staff at the doctoral level (10 returned by the end of the project). Despite the fact that overall enrolment has increased at the tertiary level, the student-teacher ratio decreased in participating universities. 4.7 The project also supported both the development and revision of new curricula in participating university, exceeding the project’s targets. Updating curricula is generally considered important to ensure that teaching remains relevant. In addition, the project supported the investment in 111 laboratories, 29 multi-media teaching facilities, and five libraries with new 38,755 titles. 4.8 Both the development of an internal quality system and an independent external accreditation system are important elements of the project and of the government’s HERA. These quality assurance systems require the university to have minimum standards for both teaching and research as well as having relevant curricula (World Bank, 2013). By the end of the project, 20 out of 22 universities were evaluated in the accreditation system, which also 20 requires the establishment of an internal quality system. In essence, accreditation validates the investments made in training and the development of new curriculum. OUTCOMES 4.9 The government measured gains in learning. During the project period, average score increased on standardized tests in both group 1 and group 2 universities. Likewise, the average time of completing undergraduate studies decreased in both groups of universities. While this definitely reflects an increase internal efficiency in participating universities, it also likely to reflect an increase in quality of teaching—better instruction is likely to lead to students graduating in a more timely fashion. 4.10 Although not strictly a measure of employability, the project’s tracer survey tracked both the number of students that pursued graduate degrees overseas and the length of time that students had to find jobs. Since the tracer survey was carried out with project financing, it can only collect data on the six months after graduation. In Group 1 Universities, the number of students going overseas within six months after graduation increased from 1.7 percent to 2.4 percent. During the PPAR mission, universities confirmed that a growing number of undergraduate students pursued graduate studies both in Vietnam and overseas. While there is no published data on where the students were hired, all universities reported that majority of their students were hired in their field. Many students find employment in their university’s laboratory prior to starting graduate education. Box 4.1. The View of Students As part of the evaluation, the PPAR team met with around 40 undergraduate students from a variety of fields. These discussions, which were carried both within their universities and outside, focused on how students benefited from the projects, their views on relevance of higher education, and challenges that they face. The students came from a variety of backgrounds. Most came from Hanoi or its environs, although a significant minority came from elsewhere. Very few had parents with university degrees. Tuition was generally low, although it was a significant cost for the poorest students. For students who came from outside of Hanoi, living costs were a concern. Dormitories are relatively cheap although space is limited. Scholarships were generally not adequate for the poorer students. Most students reported that they had some access to the laboratories supported by the project, particularly students in the final years of studies. This sometimes involved their own research projects or work done under professors. Students typically identified one or two professors involved in research as role models and most felt that the academic staff was quite open to answering questions and discussing issues related to their field of study. Employment was a concern of virtually all students. All of the students wanted to remain in their field of study and this was often seen as a challenge. While there was always potential employment as laboratory assistants, this work was often on a voluntary basis or with low pay. All students felt that a graduate degree was a necessity, preferably from an overseas university. Foreign language skills and financing were major concerns. In addition, several students discussed the lack of institutional support for job and career guidance. Nearly all students felt that their professors were supportive in this area and were optimistic about their eventual employment. Source: PPAR Mission. 21 4.11 Likewise, the percentage of students getting a job within six months increased from 68.4 to 72.4 percent. This happened in a time of economic slowdown. On the other hand, students in Group 2 Universities saw an increase in time to get a job after graduation. The poorer provinces were more affected by the economic slowdown and it is not surprising that students from Group 2 Universities had a more difficult time finding jobs. Objective 2: Increase the Quality of Research in Order to Improve the Relevance of Research 4.12 The achievement of this objective is rated high. Table 4.4. Indicators Associated with Objective 2 Baseline Target Final Indicator Remarks Value Value Value Indicators at the PDO Level Total articles published in n.a 709 1,162 Articles directly supported by national journals TRIG. Information was also Total articles published in n.a 305 645 collected by subject area. international journals Annual articles published 207 n.a. 508 Total number of articles in international journals produced in all programs in Articles per faculty .062 n.a .109 beneficiary universities member (Group 1) (including programs not Articles per faculty .004 n.a. .015 supported by TRIG). member (Group 2) Intermediate level and other indicators Percentage of revenues University revenues derived 5.2 n.a. 11.8 from SandT (Group 1) from science and technology Percentage of revenues issues. Not necessarily 1.7 n.a. 8.8 from SandT (Group 2) associated with TRIGs Analytical work on Policies Policies accreditation, quality None Policies were adopted. adopted adopted assurance, and admission Research projects n.a. 252 249 New laboratories n.a. 66 65 Includes ongoing projects Renovated laboratories n.a. 20 20 Source: Project data. OUTPUTS 4.13 The project provided significant support to research activities through the TRIGs. The grant provided financing for both equipment and training. However the TRIG program is also an output. Establishing a grant program for teaching and research is a major advance for the higher education system as it moves away from a system that is almost entirely financed by the government budget. Although the universities were pre-selected, 22 they had to compete with each other for financing. This is an important for establishing university autonomy and is considered good practice in higher education (Salmi, 2009). 4.14 Staff and students interviewed by this evaluation indicated that the grants were targeted to well-defined research areas. Interviews for this evaluation and for the ICR also showed that the research projects were focused in relevant areas, often addressing local and national needs. In the case of the better established research universities, TRIGs were focused on one or two targeted programs (for example, geology or materials science). In other universities, resources supported a larger number of laboratories with basic equipment. Table 4.5. Self-assessment of the Impact of TRIGs on Research Impact Score Enhanced substantive knowledge of research topics 3.5 Enhanced ability to identify and select research 3.5 Enhanced ability to write research proposals 3.4 Enhanced ability to collect data 3.6 Enhanced ability to analyze and interpret data 3.5 Enhanced ability to write research papers 3.5 Material to support research 3.8 Note: Scores range from 1 to 5. Source: Project data. 4.15 The project supported a total of 249 research projects, of 171 were finished by the end of the projects and another 78 were ongoing when the project closed. University faculty were surveyed to rate the quality and impact of the project’s support on research. These results are summarized in table 4.5. The total amount of TRIG amount was around US$50 million. Academic users generally give the program a rating of 3.5 (out of five); in the context of Vietnamese grading system, these are reasonable ratings. OUTCOMES 4.16 The project had a major impact on improving the quality and relevance of research in higher education in Vietnam. The project provided significant support to development and dissemination of research. According to project data, TRIGs directly supported 1,162 national and 645 international peer-reviewed articles, well exceeding the established target.8 This led to two patents directly supported by TRIG. During the same period, there was a significant increase in the total number of national and international articles, approximately doubling the annual production. While this cannot be wholly attributed to TRIG, the equipment, supplies, and research supported by the project definitely played a major role. 4.17 The project did not try to pick “winners” ex ante by restricting research to certain topics. The TRIG review panel carefully considered the relevance of each proposal as well as the “comparative advantage” of each university in its proposed areas. The fact that research generated a large number of articles published in peer reviewed international journals 8 The ICR reports 2,037 national publications and 876 international publications. This includes both peer-reviewed and non-reviewed articles. 23 suggests that it was relevant for the research community (93 percent were in the field of natural sciences). Both the ICR and the PPAR’s interviews showed that TRIG supported several partnership agreements with both private (for example, agriculture and mining) and public sector organizations. This indicates a willingness to pay for research supported by TRIG. Science and technology has contributed to a growing share of revenues in participating universities, growing from 5.2 percent to 11.8 percent. Interviews with universities suggest that these were largely in programs supported by TRIG. 4.18 The project’s direct contribution to science and technology in participating universities significantly exceeded expectations. The project made a significant contribution to the overall university environment beyond its support for specific research programs. 5. Efficiency 5.1 The project’s efficiency is rated as substantial. 5.2 Measuring the project’s efficiency includes the external efficiency, which focuses on the project’s contribution to the economy. Although this is difficult to measure, it appears that the project had a positive economic impact. Internal efficiency measures how well the project managed its own resources. The project was careful with its resources and spent them in a timely fashion. External efficiency 5.3 It is challenging to calculate the external efficiency for the project. A project focusing on teaching and research is likely to have both public benefits (largely accruing to the economy) and private benefits (largely accruing to individuals). This evaluation will consider both aspects to determine the project’s efficiency. The PAD has a good explanation of the various individual and economy-wide economic benefits that might be attributed to the project (World Bank, 2012). Table 5.1 provides a summary of the results from the PAD. 24 Table 5.1. Summary of Assumptions in PAD Internal economic Scenario Assumptions rate of return Base case  Benefits to graduates: 12.5 percent increase in 34% wages for Group 1 graduate and 4.2 percent for Group 2, over 30 year time horizon  Benefits to RandD: 75 percent rate of return for research at Group 1 Universities and 35 percent rate of return for Group 2 Universities over a 5 year time horizon Scenario 1  Benefits to graduates remain the same 28%  Benefits to RandD are halved; 38 percent in Group 1 Universities and 18 percent in Group 2 Universities Scenario 2  Benefits to graduates are reduced; 8.3 percent 29% increase for Group 1 Universities and 2.1 percent increase for Group 2 Universities  Benefits to RandD remain the same Scenario 3  Benefits to graduates halved, as in Scenario 2 24%  Benefit to RandD halved, as in Scenario 1 Source: (World Bank, 2007). 5.4 The project’s first objective largely focuses on improving options for graduates and ultimately raising their lifetime earnings. In principle, this can be measured by a number of different techniques that capture the private rates of return to education, as developed by Mincer and other researchers (Mincer, 1981).9 The rate of return for students is calculated for graduates over a thirty year horizon; given the age at graduation, probably a forty year horizon would have been more realistic. 5.5 The PAD estimated that graduates from Group 1 Universities would be able to earn 12.5 percent more as a result of the project, while students from Group 2 Universities would earn about 4.2 percent more. The project was implemented as designed and had the expected coverage for the expected costs. The assumptions about the impact of the project is also reasonable. More recent evidence suggests that the wages for university educated population is increasing faster than was expected, leading to greater wage differential for students benefiting from the program (World Bank, 2012; Giesecke, Tran, Meagher, & Pang, 2011). This suggests that the rate of return for students is higher than was expected. Likewise the estimates do not include the reduced time needed to find employment, which will also increase the rate of return for students. 5.6 However the project’s greatest benefit for employment is likely to accrue to students who pursue a graduate degree as a result of the project or who go to a better graduate 9 In addition, creating high quality employment creates important social rates of returns that are typically not captured in analyses in the external efficiency of education (World Development Report, 2014). 25 program. This was not explicitly taken into account in the economic analysis. The project only collected data on students who went overseas for graduate studies within six months, a small percentage of the total number of students that eventually do graduate studies. Thus the PAD’s estimate of the rate of return for students understates the project’s benefits. 5.7 Likewise, the PAD estimated the returns to investment in research. Although research often creates private gains, such as the patenting of discoveries, this is typically smaller than the value of the public gains from research. There is no standard methodology for calculating these benefits for the economy and the PAD makes a number of assumptions based on international literature. These assumptions are conservative (for example, the benefits for research only last for five years) and probably understate the project’s impact. It appears that the research supported by the project was more effective than expected. Thus the assumptions on the rates of returns for RandD provide a good basis for the rate of return. 5.8 Based on this, the evaluation estimates that the project’s economic rate of return is at the 34 percent that was estimated at the time of PAD. This builds on the estimate in the PAD, taking into account that the project has met or exceed its targets and therefore the assumptions. The high number of international articles attests to the quality of research. Thus the time horizon for the returns to RandD is likely to extend beyond five years that was initially assumed. Spending on research was similar to what was originally assumed and the original estimates of the rate of return for investment of RandD remains reasonable. It also builds on the new information about the structure of the labor for university graduates, which show a greater impact of higher education on income than originally assumed. In short, the project’s economic rate of return well exceeds the opportunity cost of capital. Internal Efficiency 5.9 Most of the proposals for TRIGs were prepared in a timely fashion and the PMU was able to disburse resources to the universities as expected. Although the procurement was quite complicated, involving international competitive bidding for sophisticated laboratory equipment did not affect the disbursement or the schedule of implementation. The project audits did not show any serious issues with financial management. Internal Bank reports suggest that the government was proactive in addressing allegations of important procurement, going as far as “blacklisting” two firms. Nearly all of the funds were disbursed and it closed as planned. 5.10 While it is not possible to compare the administrative cost of this project compared to that other similar projects, comments from interviews as well as comments in internal Bank reviews suggest that the PMU was quite efficient and provided good value for money. 6. Ratings Outcome 6.1 Overall, the project is rated satisfactory. 26 Table 6.1. Summary Outcome Ratings Outcome Ratings Remarks Relevance Relevance of Objective High The project was well-aligned with the government’s higher reform program and Relevance of Design Substantial the World Bank’s country strategy. Efficacy Improving Employability Substantial The project had the expected impact on Improving the Relevance of High both teaching and research. Research The estimates in the PAD provide a Efficiency Substantial reasonable basis to estimate the project’s rate of return. Project Outcome Satisfactory 6.2 The project’s high relevance of objectives, reflecting its role in the World Bank’s strategy as well as its alignment with the government’s development and higher education strategies. The relevance design is related substantial, as it addressed most elements of the PDO. The project had a substantial impact on the quality of teaching to improve the employability of graduates. The project had a high impact on promoting research in Vietnam that exceed expectations. The project was carried out with a substantial level of efficiency, leading to a reasonable rate of return. Risk to Development Outcome 6.3 The risk to development outcomes is rated modest. 6.4 The project was developed with strong support from the government, as a follow-on project and well aligned with the government’s strategy, HERA. The project was also supported by a programmatic development policy operation (see Box 1.2) that focused on supporting the policy environment following the elements laid out in HERA. 6.5 The government has continued its commitment to the project’s objectives. HERA is a 15 year program and remains in effect until 2020. This is reinforced by the approval of the Higher Education Law in June 2012.10 The Law confirms the principal aspects of the project’s objectives including those related to teaching and students (Article 5) as well as those related to research (Articles 39 and 40). In particular, the Law reaffirms the role of quality assurance in higher education (Articles 49 to 53) as well as promoting university autonomy (Articles 28, 29, 32, and 65). While the law has been criticized for “not going far enough,” it does reaffirm two central elements behind the project’s causal chain (Pham, 2012; Viet Nam News, 2014). 6.6 Increasing financial autonomy also provides universities with resources to maintain their investments. Interviews with universities show that additional resources are 10 The Law on Higher Education (Law Number 08/2012/QH13), approved by the National Assembly on June 18, 2012 and effective on January 1, 2013. 27 needed for current expenditures. While all of the visited universities confirmed that they had the necessary funds, they indicated that their budgets were tight. Growing revenues from tuition as well as more flexibility in the university budget should help alleviate this pressure. Likewise, the project supported efforts to increase university’s capacity to raise money from research projects and joint ventures. 6.7 There are sufficient resources to continue financing the activities supported by the project. HERA includes ambitious targets for investment in higher education. In addition to financial support from the World Bank and the Asian Development Bank, the government has been increasing its financing of education.11 The economy continues to grow strongly (World Bank, 2013), which both contributes to the government’s capacity to finance higher education as well as maintaining the demand for higher education (Huy, 2012). Table 6.2 shows the growing spending on education, which has continued after the project closed. 6.8 It is likely that the government will continue with grant funding for higher education. Interviews with government officials indicate that the Ministry intends to maintain some form of the TRIG program. Currently the government and the Bank are discussing a follow-on project that is likely to include TRIG or its successor among its components. In addition, the Ministry has allocated the revenues from the Development Policy Operation (US$150 million). Although the final disposition of these funds has not been determined, interviews suggest that these resources are likely to complement TRIGs. Table 6.2. Total Public Spending on Education, 2005-2013 Total Education, Total Education, GDP, Percent Year Billions VND Billion US$ Billions US$ GDP for (2005) (current) (current) Education 2005 35,203 2.2 57.6 3.9% 2006 44,027 3.0 66.4 4.5% 2007 51,283 3.8 77.4 4.9% 2008 52,754 4.2 99.1 4.2% 2009 52,494 4.9 106.0 4.6% 2010 63,167 5.7 115.9 5.0% 2011 72,185 6.7 135.5 4.9% 2012 72,994 8.1 155.8 5.2% 2013 76,909 9.4 171.4 5.5% Sources: World Bank Development Indicators, evaluation calculations. Note: Education spending based on budgeted amount and includes development (investment) and current budget. Actual amounts range from 93% to 111% of the budgeted amount. 11 This includes all public spending on education at the national and local levels. At the present, Vietnam does not have a consolidated estimate of public spending on higher education. 28 Bank Performance 6.9 Overall Bank performance is rated satisfactory. 6.10 Quality at entry benefited from lessons learned from previous project (Higher Education Project 1) as well as the careful technical work that was done as part of the preparation process. However, there were a number of shortcomings in the development of the project’s results framework. QUALITY AT ENTRY 6.11 Quality at entry is rated satisfactory. 6.12 The design period was quite long, allowing the Bank time for detailed technical work as well as project preparation. The Bank held an identification mission and prepared a concept note in 2003. This was followed by a number of preparation missions that culminated in appraisal in March 2007 and approval in June 2007. The project became effective in March 2008. During the four years of preparation, the Bank financed a wide range of policy documents supporting all elements of the objectives as well as number of assessments of different elements of the project. The project preparation was delayed approximately 24 to 30 months. This was caused by delays in the First Higher Education Project (HEP1),12 which closed two years late. The delays in HEP1 happened in the first years of implementation, while the Ministry and the universities built their capacity to implement the project (World Bank, 2008). 6.13 The Bank drew heavily from the lessons from the HEP1. These lessons included efforts to improve project efficiency that complemented efforts to improve the project’s equity. While HEP2 had a similar design as HEP1, with grants awarded to universities based on competitive proposals, the Bank worked to make the grant process more efficient. HEP2 reduced the number of universities included in the program from 36 to 22 universities, allowing the project to be more focused. These universities were chosen by MOET based on their quality as well as readiness to receive grants. In addition, HEP2 reduced the number of calls for grants from three to one, with provisions for MOET to approve some grants before the project became effective. The Bank also agreed to raise the limits for prior approval in procurement (World Bank, 2012). While the aim of these measures was to increase the project’s efficiency, it was also well-aligned with the government higher education strategy (HERA) - the greater selectivity allowed the project to focus on universities that are active in research and are likely to be accredited. In addition, the Bank and the government agreed to increase the project’s equity by creating the TRIG 2 program that provided grants to “equity” universities in disadvantaged regions. This responded to concerns that universities in poorer areas were increasing falling behind key universities. TRIG 2 represented an attempt to address this perceived inequality in a way that was consistent with the existing design. 6.14 The design of HEP2 was informed by strong consultation and technical work. The Bank took advantage of the long project preparation period to organize a number of 12 Higher Education Project (P004828), approved in August, 1998 and closed in June 12007. 29 technical workshops and high-level meetings. For example, in March 2006, the Bank organized a high level retreat bringing together experts in various national organizations along with Bank specialists. In addition to providing technical advice, these meetings helped the design team understand what was feasible and realistic in Vietnam (Dang, 2009). The technical work was specialized and prepared both by international and domestic consultants. The Project Operational Manual identified 22 different technical papers, prepared in two series: (1) developing research capacity and enhancing teaching and learning and (2) reforms for higher education sector policy and governance (Socialist Republic of Vietnam, 2007). It is likely that these papers contributed to the development of HERA (Dang, 2009). Likewise, the technical work contributed to the World Bank’s 2008 Higher Education and Skills for Growth report (World Bank, 2008). The Bank also supported an independent review of the TRIG process in 2007 (Khang, 2007). 6.15 The Bank prepared detailed economic and financial analyses during appraisal. The PAD included a detailed economic analysis (Annex 9), which focused on the costs and benefits of the project as well as a financial analysis (Annex 10), which focused on the project’s financial impact and the government’s capacity to finance it. Both analyses clearly lay out the underlying assumptions. The economic analysis was more detailed analysis than is typically seen in World Bank project appraisals. The financial analysis focused on the government’s capacity to support higher education and sustain the project’s investments. 6.16 The Bank was proactive in addressing fiduciary issues. During project preparation, the Bank financial management (FM) team carried out a comprehensive review of HEP2’s capacity, under the assumption that the PMU would change its FM staff. The Bank carried out visits at both the central and the university levels. The FM strategy (Annex 7 of the PAD) identified concerns about the large number of implementing agencies (22 universities) that had limited FM capacity. Annex 7 proposed training and capacity building at a number of levels as well as requiring the PMU to hire experienced FM staff. The Annex indicated that the government would establish internal controls and have an internal auditor, which would report directly to the Vice Minister. The Annex included an Action Plan that summarized FM measures and at which level they needed to be implemented. The Bank was pro-active in addressing the challenges in procurement. Procurement was to be carried out by each university and was likely to be complex, involving scientific equipment. The procurement plan (Annex 8) included reasonable procurement limits, allowing shopping procedures for less than $30,000, as well as the requirements for prior Bank approval. The Annex outlined procedures for procurement from state-owned enterprises and from universities, which were likely to be relevant for the project. The Bank carried out a procurement capacity review and also prepared an action plan for procurement. QUALITY OF SUPERVISION 6.17 The quality of supervision is rated satisfactory. 6.18 Overall, the Bank’s supervision was proactive, efficient, and supportive. The World Bank was generally quite responsive to the project’s need and worked to prevent and solve bottlenecks. This included a relatively fast approval (“no objection”) to project decisions. The Bank started this supervision before the project became effective. For 30 example, in late 2007, the Bank offered comments on 14 TRIG proposals less than one month after they were submitted for review. This was important in preventing a major bottleneck, as TRIG accounted for the largest single expenditure in the project. Likewise, despite the complicated nature of much of the procurement, the Bank was able to process no objectives in a relatively efficient fashion, contributing to the relatively stable disbursement. 6.19 During the implementation period, the Bank held at least one supervision mission per year, including a review in Hanoi, with meetings with university authorities both in Hanoi and in the field. In addition, the Bank organized visits by the financial management team to a number of universities. It supported the PMU with technical advice and provided solution to pressing problems. In addition there was constant contact and dialogue between the Bank and the government and implementers. Both Bank task team leaders were based in Hanoi and were supported by local education, operations, and fiduciary staff. The interviewed universities indicated that the Bank was provided support during all phases of the project. This was particularly important given the complexity of the TRIG procurement packages. 6.20 The Mid-Term Review was used as an opportunity to make course corrections. This included adjusting the size of a number of different sub-categories as well as a reallocation of resources among the universities, based on their performance. Borrower Performance 6.21 The overall borrower’s performance is rated as satisfactory. GOVERNMENT PERFORMANCE 6.22 The government performance is rated as moderately satisfactory. 6.23 Central government ministries, the Ministry of Finance and the Ministry of Planning and Finance, fully supported the project. Government counterpart financing was provided in a timely fashion, even during the global economic crisis in 2008. The government provided the MOET with the necessary authority to implement the project. And perhaps more importantly, the government ensured that the universities had sufficient autonomy to implement the project. New reform efforts (such as the Higher Education Law) were consistent with the project and incorporated many of its lessons. 6.24 The Ministry of Education and Training was well engaged with the project. It established the PMU and ensured that it was staffed as agreed. This was a major part of the Financial Management Action Plan. The Ministry also supported efforts to promote university autonomy. Several units within MOET—notably the Department of Planning and Financing and the Department of Education, Testing, and Accreditation—contributed to the implementation of Component 1 (World Bank, 2012). Internal records suggest that the government provided necessary refunds in a timely fashion. 6.25 There were several minor shortcomings in the government performance. The first was the cancellation of Higher Education Management and Policy Information System was due to the lack of coordination across a number of Ministries and universities, including 31 those that did not participate in the project. There was no consensus on the system’s purpose, which further complicated efforts to establish the system. IMPLEMENTATION AGENCY PERFORMANCE 6.26 The implementation agency performance is rated as moderately satisfactory. 6.27 The Project Management Unit provided sound management to a complex project. According to the ICR and interviews in the field, the PMU had a strong sense of ownership of the project and a clear understanding of how the project fit into the government’s reform program for higher education (World Bank, 2012). All of the interviewed universities indicated that PMU provided timely technical and fiduciary advice that was crucial in implementing the project, in particular for the complicated procurement processes. This involved both training as well as working with the individual universities to improve their fiduciary capacity. The PMU also preformed its FM role as expected and consolidated reports were submitted in a timely fashion. 6.28 The PMU was able to implement an ambitious monitoring and evaluation system that went beyond what was indicated in the PAD. This included collecting data (both project- level data measuring outputs and impacts as well as administrative indicators) from universities and processing them at the central level. The PMU monitored the status of each university and provided “grades” as well as feedback to improve performance. The PMU used this data to reallocate resources among universities to ensure that the best performing institutions receive more support. Decisions about grants were made based on a technical review of grants by an Assessment Panel made up of national experts and (for Group 1 Universities) international experts as well. The TRIG assessment report included comments for each university. According to project records, the Panel met and provided comments in a timely fashion. 6.29 As a whole, the universities carried out their functions satisfactorily, however there were a number of shortcomings. Some universities performed better than others; according to the PMU’s ranking 18 out of 22 universities were rated “average” (equivalent to a grade of A, B, or C) or greater. This rating reflects the university’s physical and financial implementation of the project and not on the project’s impact. The remaining four universities had shortcomings, such as mistakes in procurement and limited implementation of their TRIG projects. Universities with a rating of C had minor shortcomings, leading to delays but not requiring major adjustments to their investment plan. The participating universities all prepared the necessary proposals for TRIG, largely before the project become effective. Each university established a management unit, with qualified staff. The universities complied with the reporting requirements, generally in a timely fashion. However Bank reports show that a few universities were often late with their financial reports and delivered reports of low quality. Monitoring and Evaluation 6.30 Monitoring and evaluation is rated substantial. 32 MONITORING AND EVALUATION DESIGN 6.31 The project had a strong monitoring and evaluation framework. The government had a detailed results framework both for higher order objectives, which were outlined in the Higher Education Reform Agenda, and project level indicators, which were included in the Project Implementation Manual. These indicators also had baselines, when appropriate, as well as yearly targets. In contrast, the World Bank developed a weak results framework in its PAD. It did not have indicators for most aspects of the objectives or the intermediate objectives. The Bank’s results framework had two objective-level indicators that focused on the second objective (“improving the relevance of research”). The Bank did not have an objective-level indicator for the first objective (“improving employability”). 6.32 The results framework included a realistic strategy to collect data, which was largely based on collecting data from universities. The project included support for universities to strengthen their data collection and analysis capacity. Data were also collected centrally, such as tracer surveys that was planned at the end of the project. 6.33 The PAD included baseline values for the two objective-level indicators; the intermediate outcome indicators started at zero. The Bank set yearly targets for each indicator. The Bank’s parallel development policy operations did have more detailed indicators focusing on policy objectives; these operations started after the HEP II was approved. MONITORING AND EVALUATION IMPLEMENTATION 6.34 The project was able to monitor all of the required data as expected. Universities provided a variety of information on their inputs (training, procurement, etc.) as well as outputs and outcomes (articles published, staff with graduate degrees, etc.) in a timely fashion. The PMU collected the data from the universities and consolidated it in periodic reports that were available to the World Bank. Both for the mid-term review and project closing, the PMU produced a detailed report on the project build around the monitoring framework. In addition to monitoring trends, the PMU also calculated aggregate scores for each university and gave each university a letter grade (from A to D) based on performance. This allowed the project to compare the progress of each university and to make adjustments in the allocation of resources. 6.35 The World Bank monitored the project on a regular basis, using data provided by the PMU; this included data that the government collected for its M&E system. The Bank’s internal reports noted that the project’s monitoring system was strong and effective. The universities visited by the evaluation indicated that the project provided effective support to strengthen their monitoring and evaluating capacity. MONITORING AND EVALUATION UTILIZATION 6.36 Data played an important role in decision making in the project. The project’s design included a mechanism to allocate TRIGs based on performance, which the PMU used for the second tranche of TRIG. Universities that performed well were eligible to receive 33 additional resources. Likewise, some universities lost resources after the mid-term due to poor performance. 6.37 Universities used data to strengthen their management capacity. One of the principal goals of HERA is to create a system of autonomous universities. The universities visited as part of this evaluation all confirmed that the project played an important role in creating an internal monitoring system. They reported that this was useful as the university management starts to play a larger role in decision making. Several universities reported that they collected similar type of data for units that did not directly benefit from the TRIGs. The PMU was able to use the monitoring system to target technical assistance to universities that had implementation difficulties. 6.38 The World Bank used the results framework to inform its other higher education operations in Vietnam. During the period of project implementation, the Bank prepared two additional higher education projects. Both of these project made use of HEP II’s monitoring framework in their design and to prepare their own results framework. 7. Lessons 7.1 The Second Higher Education Project was largely successful in meeting its development objectives in an efficient manner. This generates a number of lessons both for Vietnam and the World Bank. 7.2 The key ingredients for project success often come from beyond the sphere of project itself. In no small way, the project’s close alignment with the reform contributed to its strong performance. The project was developed within the context of an extensive higher education reform that had a high level of sustained government and university support. It was also implemented in the context of a larger World Bank program for higher education as well as full support from the Bank’s Country Strategy. Although it was an investment project, HEP II supported many aspects of the reform, which, in turn, strengthened the project. Many of the actions supported by the project directly feed into the reform and this is reflected in the results framework. As illustrated by the experience of the HEP II, a project can effectively build upon the implementation experience of preceding and parallel operations to enhance both the achievement of objectives. 7.3 A combination of Bank support instruments can contribute to a complex reform. The Bank’s support for Vietnam’s higher education reform efforts has adopted a multi- pronged strategy, combing investment and development policy operations with knowledge work and a continuous policy dialogue. The investment operations, including HEP 1 and HEP 2 and to a lesser extent the New Model (German) University, supported both analytical inputs and investments to support the reform. In addition, they were aimed at building much needed technical and institutional capacity at the central government and university levels. In turn, the development policy series has provided substantial financial incentives to move forward more with the challenging policy reforms. It is clear from a review of the development policy activities that the support from the HEP II was crucial in carrying many elements that HERA needed to advance. 34 7.4 Monitoring and evaluation can support the implementation of investment projects. A robust M&E system was a key feature of the HEP II design and contributed to its success. The PMU actively tracked and monitored implementation, beyond the World Bank’s results framework. This led to development of university level monitoring systems. This was a complex task given the diverse nature of TRIG sub-projects, and the large number of implementing entities. The PMU used the data to provide targeted support to the universities, identifying those that needed additional assistance. The PMU also used these indicators to adjust the grants that universities receive. Universities that were implementing the grant slowly saw a reduction in their allocation in the second tranche, which focused on better performing institutions. 7.5 While a grant program may play an important role in developing autonomy, it is not sufficient to ensure autonomy. When well defined, grants in higher education can contribute to strengthen the capacity of universities to act autonomously. Competitive grant programs are common in World Bank-financed higher education projects, building on the positive experience of higher education systems in high income countries. Strengthening autonomy was a central tenant of the ongoing higher education reform, HERA. When the project was initiated, universities were centralized under the direct authority of a Ministry or local government. Interviews show that while individual researchers and research teams had some experience in writing grant proposals, university faculties and school lacked this experience. The introduction of TRIG gave faculties an opportunity to work together and to start to act with some autonomy. The evaluation shows that despite this progress, developing university autonomy is more complicated. 35 References Dang, Q. A. 2009. Recent Higher Education in Vietnam: the Role of the World Bank. Aarhus, Denmark: Danish School of Education. Giesecke, J., Tran, N., Meagher, G., and Pang, F. 2011. Growth and Change in the Vietnamese Labour Market. Monash University. Government of the Socialist Republic of Vietnam. 2006. The Five Year Socio-Economic Development Plan, 2006-2010. Hanoi, Vietnam: Government of Socialist Republic Plan. Government of the Socialist Republic of Vietnam. 2011. The Five Year Socio-Economic Development Plan, 2011-2015. Hanoi, Vietnam: Government of Socialist Republic of Vietnam. Government of Vietnam. 2012. Viet Nam Sustainable Development Strategy for 2011-2020. Retrieved from Government Portal: http://www.chinhphu.vn/portal/page/portal/English/strategies/strategiesdetails?categoryId=30anda rticleId=10050825 Hayden, J., and Thiep, L. Q. 2010. Vietnam's Higher Education System. In Reforming Higher Education in Vietnam, ed. G. Harman, M. Hayden, and P. T. Nghi. London, UK: Springer. Huy, V. Q. 2012. “Determinants of Educational Expenditure in Vietnam.” Internaional Journal of Applied Economics 9(1): 59-72. Khang, D. B. 2007. TRIG Proposals Assessment Report. Bangkok: Asian Institute of Technology. Mincer, J. 1981. Human Capital and Economic Grwoth. Cambridge, USA: National Bureau of Economics. Nghi, P. 2010. The Higher Education Reform Agenda. In Reforming Higher Education in Vietnam, ed. G. Harman, M. Hayden, and P. Nghi. London: Springer. OPCS. 2011. Implementation Completion and Results Report: Guidelines. Washington, DC: World Bank. Pham, H. 2012. New higher education law approved, but sparks criticism. University World News. Retrieved from http://www.universityworldnews.com/article.php?story=20120619111558817 Salmi, J. 2009. The challenge of establishing world-class universities. Washington, DC: World Bank. Sheriden, G. 2010. Viet Nam: Preparing the HIgher Education Development Project. Manila: Asian Development Bank. Socialist Republic of Vietnam. 2007. Project Operational Manual. Hanoi, Vietnam: Ministry of Education and Training. St. George, E. 2010. Higher Education in Vietnam 1986-1998: Transition to a New Era? In Reforming Higher Education in Vietnam, ed. G. Harman, M. Hayden, and P. T. Nghi. London, UK: Springer. Viet Nam News. 2014. Universities need more autonomy to implement education reforms. Viet Nam News. World Bank. 2007. “Project Appraisal Document on a Proposed Credit in the Amoung of SDR 39 Million SDR (US$59.4 million equivalent) to the Socialist Republic of Vietnam for a Second Higher Education Project.” Washington, DC. ______. 2008a. “Implementation Completition and Results Report on a Credit in the Amount of US$83.3 million to the Socialist Republic of Vietnam for a Higher Education Project.” Washington, DC. ______. 2008b. “Vietnam: Higher Education and Skills for Growth.” Washington, DC. ______. 2011a. “Country Partnership Strategy for the Socialist Republic of Vietnam.” Washington, DC. ______. 2011b. “Vietnam- Country partnership Strategy for the period FY12-FY16.” Washington, DC. ______. 2012a. “Implementation Completion and Results Report (IDA-43280, TF-56736) on a credit in the amount of SDR 39.0 SDR (US$59.4 million equivalent) to the Socialist Report of Vietnam for Second Higher Educaiton Project.” Washington, DC. 36 ______. 2012b. “Well Begun, Not Yet Done: Vietnam’s Remarkable Progress on Poverty Reduction and the Emerging Challenges.” Washington, DC. ______. 2013. “Program Document for a Proposed Credit in the Amount of SDR 32.6 million (US50.0 equivalent) to the Social Republic of Vietnam for a Higher Education Development Policy Program Third Operation.” Washington, DC. ______. 2014a. Data. Retrieved March 27, 2014, from World Bank Internet Site: data.worldbank.org ______. 2014b. EdStats. Retrieved from http://datatopics.worldbank.org/education/ ______. 2015. “World Development Indicators.” Retrieved from World Bank Website: data.worldbank.org Yorke, M. 2004. Employability in higher education: what it is –what it is not. York, UK: Higher Education Academy. 37 ANNEX A Annex A. Basic Data Sheet VIETNAM SECOND HIGHER EDUCATION PROJECT (P079665) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 70.50 53.50 75.88 Loan amount (IDA) 59.39 55.31 93.13 Cofinancing (PHRD, Japan) 4.80 2.58 53.75 Cancellation 0.00 7.20 0.00 Cumulative Estimated and Actual Disbursements FY08 FY09 FY10 FY11 FY12 FY13 FY14 Appraisal estimate (US$M) 4.56 21.32 39.59 56.35 59.39 59.39 59.39 Actual (US$M) 6.38 17.85 25.33 37.70 57.04 55.44 55.31 Actual as % of appraisal 139.91 83.72 63.98 66.90 96.04 93.34 93.13 Date of final disbursement: 07/23/2012 Project Dates Original Actual Initiating memorandum 05/01/2003 09/11/2003 Negotiations 04/16/2007 05/10/2007 Board approval 05/01/2005 06/20/2007 Signing 11/22/2007 11/22/2007 Effectiveness 03/26/2008 03/26/2008 Closing date 06/30/2012 06/30/2012 ANNEX A 38 Task Team Members Names Title Unit Lending ALBERT-LOTH Agnes Senior Finance Officer LOAG2 ARISTORENAS Rosario Program Assistant EASHD BALL Carol Senior Operations Officer EASHD DUONG Quoc Vinh Environment Specialist EASES Di GROPELLO Emanuela Lead Human Development Economist AFTEW KIJIMA Rie M&E Specialist MNSHD MAI Thi Thanh Senior Education Speciatlist EASED MOOCK Peter Education Economist Consultant NESMITH Katherine Junior Professional Associate EASHD NGUYEN Hoai Linh Operations Analyst EASHD NGUYEN Hoi Chan Senior Counsel LEGEA NGUYEN Minh Nguyet Program Assistant EACVF NGUYEN Thi Phuong Trang Environment Specialist EASRE NGUYEN Thuy Anh Operations Analyst EASHD NGUYEN Van Linh Program Assistant EACVF PHAM Thi Mong Hoa Senior Social Development Specialist EASSO PHAN Thi Xuan Quynh Financial Management Analyst EAPCO THOMSON Jennifer Senior Financial Specialist EAPCO TRAN Thi Hai Chau Finance Analyst LOAG1 TRAN Quang Thong Financial Management Specialist EAPCO TRAN Thi Hang Hiet Procurement Specialist EAPCO WAITE Jeffrey Team Leader EASHD WATKINS Alfred Lead Science and Technology Specialist HNDED WEBER Luc Higher Education Specialist Consultant WOODHALL Maureen Higher Education Grants Specialist Consultant Supervision/ICR Thanh Thi Mai Senior Education Specialist, TTL EASHE Sylvester Kofi Awanyo Lead Procurement Specialist EAPPR Carol Chen Ball Consultant EASHD Rie Kijima Consultant EASHD Hoi-Chan Nguyen Consultant OPCIL Anh Thuy Nguyen Operations Officer EASHH Nguyet Minh Nguyen Program Assistant EACVF Quynh Xuan Thi Phan Financial Officer GEF 39 ANNEX A Names Title Unit Jennifer K. Thomson Financial Management Specialist SARFM Hiet Thi Hong Tran Senior Procurement Specialist EAPPR Anh Lan Vu Temporary EACVF Binh Thanh Vu Senior Education Specialist EASHE Jeffrey Waite Lead Education Specialist MNSHE Cecilia Zanetta ICR Specialist Consultant Staff Time and Cost Stage of Project Cycle Staff Time and Cost (Bank Budget Only) US$ Thousands (including No. of staff weeks Lending travel and consultants costs) FY03 45.08 FY04 64.57 FY05 49.62 FY06 125.35 FY07 142.35 FY08 0.30 Total: 427.19 Supervision/ICR FY08 52.88 FY09 45.51 FY10 34.85 FY11 57.58 FY12 116.54 FY13 41.85 Total: 349.21 40 ANNEX B Annex B. Participating Universities Grant Name Region Rating Description of Grant (US$ million) An Giang Mekong 1.15 MS Can Tho University Mekong 3.50 HS Broad support. Equipment, education for faculty Da Nang University N. Central Coast 3.50 S Four projects. Equipment, education for faculty Dong Thap University Mekong 0.75 HS Foreign Trade University Red River 3.00 S Five projects. Equipment, library Hanoi Agriculture University Red River 3.30 HS Five projects. Lab equipment, education for faculty, library Hanoi Medical University Red River 4.00 S One project. Advanced lab equipment Hanoi Pedagogic University Red River 2.60 S Hanoi Polytechnic University Red River 3.50 S One project. Lab equipment HCM City Economics Univ. Southeast 2.40 MU Three projects. Library, facilities, research projects HCM Medical University Southeast 3.50 MS One project. HCM City Pedagogic University Southeast 3.00 U Two projects. Facilities for e-learning Hue University N. Central Coast 3.50 S Broad support. National Economics University Red River 3.20 MS Five projects. Library, IT infrastructure Tay Bac University N. Central Coast 0.75 MS Tay Nguyen University Highlands 1.15 U Thai Nguyen University Northeast 2.90 HS Four projects. Education for faculty, Tra Vinh University Mekong 1.20 HS Transport University Red River 3.00 S Three projects. Education for faculty, advanced lab equipment Vietnam National Univ.- Hanoi Red River 4.00 MU Three projects. Vietnam National Univ.- HCM City Southeast 3.50 S Three projects. Library Vinh University N. Central Coast 2.60 S Five projects. IT infrastructure, education for faculty Note: Ratings are at the end of the project (HS=highly satisfactory, S=satisfactory, MS=moderately satisfactory, MU=moderately unsatisfactory, U=unsatisfactory). Description of grant is based on panel report for the first round. Grant amount is based on initial approved amount. Source: Project documents. 41 ANNEX C Annex C. List of Persons Met World Bank Michael Crawford Thanh Thi Mai Suhas Parendekar Dung Kieu Vo Jeffrey Waite Michel Welmond Beneficiaries Universities Hanoi Agriculture University Vu Van Liet Vice Rector Deputy Director, International Le Thi Bich Lien Cooperation Office Vice Dean, Faculty of Food Nguyen Hoang Anh Science Nguyen Thi Phuong Thao Dean, Faculty of Biotechnology Hanoi University of Science and Technology Nguyen Canh Luong Vice President Doan Xuan Huong Coordinator, Industry Linkage Quan Le Ha Dean, School of Biotechnology Troung Quoc Phong Head, Proteomics Laboratory Nguyen Ngoc Trung School of Engineering Physics National University of Vietnam- Hanoi/University of Engineering and Technology Nguyen Ngoc Binh Rector Nguyen Anh Thai Vice Rector National University of Vietnam- Hanoi/University of Science Nguyem Van Noi Rector Nguyen Hoang Luong Director, Nano and Energy Center Ngac An Bang Dean, Faculty of Physics Hoang Minh Thao Vice Dean, Faculty of Geology Nguyen Duc Hoai Dept. of International Relations University of Medicine and Pharmacy- Ho Chi Minh City Do Van Dung Vice President Tran Hung Dean, Faculty of Pharmacy Nguyen Thien Hai Vice Dean and Head of Planning Tran Quang Trung Direct, Medical Training Center National University of Vietnam- Ho Chi Minh City ANNEX C 42 Others Professor and former director-HEP Nguyen Duc Thuan 2 Tran Thi Hoa Director, CHI Health Corporation