Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004197 IMPLEMENTATION COMPLETION AND RESULTS REPORT Credit No. 4685-IN ON A CREDIT IN THE AMOUNT OF SDR 186.4 MILLION (US$ 300 MILLION EQUIVALENT) TO THE REPUBLIC OF INDIA FOR A INDIA: TECHNICAL ENGINEERING EDUCATIONAL QUALITY IMPROVEMENT II ( P102549 ) September 28, 2017 Education Global Practice South Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective as of Sep 16, 2017) Currency Unit = Indian Rupees INR 64.17 = US$1 US$ 1.40 = SDR 1 FISCAL YEAR April 1 - March 31 Regional Vice President: Annette Dixon Country Director: Junaid Kamal Ahmad Senior Global Practice Director: Jaime Saavedra Chanduvi Practice Manager: Keiko Miwa Task Team Leader(s): Tara Beteille, Francisco Marmolejo ICR Main Contributor: Namrata Raman Tognatta ABBREVIATIONS AND ACRONYMS AICTE All India Council for Technical Education ATU Affiliating Technical University BoG Board of Governors B.TECH Bachelor of Technology CoE Centres of Excellence eSAR e-Self Assessment Report EA Environmental Assessment EAP Equity Action Plan EIRR Economic Internal Rate of Return EMF Environment Management Framework FICCI Federation of Indian Chambers of Commerce and Industry FM Financial Management FMM Financial Management Manual GDP Gross Domestic Product GoI Government of India HR Human Resources IDA International Development Association IIM Indian Institute of Management ISP Implementation Support Plan IIT Indian Institute of Technology IT Information Technology KIT Knowledge Incubation Cells KPI Key Project Indicators M&E Monitoring and Evaluation MHRD Ministry of Human Resource Development MIS Management Information System MOOC Massive Open Online Course MOU Memorandum of Understanding M.TECH Master of Technology NBA National Board of Accreditation NIT National Institute of Technology NPV Net Present Value NPD National Project Director NPIU National National Project Implementation Unit NSC National Steering Committee NSS National Sample Survey PDO Project Development Objective PMSS Procurement Management Support System R&D Research and Development RUSA Rashtriya Uchchatar Shiksha Abhiyan SIL Specific Investment Loan SPFU State Project Facilitation Unit SAR Self-assessment report TEQIP Technical Education Quality Improvement Project UG Undergraduate UGC University Grants Commission UT Union Territory QEEE Quality Enhancement in Engineering Education TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .......................................9 II. OUTCOME .................................................................................................................... 13 A. RELEVANCE OF PDOs ............................................................................................................ 13 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 15 C. EFFICIENCY ........................................................................................................................... 19 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 20 E. RATING WITH REVISED OBJECTIVES ....................................................................................... 20 F. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 22 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 23 A. KEY FACTORS DURING PREPARATION ................................................................................... 23 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 24 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 25 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 25 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 27 C. BANK PERFORMANCE ........................................................................................................... 29 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 30 V. LESSONS AND RECOMMENDATIONS ............................................................................. 30 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 33 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 40 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 42 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 43 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 46 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 58 ANNEX 7. ACHIEVEMENTS BY COMPONENTS ....................................................................... 59 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name INDIA: TECHNICAL ENGINEERING EDUCATIONAL QUALITY P102549 IMPROVEMENT II ( P102549 ) Country Financing Instrument India Specific Investment Loan Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Republic of India Ministry of Human Resource Development Project Development Objective (PDO) Original PDO The second phase of the Technical/Engineering Education Quality Improvement Project seeks to strengthen selected institutions to produce more employable and higher quality engineers and prepare more post-graduate students to reduce faculty shortage. Page 1 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 300,000,000 217,910,321 191,908,912 IDA-46850 Total 300,000,000 217,910,321 191,908,912 Non-World Bank Financing Borrower 200,000,000 175,454,592 157,966,128 Total 200,000,000 175,454,592 157,966,128 Total Project Cost 500,000,000 393,364,913 349,875,040 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 18-Mar-2010 06-Aug-2010 13-Dec-2013 31-Dec-2014 31-Mar-2017 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 06-Jan-2014 46.61 Change in Results Framework Change in Components and Cost Cancellation of Financing Change in Financing Plan 30-Sep-2014 68.26 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) 26-Sep-2016 133.07 Change in Loan Closing Date(s) Change in Disbursements Arrangements Change in Institutional Arrangements Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Highly Satisfactory Satisfactory Substantial Page 2 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 30-Jun-2010 Satisfactory Satisfactory 0 02 31-Jan-2011 Satisfactory Satisfactory 8.37 03 13-Aug-2011 Satisfactory Satisfactory 8.37 04 03-Mar-2012 Satisfactory Satisfactory 8.37 05 14-Sep-2012 Satisfactory Satisfactory 30.54 06 28-Jan-2013 Satisfactory Moderately Satisfactory 30.54 07 13-Sep-2013 Moderately Satisfactory Moderately Satisfactory 30.54 08 12-Jan-2014 Moderately Satisfactory Moderately Unsatisfactory 30.54 09 01-Jun-2014 Satisfactory Moderately Satisfactory 44.49 10 09-Dec-2014 Satisfactory Moderately Satisfactory 52.36 11 17-Jan-2015 Satisfactory Moderately Satisfactory 52.36 12 30-Jun-2015 Satisfactory Moderately Satisfactory 68.53 13 15-Sep-2015 Satisfactory Satisfactory 73.90 14 21-Mar-2016 Satisfactory Satisfactory 78.79 15 27-Aug-2016 Satisfactory Moderately Satisfactory 90.42 16 08-Nov-2016 Satisfactory Moderately Satisfactory 97.89 17 31-Mar-2017 Satisfactory Satisfactory 114.17 SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Public Administration - Education 6 Tertiary Education 94 Page 3 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Economic Policy 13 Trade 13 Trade Facilitation 13 Private Sector Development 113 Business Enabling Environment 13 Innovation and Technology Policy 13 Jobs 100 Public Sector Management 2 Public Administration 2 Municipal Institution Building 2 Human Development and Gender 72 Education 72 Access to Education 18 Science and Technology 18 Teachers 18 Standards, Curriculum and Textbooks 18 ADM STAFF Role At Approval At ICR Regional Vice President: Isabel M. Guerrero Annette Dixon Country Director: N. Roberto Zagha Junaid Kamal Ahmad Senior Global Practice Director: Michal J. Rutkowski Jaime Saavedra Chanduvi Practice Manager: Amit Dar Keiko Miwa Tara Beteille, Francisco Task Team Leader(s): Andreas Blom Marmolejo ICR Contributing Author: Namrata Raman Tognatta Page 4 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Page 5 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Country Context. TEQIP II was prepared at a time of accelerated economic growth in India. The country’s GDP was growing at over 7 percent per annum– it was the fourth largest economy in PPP terms and a leading player in sectors such as information technology, telecom and business outsourcing. However, there was less progress on poverty reduction and estimates showed that over 400 million people were living on less than $1.25 per day 1. The Government of India’s (GoI’s) priorities for the project period, articulated through the 11th Five-Year Plan (2007-2012), envisioned sustaining economic growth while making it more inclusive. A major challenge in sustaining rapid economic expansion was the shortage of skills in the economy. Sectors most severely affected by the skills shortage were IT, infrastructure and power, all engineering intensive areas. Employers were dissatisfied with the skills of newly educated engineers –64% of employers were only somewhat satisfied with the quality of new hires and found severe gaps in graduates’ higher-order thinking skills and soft skills.2 Strengthening the technical education system3 of the country was seen as core to addressing skills shortages and driving the country’s exports, especially in IT and business-services. 2. Sector Context. India had made notable progress in primary education; net enrollment had increased to 92 percent in 2007 and efforts were being made to improve and expand access at the secondary level. At higher levels, however, progress was slower. Although the number of engineering institutions in the country was growing4, enrollments in higher secondary school and higher education were quite low5 – 48 percent and 16 percent of the relevant age group, respectively. Comparable figures for higher secondary and higher education in Brazil were 89% and 65%, and in China were 37% and 23%. The rapid expansion of technical education had also come at the cost of quality. Outdated curricula, inadequate academic infrastructure, shortage of qualified teachers, poor industry interactions and non-participative and over-centralized governance mechanisms characterized the system. 3. The 11th Five Year Plan following TEQIP I thus articulated the following priorities for technical/engineering education in India and mentioned TEQIP II as key to improving the quality of existing institutions. a. Quality enhancement: Improving teacher competence; motivation and teaching practices; software; equipment; flexible and frequently updated curricula; research and systems; including Information and Communication Technologies b. Equity: Expansion of access in terms of institutional infrastructure c. Autonomy and accountability for better governance 4. Rationale for Bank assistance. The Country Assistance Strategy (2009-2012) for India during TEQIP II 1 World Development Indicators (2013) 2 Blom & Saeki (2011). Employability and skill set of newly graduated engineers in India. Policy Research Working Paper 5640, World Bank, Washington DC. 3 Technical education in India includes programs in engineering, technology, management, architecture, town planning, pharmacy, applied arts and crafts and hotel management and catering technology. 4 Nearly 1000 engineering institutions were added between 2007 and 2009. All India Council of Technical Education: http://www.aicte-india.org/downloads/Growth_Technical_Institutions_310514.pdf 5 Figures for 2009 from UNESCO, UIS Page 6 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) preparation focused on achieving rapid, inclusive growth, ensuring sustainable development and improving the effectiveness of service delivery. The project contributed to two pillars: improving effectiveness of service delivery (governance reforms and strengthening systems such as data management and procurement) and removing skills constraints to growth to achieve rapid and inclusive growth (quality and equity oriented activities focusing on student learning and employability, faculty training and research). The project also contributed to the cross-cutting focus on effective public spending and results through monitoring and evaluation, implementation of governance reforms and strengthening procurement and financial management systems. Theory of Change (Results Chain) Figure 1. TEQIP II Theory of Change Long-term Improved learning outcomes, higher placement rates, higher wages. Outcomes More employable and higher quality engineers and reduction in faculty shortages 1. Share of supported programs that are accredited or applied for PDOs/ 2. Percentage of institutes with academic autonomy Outcomes 3. Percentage of faculty with at least M.Tech (regular and contract) 4. Total number of Masters and PhD students enrolled Higher satisfaction among students and faculty, improved student transition rates, student Intermediate internships, partnerships with industry, tracking student placement, publications and co-authored Outcomes articles in refereed journals, more revenue from externally funded R&D, more post-graduate students, more faculty trained, student satisfaction with teaching, more teachers with MTech/PhD, institutional governance self-reviews and development plans, training in strategic planning, more reliable and verified data to guide implementation and assess performance. Outputs Equity action plans, placement cells in institutes, pedagogical and subject area training for faculty, institutional reforms, more faculty and staff, Four Funds established, satisfaction surveys, feedback from mentors and auditors, bibliometric study, resource utilization study, interim impact evaluation, BoG guidelines, management training for institutes. Activities Improving teaching-learning facilities and research equipment, curricular reforms, training/research opportunities for faculty and students, TAs/RAs, industry collaborations, scholarships, workshops on good practices and efficient policies in technical education and institutional quality assurance, leadership development including developing institutional strategic plans, mentoring, merit-based selection of institutes, performance benchmarking. 5. The theory of change underlying TEQIP II is depicted in Figure 1. The project had two main objectives – (i) to strengthen institutions and (ii) to produce more postgraduate students. These were necessary for achieving the longer term goals of improving quality, employability and reducing faculty shortages. Key principles guiding the project were: a. Political and bureaucratic commitment, especially commitment by participating states to provide an enabling environment for institutions to implement academic and non-academic reforms6 b. Rewarding merit through competitive selection of institutions to build a culture of excellence and accountability c. Effectively exercise autonomy and enhancing the ability of institutions to determine goals, gather, analyze and use data for decision making d. Spreading reforms by building a culture of sharing that enabled learning from each other Together, these principles played a key role in facilitating behavior change in the project, cementing the shift from a compliance-based culture to one rewarding performance and sharing. 6 States committed to granting project institutes autonomy and facilitating internal governance like keeping the composition and functioning of the Board of Governors free of political presence. Page 7 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) 6. The project aimed to address key challenges facing the sector through a series of academic and non-academic reforms in participating institutions. The former focused on revising curriculum, improving transition rates, faculty development, research, industry linkages and employment-focused activities. The latter focused primarily on governance oriented reforms, such as strengthening the functioning of the Board of Governors, greater use of data for decision-making, rewarding merit, emphasizing accreditation, transparency and building mechanisms for accountability and sustainability. Project Development Objectives (PDOs) 7. The project sought to “strengthen selected institutions to produce more employable and higher quality engineers and prepare more post-graduate students to reduce faculty shortage�. Key Expected Outcomes and Outcome Indicators 8. The following PDO indicators were identified for each objective. Original outcome Indicators for Objective 1 – Institutional strengthening to produce more employable and higher quality engineers: a. Employer satisfaction index b. Share of supported programs that are accredited c. Percentage of institutions with academic autonomy d. Percentage of faculty with at least a Masters of Technology (M.Tech) Original outcome Indicators for Objective 2 – Producing more graduate students to reduce faculty shortage: e. Number of Masters and PhD students enrolled 9. ‘Employer satisfaction index’ was not included in the list of Key Project Indicators (KPIs) in the supplemental letter which was signed along with the legal agreement. However, it was included as a PDO indicator in the Results Framework in the PAD, which also specified that this was a long-term objective.7 PDO indicators on accreditation8 and autonomy (indicators for Objective 1) were also indicative of progress towards long-term objectives of producing more employable and higher quality engineers. Components 10. Two components were identified to achieve the PDO9 – a. Component 1 - Improving Quality of Education in Selected Institutions Estimated: US$ 453 million (IDA – US$ 280 million) Actual: US$ 334.2 million (IDA – US$ 183 million) Institutions were competitively selected under two funding windows. The first competitive window focused on improving learning outcomes and employability of graduates in selected institutions and used a modest set of eligibility 7 See pp. 5 and 29 of Project Appraisal Document 8 Accreditation of programs in technical education is carried out by the National Board of Accreditation (NBA), an autonomous body. NBA evaluates if technical education programs adhere to specified norms and standards for quality set by the AICTE. Key parameters of the outcomes-based accreditation process include processes in place for continuous improvement of outcomes, training and placement facilities, student performance and placements. 9 The following descriptions are from the project PAD. Page 8 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) criteria for selection. This component did not include institutions from TEQIP I. The second window supported more advanced institutions (including those from TEQIP I) and applied a more stringent set of criteria for selection. The window focused on scaling up postgraduate education and research and innovation. A subset of institutions under the second competitive window were supported to establish ‘Centers of Excellence’ to carry out relevant interdisciplinary R&D activities in collaboration with industry. A faculty development program was designed and separately funded for pedagogical training to faculty at competitively selected institutions.10 The 3 sub-components were: a. Component 1.1 - Strengthening institutions to improve learning outcomes and employability of graduates, b. Component 1.2 - Scaling up post-graduate education and demand-driven research and development and innovation, i. Establishing Centers of Excellence with private sector collaboration, c. Component 1.3 - Faculty development for effective teaching b. Component 2 - Improving System Management Estimated: US$ 38 million (IDA US$ 17 million) Actual: US$ 15.7 million (IDA US$ 8.6 million) This component focused on capacity building of key implementers at the central, state and institutional level for effective implementation of reforms required in the competitive funds. This component also focused on improving evidence-based policymaking and administration in the sector through the provision of precise, reliable and timely information on performance of selected institutions. The 2 sub-components under this component were: a. Component 2.1 - Capacity building to strengthen management, and b. Component 2.2 - Project monitoring, evaluation and management 11. US$ 9 million was unallocated and the PAD provided the flexibility to allocate this to any well-performing activity or component. Hence, the Financing Agreement only included a single category for project expenditures and did not include a separate category for “unallocated funds�. 12. There is a difference of US$ 105 million between estimated and actual costs. The project had some savings due to (i) significant fluctuation in the exchange rate during the project period11, (ii) changes in financing terms12, and (iii) economy from procurement due to the innovative PMSS system. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 13. The PDO was not revised during the project period. Outcome indicators and target values were modified; the modifications and justifications are presented in Table 1.13 Table 1. Changes to PDO and IO indicators during the project period Original Indicators and Targets Modifications at First Restructuring Modifications at Second (January 2014) Restructuring (September 2014) PDO Indicator: Employer satisfaction Modification: This indicator was dropped. 10 A summary of activities undertaken under each component is included in Figure 1. 11 Details on savings from exchange rate fluctuations are discussed in #19. 12 Details on changes to financing terms are discussed in #20. 13 This information is available in the Restructuring Papers. Page 9 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) index Justification: This indicator measured a longer-term goal as per PAD (pp.5 and 29) and was not included as a “key performance indicator� in the supplemental letter which was signed along with the FA. As this indicator measured a longer-term goal, the baseline survey included all employers and not just those employing TEQIP graduates. PDO Indicator: Share of supported Modification: Share of supported programs that are Modification: programs that are accredited accredited or applied for Original Target: 50 Revised Target: 55 Justification: The definition of this indicator was modified to acknowledge and accommodate delays in Justification: End target value raised the accreditation process and procedural norms laid to incentivize further achievement* down by NBA that could unfairly penalize institutions 14 ready for accreditation. PDO Indicator: No Change Modification: Percentage of institutions with Original Target: 65 academic autonomy Revised Target: 70 Justification: End target value raised to incentivize further achievement* PDO Indicator: Percentage of Faculty Modification: Percentage of Faculty with at least an Modification: with at least an M.Tech. M.Tech (regular and contract) Original Target: 60 Revised Target: 88 Justification: The revision specified that faculty included those who were regular as well as those on Justification: End target value raised contract. to incentivize further achievement* PDO Indicator: Number of Master and Modification: Total number of Master and PhD Modification: PhD students enrolled students Original Target: 22500 st Baseline and target values were changed as follows – Target revised during 1 Original Baseline: 15000 restructuring: 34000 Original Target: 22500 Revised Target: 41000 Revised Baseline: 30000 Revised Target: 34000 Justification: End target value raised to incentivize further achievement* Justification: The definition was revised to include all students and not just those in the first year of the program. This also required a change in baseline and target values. IO Indicator: Increase in the number of Modification: Number of publications in refereed Modification: 14 NBA’s accreditation process involves various stages (applying, fulfilling pre-qualifiers, completing and filing a detailed self- assessment report (SAR), and an evaluation visit by an external team) and entails significant cost on the part of institutions. The fee for accrediting an undergraduate engineering program is INR 200,000 [3,220 USD equivalent] which is paid in full by institutions in 2 installments prior to the evaluation visit. At the time of application, institutes are often asked to reapply a number of times until they meet all requirements and criteria and get assigned an e-SAR number. Thus, by the time of the evaluation visit, institutions have gone through a voluntary, time-consuming and expensive process and signaled, through the e-SAR, completion of requirements for accreditation. However, delays in scheduling the evaluation visit (due to an insufficient supply of accreditors) could unfairly penalize institutions. In some cases, the visits can take so long that the information included in the e-SAR becomes dated and institutes are expected to reapply. Further, accreditation is valid for 3 years only and the nature of this activity required institutions to reapply once their accreditation period has ended. Finally, NBA modified the accreditation process midstream through the project making it outcomes-based. This caused significant delay as institutions had to determine the implications of this change to the requirements and application process. Page 10 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) publications in refereed journals (within journals (within the field of Engineering) Original Target: 9000 the field of Engineering) Revised Target: 15000 Justification: The indicator was revised for clarity. Justification: End target value raised to incentivize further achievement* IO Indicator: Percentage of Faculty with Modification: Percentage of Faculty with or pursuing Modification: or enrolled in M.Tech and PhD. M.Tech and PhD (regular and contract) Original Target: 73 Revised Target: 90 Justification: The revision specified that faculty included those who were regular as well as those on Justification: End target value raised contract. to incentivize further achievement* IO Indicator: Number of faculty Modified as: Number of faculty members that have Modification: members that have benefitted from the benefitted from the teaching effectiveness training Original Target: 1000 teaching effectiveness training under sub-component 1.3 Revised Target: 2200 15 Justification: Institutes of National Importance were Justification: End target value raised leveraged to provide pedagogical training to faculty. to incentivize further achievement* The modified indicator definition reflected this change in approach to training. IO Indicator: Transition rate of students No Change Modification: from disadvantaged backgrounds from Original Target: 55 the first year to second year of Revised Target: 60 undergraduate study Justification: End target value raised to incentivize further achievement* IO Indicator: Percentage of externally Modification: Baseline and target values were No Change funded research and development changed as follows – projects and consultancies in total Original Baseline: 10 revenue. Original Target: 20 Revised Baseline: 6 Revised Target: 12 Justification: Revisions were based on changes in the calculation of total revenue earned by institutions. IO Indicator: Difference in the Modification: aggregate index for accreditation, This indicator was dropped autonomy and faculty with at least an M.Tech between institutions in states Justification: The formula used to calculate this lagging in technical education and indicator was not agreed to at baseline. others. IO Indicator: Number and share of Modification: Direct beneficiaries (number) of which Modification: female students in the project female (percentage) Original Target: 350000 institutions Revised Target: 405000 Justification: This indicator was revised to bring in line with IDA Core Indicator. Justification: End target value raised to incentivize further achievement* IO Indicator: State Governance Capacity Modification: Index and Institutional Governance This indicator was dropped Capacity Index Justification: The calculation methodology for this indicator was not clearly determined at the time of preparation. 15 ‘Institute of National Importance’ is a status conferred on a higher education institute by an act of parliament and recogniz es the institute as a pivotal player in developing highly skilled personnel. These institutes receive special recognition and funding. Page 11 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) IO Indicator: Fully functional MIS - Modified as: Fully functional MIS - Number of Modification: Number of institutions reporting at institutions reporting at least 70% of the indicators Original Target: 150 least 50% of the indicators Revised Target: 160 Justification: Experience from implementation indicated that the definition of this indicator could be Justification: End target value raised stretched. to incentivize further achievement* New Indicator: Number of governance Modification: Modification: self-reviews received These indicators were added. Governance self-reviews: Original Target: 80 Justification: The indicators were introduced to Revised Target: 190 document progress being made on governance reforms. Baseline and targets were established. Governance development plans: New Indicator: Number of governance Original Target: 20 development plans received Revised Target: 180 Justification: End target value raised to incentivize further achievement* Note: * In light of the introduction of performance benchmarking, the project had made significant improvements in implementation since the MTR, with steady progress against targets and even exceeding some targets. Given the request for an extension of closing date, target values were stretched to incentivize further achievements. Revised PDO Indicators 14. Changes to PDO and IO indicators are described above. Revised Components 15. The component, Improving System Management, was revised in September 2016 as part of a Level 2 Restructuring16. TEQIP III was approved by the Board in June 2016 and before effectiveness the Government of India requested that some resources from TEQIP II (approximately US$ 2 million) be used to set up structures to support initial implementation of TEQIP III. The All India Council for Technical Education (AICTE) would implement two activities under Component 2 of TEQIP II – (i) introducing a uniform learning assessment system for engineering students, and (ii) undertaking a review of the curriculum. Consequently, implementation arrangements were revised and changes made to the Financing Agreement. Other Changes 16. Project extensions: The closing date was extended twice. The initial period of project implementation, slowed by delay in selection of institutes, focused on building the capacity of selected institutions and state implementing agencies.17 Since the project had picked up pace and made significant improvements, an extension would allow further development of project institutions by way of more academic and governance-focused reforms. The extension also gave the team an opportunity to test innovations like performance benchmarking and incentives in the form of additional funding to motivate institutions towards better performance. Thus, an 18-month extension – from May 1, 2014 to October 31, 2016 – was processed as part of a Level 2 restructuring in September 2014. The second extension – from 16 The Restructuring Paper is the main source for the information provided here. 17 Selection of institutions was drawn out for several reasons; MHRD and NPIU received many inquiries from state secretaries, institutional directors and parliamentarians on lowering eligibility criteria, the institutional development plan proposals were long and cumbersome to fill and evaluate, several institutions requested re-consideration of proposals which required a review process, and some private institutions filed representation letters due to disagreement with selection decisions. The first round of selection nd took about 12 months to complete. The 2 round had a quicker turnaround time. Page 12 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) October 31, 2016 to March 31, 2017 – was to enable institutions to increase utilization of project funds and improve performance. The extension was processed through a Level 2 restructuring in September 2016. 17. Cancellation of funds: On December 27, 2013 the Government of India requested a cancellation of SDR 51.77 million (approx. US$ 80 million equivalent) of the remaining credit. Considerable shift in the exchange rate had resulted in significant savings under the project.18 The implementing agencies indicated that total spending at closing would be equal to the local currency amount of the original credit. Since this local currency amount was significantly less than the SDR amount remaining in the credit, the extra funds were cancelled. The cancellation was processed as part of a Level 2 restructuring in December 2013. 18. Financing changes: In March 2016 the Government of India requested the World Bank to retroactively modify the percentage of expenditures financed under “eligible expenditures� through the Credit from 60 percent to 50 percent as of December 2015. This was a result of changes in GoI’s arrangements for co -financing with state governments, with the center-state funding proportion reduced from 60:40 to 50:50. The resulting change in financing terms was part of a Level 2 restructuring processed in September 2016. Rationale for Changes and Their Implication on the Original Theory of Change 19. The changes discussed did not impact the original theory of change and did not have implications on project scope. Changes to one PDO indicator warrant further explanation. The “share of supported programs accredited� was as a key indicator for institutional strengthening and quality. Delays in the accreditation process due to short supply of credible accreditors and change in procedures to an outcomes-based accreditation process was a serious concern which the team tried to address through multiple meetings with NBA and MHRD. The team decided to broaden the PDO indicator to include institutions that had applied for accreditation and those that had received accreditation to not penalize institutions for delays that were not under the control of the project. The justification for including “applied for� to the indicator definition was that being assigned an e-SAR number (this stage in the application process indicates that the only step pending is visit by the evaluation team) implied that institutions had fulfilled the necessary requirements and eligibility criteria for accreditation (often after multiple rounds of application) at significant cost and time (see footnote #14). Completing the SAR (i.e. institutions identified as ‘applied for’) therefore demonstrates a certain level of commitment and preparedness on the part of institutions to undertake a voluntary, time-consuming and expensive process, and signaling, through the SAR, completion of requirements for accreditation. Further, these institutions are also very likely to successfully achieve accreditation at the end of the process since 85% of institutions completing the e-SAR successfully achieve accreditation.19 II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 20. The Relevance of PDOs is rated as ‘High’. The objectives of TEQIP II were relevant at the time of project approval and continue to be so today. The project is well aligned with GoI’s continued focus on sustainable and inclusive growth in the 12th Five Year Plan. The Plan focuses on increasing the supply of skilled workers to drive the economy while 18 The exchange rate was INR 46: 1 US$ in 2010 and had shifted to INR 60:1 US$ in 2013. 19 In discussion with officials from the National Accreditation Board. Page 13 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ensuring that low-income states receive special emphasis to catch up. The current Country Partnership Strategy (2013- 17) similarly identifies two areas of engagement – integration and inclusion – participation of low-income states and increasing the production of high quality workers for growing sectors of the economy and “improving demand -driven skills for productive employment� as one of the strategic outcomes. 21. The components of the project are highly relevant and respond to these engagement areas and objectives. Low quality, outdated/irrelevant curriculum, faculty shortages and small research output are among the biggest challenges in technical/engineering education in India. TEQIP II addressed these by supporting institutions to achieve excellence through academic reforms and improving institutional governance, management and administrative practices to build autonomy and accountability (Component 1), establishing Centers of Excellence and scaling up R&D activities (Component 1.2) and improving quality of teaching through faculty development (Component 1.3). Improvements in these areas led to higher participation in postgraduate engineering programs, with the potential for addressing faculty shortages in the sector. The project also recognizes the need to build capacity in the system to support reforms, through governance and management reforms at the institute and system level, and this is addressed by Component 2. 22. Involving the private sector was a key mechanism used in TEQIP II to improve the relevance and quality of engineering education and make it more demand-driven. Industry was represented as Chairs on many Boards of Governors (BoGs) across TEQIP II institutions. On academic and research aspects, institutions built close linkages with industry involving them in curriculum revisions, lectures and faculty consultancies, and in establishing strategic research partnerships. At the College of Engineering, Pune industry advisory boards have been set up in each department. These boards chart students’ future learning needs in their specific sector and have been instrumental in expanding opportunities for collaborative research between faculty/students and industry. The Institute of Chemical Technology, Mumbai has set up the ‘ICT Research and Innovation Park’ in collaboration with industry. This activity aims to facilitate the transfer of technology and research know-how generated at ICT to other research organizations and universities, assist start-ups and SMEs, and work with industry to maintain a competitive edge in the manufacturing sector. 23. Finally, the project aimed at leveling the playing field through inclusion of institutions in educationally lagging states in the project and development of institutional Equity Action Plans to support students from disadvantaged groups (Scheduled Castes, Scheduled Tribes and Other Backward Classes) and girl students. The project targeted selecting 20% of project institutions from states lagging in technical education.20 As per this definition, 24% of institutions were from lagging states. Implementation of the project in these states was closely supported by mentors, who are senior academics from leading institutions in the country, and NPIU. At the institutional level, the project tracked key outcomes such as transition rates separately for girl students and students from disadvantaged groups while making efforts to support these students through activities articulated in their Equity Action Plans. The focus on equity and inclusion continues through TEQIP III activities – the project’s objective is to enhance quality, equity and efficiency in institutes from the states described above. Key components to achieve these objectives include mentoring and twinning of these institutes with well-performing institutes from TEQIP I and II. 24. The activities initiated under the two components fundamentally changed the way institutes behaved. From being demotivated participants in an overly-centralized system, institutes were now making decisions leading to excellence in learning and research, adapting to local contexts. 20 States lagging in technical education were defined as those that either have only one Engineering Institution or less than one Engineering Institution per million population as per AICTE’s approved list of Engineering Degree Institutions in 2004. As pe r this definition, 19 states were identified as lagging states in technical education. This list also included 7 states identified as low-income in the Country Partnership Framework. Page 14 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Rating: HIGH B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 25. Project beneficiaries included 190 institutions21 from 23 states and union territories (UTs). Of these, 26 institutions were centrally funded, 126 were public or public-aided and 38 were private.22 26. TEQIP II had two main objectives - to strengthen selected engineering institutions and to prepare more post- graduate students. The PDOs were meant to achieve higher-level, longer term objectives of improving the quality of engineering education and producing more employable graduates, and addressing faculty shortages. Four indicators23 were identified to measure the project’s performance in meeting the PDO. The project exceeded or achieved all revised, stretched targets identified for the project period and made significant progress towards higher-level objectives. The overall efficacy is rated as ‘High’. 27. PDO 1 – To strengthen selected institutions: Achievement on this PDO was measured using 3 indicators. a. PDO Indicator 1 – Share of supported programs that are accredited or applied for: Achieving accreditation from the NBA is a quality assurance process in the Indian context that recognizes excellence in technical education institutions and provides external verification that an institution has processes in place for continuous quality improvement.24 The indicator on accreditation was modified during the project period to include institutions that had applied for accreditation. As noted in para #19 and footnote #14, the application process requires institutions to fulfill several eligibility criteria (including quantity and quality of physical, technical, financial and human resources, student learning and employability outcomes, faculty contributions, governance and institutional support mechanisms, and so forth) over a long and expensive application process even before NBA accreditors visit the institution for the final evaluation. Completing the application process not only sends a strong signal of the institution’s commitment to quality and the institution’s progress towards eligibility criteria required for accreditation, but also suggests that the institution is very likely to achieve accreditation (about 85% of institutions successfully completing the SAR achieve accreditation). The project surpassed the end target on this indicator by 10 percentage points. By the end of the project period, the share of accredited programs in project institutions more than doubled – going from 30 percent at baseline25 to 65 percent in 2014-15. Further, TEQIP supported programs received higher average accreditation scores (about 30 percentage points higher) than the national average26, further attesting to the quality of institutions under TEQIP. i. Of 2686 eligible programs, 22% were accredited. The remaining (43%) had completed application 21 191 institutions were originally included. On the recommendation of the state government, one private institution in Andhra Pradesh was weeded out of the project due to non-performance. The institution pursued legal action and based on the court’s decision was included back in the project with half the allocated funding. This institution has not been included in the assessment of outcomes. 22 Sub-component 1.1 and 1.2 included 113 and 77 institutions, respectively. 23 The assessment of outcomes uses revised indicators and target values. 24 This is specified in NBA’s developmental approach to promoting excellence in technical education. See http://www.nbaind.org/accreditation.aspx 25 The baseline for this indicator was not modified at the time of restructuring. Before restructuring, 9% of programs were accredited and 14% had applied for accreditation. These achievements were below the 23% target identified for that year. 26 From discussion with NBA officials Page 15 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) requirements for accreditation, perhaps not for the first time given that institutes received accreditation for 3 years after which they had to re-apply. It is likely that a major share of the remaining 35% that were neither ‘accredited’ nor ‘applied for’ belonged to institutes where overall progress had been slower (see para # 55). ii. The focus on accreditation brought about a culture of performance where accreditation/ applying (e-SAR) was accepted as an indicator of well-performing institutes providing quality engineering education. This indicator was also included in performance benchmarking of institutes which created a spirit of competitiveness among them, motivating higher achievement. b. PDO Indicator 2 – Percentage of institutes with academic autonomy: During the project period, the percentage of programs with academic autonomy went from 30 percent at baseline to 70 percent thus achieving the target laid out for the project. c. PDO Indicator 3 – Percentage of Faculty with at least an M.Tech (regular and contract): As part of project activities, institutes were encouraged to spend 10% of their outlay on faculty and staff development including conducting a training needs analysis and deputing faculty to other institutes for upgrading their qualifications. As a result, the proportion of faculty with M.Tech degrees doubled between 2009 and 2017. At the end of the project period, over 92 percent faculty had an M.Tech degree against an end target of 88 percent. 28. PDO 2 – To prepare more post-graduate students: d. PDO Indicator 4 – Number of Master and PhD students enrolled: The number of Master and PhD students enrolled in project institutions went from 30,000 at baseline to 45,658, exceeding the end target by nearly 5000 students. About 5% of these students were enrolled in PhD programs. Table 2. Progress on PDO indicators during project period Baseline PDO Indicators 2010-11 2014-15 2015-16 Target 2009-10 Share of supported programs 30% 23% 65% 65% 55% accredited or applied for Percentage of institutes with 30% 51% 65% 70% 70% academic autonomy Percentage of faculty with at least an 45% 86.05% 90.89% 92.35% 88% M.Tech (regular and contract) Total number of Master and PhD 30,000 35,132 49,251 45,658 41,000 students 29. Contributions to higher level objectives: TEQIP II contributed to the following higher-level objectives – a. Improvement in the quality of engineering education: The project aimed to use governance reforms to strengthen engineering institutions that would in turn improve the quality of education in engineering. The use of accreditation and autonomy as metrics of institutional strengthening and quality have been used globally as well as in India.27 Accreditation is a widely-accepted and well-established proxy indicator of quality that includes not only outcome measures but also the inputs that contribute to overall institutional effectiveness. Further, governments use accreditation to guide their understanding of quality through commonly accepted standards and make decisions about institutions in their country. Autonomy is a 27 See https://openknowledge.worldbank.org/bitstream/handle/10986/26486/113867-WP-PUBLIC.pdf?sequence=5&isAllowed=y Page 16 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) justified proxy indicator of quality in higher education because of the established correlation between greater autonomy, accountability and well-performing institutions. In the Indian case, effectively exercising autonomy releases institutions from the regulations and limitations that could constrain them from effectively responding to the needs of stakeholders. This includes the ability to make independent decisions about academic reforms, evaluation systems, staffing decisions, financial management and sustainability and private sector collaborations. b. Accreditation by the NBA is an outcomes-based evaluation of programs that includes student performance, institutional processes for continuous improvement of outcomes, training and placement facilities at institutions, student placements, faculty contributions and physical, technical and financial resources of the institution as some of the criteria for assessment. Accreditation/applying (e-SAR) is used as a proxy for quality in this project. (See #19, #26 and footnote #14 for a detailed explanation). Achievement on this indicator during the project period suggests positive trends towards quality improvement in the engineering sector. c. 70% of project institutions obtained academic autonomy. Institutions that effectively exercised autonomy 28 made decisions about their leadership, faculty appointments, curriculum, teaching, examinations and research priorities. Institutions restructured or revised curricula – often with the involvement of industry.29 With support from mentors and collaborations with the country’s leading technical and management institutions integrated in the project, autonomous institutions also improved their capabilities in research and innovation. d. Strategic interventions involving partnerships with leading Indian institutes led to student and teacher development programs (discussed in detail in the next section) like blended MOOCs, building networks between institutions to share pedagogy, content, testing methods, trends in different disciplines, student exchange, winter/summer schools and so forth. These interventions are likely sustainable and will continue to enhance the quality of project institutes. e. Employability: All institutes were required to set up placement cells and track placement rates over the project period. Placement rates and activities were reviewed in JRMs, but due to differences in methodology adopted by colleges in calculating the rates, there is no standardized number in the early years of the project. The methodology was standardized in 2015, and was subsequently used in a tracer study30 of graduates from TEQIP II institutions in 2016. The study found that most placement cells offered a variety of pre-placement and placement-related services (personality development workshops and soft skills training, career guidance and campus interviews and informational sessions by employers and alumni). About 75% of graduates surveyed in the tracer study were employed; and over 60% responded that they found employment through campus placements. This is significantly higher than a 2015 survey of 27,000 engineering graduates across the country, covering different engineering disciplines and institution types 28 About 30% of project institutes did not acquire autonomy as indicated in para #24. The reasons were related to delays on the part of University Grants Commission and in some cases, weak state support. 29 Data from the project MIS suggests that upwards of 2000 curricula were revised with the involvement of industry and nearly 700 by institutions themselves. These figures are not available by institution and reflect curriculum revisions at the program level. 30 The tracer study surveyed students who graduated from TEQIP institutions in 2013-14 and 2014-15. About 5600 students from 116 institutions responded to the survey. The number of respondents constitutes ~2-3% of the number of expected graduates each year (~100,000). The study, like all tracer studies in education, faced challenges as graduates had no incentive to respond and sometimes their contact details had changed. Institutions from 60 percent of the project states responded to the survey (this does not include states with very low response rates). The sample sizes and response rates thus warrant some caution in interpretation. Page 17 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) which reports an employment rate of 20-30% depending on tier of college.31 Data from the MIS submitted by project institutions provides evidence of improvements in annual salaries offered to undergraduate students -- the average annual salary for UG students increased by nearly 20 percent during the project period. Student satisfaction32 with placement activities and institutional efforts to improve employability also showed some improvement over the project period. At the end of the project students rated their satisfaction with these activities at 3.8 on a 5-point scale (up by 0.1 points the previous year). An employer satisfaction survey conducted by the World Bank and FICCI in 2009-10 had found that only 33% of employers were satisfied with newly recruited engineers, with soft skills being a key area of discontent. As a result, TEQIP II laid considerable emphasis on building soft skills. A repeat survey in 2014-15 found that 75% of employers were very satisfied with newly recruited engineers with the key area of improvement being soft skills.33 f. Reduction in faculty shortages: The project aimed to address faculty shortages by attracting more students to post-graduate programs thus making academia a viable option for these students. There was a significant increase in postgraduate engineering participation during the project period. More time will be needed to assess how this impacts faculty shortages. The project also required institutions to recruit adequate faculty to participate in the project. The faculty to student ratio over the course of the project went from 1:17 in 2010 to 1:14 in 2015-16. The number of faculty at TEQIP institutions increased, on average, by about 40 percent – with some states showing phenomenal progress (Himachal Pradesh, Haryana, Odisha) while some states like Bihar, Tripura and Gujarat still face substantial shortages. 30. Variations in outcomes across states and institutions: There was some variation in achievement of outcomes across states. a. Of the PDO indicators, the largest variation was observed in the percent of institutions with academic autonomy and share of accredited/applied for programs. States like Maharashtra and West Bengal showed proactivity in effecting these reforms and strengthening their SPFUs. Institutions in these states, even those not part of TEQIP, benefitted and made progress towards improving quality. On the other hand, progress on accreditation in Punjab, Haryana, Rajasthan and Telangana was slower. Faculty-student ratios are a key component to achieving accreditation and in states that could not appoint adequate faculty (either because of state finances or court cases), programs could not apply for accreditation. Appointment of faculty is a complex and political economy issue which was outside the scope of the project. Among states identified as lagging in technical education, most exceeded the project target for accreditation/applied for and met 90 percent of the target for percent of faculty with an M.Tech degree. b. Responses from the tracer study and supervision missions showed that states and institutions built strong linkages with industry with varying degrees of success. States with a considerable industrial base (such as Maharashtra and Karnataka) benefitted from industry linkages but states with little industry such as Rajasthan, faced limitations. As a consequence, summer internship and employment opportunities for undergraduate students in these states were limited. 31 Aspiring Minds (2015): http://www.aspiringminds.com/sites/default/files/National%20Employability%20Report%20- %20Engineers%20Annual%20Report%202016.pdf 32 Three rounds of student and faculty satisfaction surveys were conducted during the last 1.5 years of the project. Although an initial round had been conducted in July 2014, the data were found faulty and the entire process had to be restarted. Decisions on vendor selection took a while, and it was ultimately decided that NPIU would implement the survey in the interest of time. 33 The survey included TEQIP and non-TEQIP institutes. The qualitative part of the survey indicates that several employers recruit from TEQIP institutes and supervision missions during the project period suggest similar findings in terms of improvement of soft skills. Page 18 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) 31. Link between outputs and outcomes: It is highly likely that the project contributed significantly to the PDO outcomes achieved by participating institutions. The project spearheaded reforms at the state and institution level, leading to behavior change at both levels. Consequently, the resulting outcomes are attributable to the project. As part of their MoU with the national government for participation in TEQIP II, states committed to supporting academic autonomy and granting institutional leaders financial discretion, leading to long-term changes in the way institutes functioned. This included authorizing filling up of faculty vacancies through transparent mechanisms, enabling formation of a BoG and supporting academic and non-academic reforms in project institutions. This led to institutions with more teachers, training and placement cells, revisions to curriculum, and activities for student and faculty development which contributed to improvements quality. The role of IITs and IIMs was also conceived of as part of TEQIP II to build bridges between institutes. In the absence of these partnerships the scale and quality of faculty development programs, student internship and learning activities would have been much more modest. Better infrastructure at project institutions, better qualified teachers, industry collaborations, new postgraduate programs, scaled up R&D activities through Centers of Excellence and effective institutional leadership through the Board of Governors all contributed to strengthening institutions and improving the quality of engineering education provided. Justification of Overall Efficacy Rating 32. As per the latest guidance for ICRs, the overall efficacy rating considers progress on PDO indicators and higher- level objectives. However, it must be noted that the project was designed with institutional strengthening as the development objective and clearly states in the PAD that impact on learning and employment outcomes would take a few years. The project achieved, and even exceeded, the PDO indicators identified for the project period. Based on this assessment alone, the efficacy of the project would be rated as ‘High’. Efficacy is also rated as ‘High’ when higher-level objectives are accounted for. Considerable progress was made towards higher-level objectives; there is adequate evidence that mechanisms are in place for improving quality of institutions over the long term (programs like QEEE, networking of institutions) and it is highly likely that higher-level objectives will be achieved. RATING: HIGH C. EFFICIENCY Assessment of Efficiency and Rating 33. The efficiency of the project is assessed in terms of internal and external efficiency indicators. Internal efficiency varied across project institutions but increased overall due to many of the outcomes achieved through the project -- increase in the number of faculty and share of those with higher credentials, higher skills among faculty from training and development activities, better facilities and resources at project institutions for teaching and learning and R&D activities. 34. A study on the utilization of resources showed that by February 2017 most institutions had purchased laboratory equipment, software, books, office equipment and so on utilizing 75 percent or more of their planned expenditure in keeping with their procurement plans.34 On average, resources were divided equally across departments which allowed for uniform upgradation of departments, giving them equal opportunity to attract students, faculty and industry assignments. Collaboration between departments, however, was not as evident. The resources purchased 34 In Punjab some institutions reported lower utilization (about 52 percent). Reviews during each Joint Review Mission (JRM) and discussions with the state government were carried out to address this issue. Page 19 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) under the project have likely contributed to the increase in the number of publications in refereed journals, more post- graduate students and more faculty pursuing masters and PhD degrees in project institutions. 35. Efficiency in project implementation and management was enabled using technology. Procurement under the project was decentralized and enabled through an E-Procurement system which was accompanied by trainings to institutions and a dedicated helpline to ensure that procurement activities proceeded smoothly, which was a key innovation across procurement activities in South Asia projects.35 The web-based MIS provided real-time information on implementation and progress and enabled setting up a performance-based benchmarking system which was used to incentivize project institutions towards better performance. 36. External efficiency of the project is estimated by an Internal Rate of Return analysis detailed in Annex 4. The results compare favorably with the prospective IRR at the time of appraisal. The rate of return estimated is substantially higher from that estimated at appraisal. This is driven by (i) higher actual number of student beneficiaries (150,000 versus ~360,000) and (ii) lower costs in USD due to savings from shifts in exchange rates (USD 250 M versus USD 192 M). The estimated IRR ranges from 7% to 72% under different scenarios. In the base (or moderate case), the project shows high returns (between 40-50%). This rate of return only considers benefits in terms of better labor market outcomes among undergraduate students and does not take in to account other benefits accruing to faculty, institutions and society. Rating: HIGH D. JUSTIFICATION OF OVERALL OUTCOME RATING 37. The overall outcome rating of the project is assessed both, with and without considering revised objectives. Under both scenarios, the overall outcome is Highly Satisfactory based on high relevance, high efficacy and high efficiency. The overall outcome rating with revised objectives is calculated using the split rating method. See Section E below. Without Split Rating: HIGHLY SATISFACTORY With Split Rating: HIGHLY SATISFACTORY E. RATING WITH REVISED OBJECTIVES 35 The procurement innovation is documented at: http://globalpractices.worldbank.org/governance/PracticeBlog/Lists/Posts/Post.aspx?List=fcb0ea05-27f9-42d9-b830- 5001fcb6d430&ID=48&Source=http://globalpractices.worldbank.org/governance/Pages/en/ManageSites.aspx?Category=C ommunications Page 20 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) 38. As detailed in Section I B #20, modifications to the indicator in January 2014 measuring share of programs accredited including expanding the definition to include programs that had applied for accreditation.36 A split rating has been applied in this case because the ICR does not have evidence on the proportion of institutions that get accredited at the end of the application process. Table 5 shows target and actual values for original PDO indicators before restructuring. In Year 3, the project had not made any progress on 1 indicator and achieved 27% of the target on accreditation. The project had almost met or exceeded remaining targets. 39. In Table 6 the split rating method is used. The outcome is rated before and after restructuring and weighted by the proportion disbursed at each stage. Based on this calculation the final outcome score is 5.76, which is rated as ‘Highly Satisfactory’. Table 5. Target and Actual Indicator Values for Original PDO indicators – Status as of January 2014 Target for Actual for % of target Indicators Year 3 Year 3 achieved Employer satisfaction index 3.4 - 0 Share of programs that are accredited 45% 12% 27 Percentage of institutions with academic autonomy 60% 59% 98 Percentage of Faculty with at least an M.Tech 55% 70.30% exceeded No. of master and PhD students enrolled 20000 19492 97 Table 6. Split Rating (TEQIP II) Up to Jan 2014 Feb ’14 to Mar ‘17 Relevance High Efficacy Substantial High Employer satisfaction index Negligible - Share of programs that are accredited Modest High Percentage of institutions with academic autonomy Substantial High Percentage of Faculty with at least an M.Tech High High No. of master and PhD students enrolled Substantial High Efficiency High Outcome S HS 37 Numerical outcome 5 6 Share of disbursement 24% 76% Weighted value 1.2 4.56 (Numerical outcome x Share of disbursement) Highly Satisfactory Final Outcome Rating 0.96 + 4.56 = 5.76 (rounded to 6) 36 nd Modifications to the indicators as part of the 2 Restructuring in September 2014 significantly stretched targets on most indicators being measured. These changes effectively increased the project’s ambition. 37 1= Highly Unsatisfactory, 2=Unsatisfactory, 3=Moderately Unsatisfactory, 4=Moderately Satisfactory, 5=Satisfactory, 6=Highly Satisfactory Page 21 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) F. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 40. The project’s impacts on gender and social development were operationalized through Equity Action Plans. Institutes made special efforts to provide girl students and students from disadvantaged social groups access to resources and services to help them cope with their education as described in Section II B, #32. Thus, transition rates between the first and second year improved substantially among all students including girls and particularly among those from disadvantaged backgrounds. Female students constituted 30 percent of all student beneficiaries in the project – this is slightly higher than the national average (28 percent).38 Institutional Strengthening 41. The project’s contributions towards institutional strengthening have been discussed in detail in the previous section (see discussion in para #29(a-d)). The project also contributed to institutional strengthening through financial sustainability. Participating institutions were required to set up funds in four areas that could be used to continue reform activities and sustain gains made during the project period. Of the 190 participating institutes, 183 (95%) established four funds -- Corpus Fund, Faculty Development Fund, Equipment Replacement Fund and a Maintenance Fund -- with substantial resources in each. At the end of the project period, the corpus fund had accumulated nearly US$ 150 million. Mobilizing Private Sector Financing NA Poverty Reduction and Shared Prosperity 42. The project had direct and indirect impacts on poverty and shared prosperity. Supporting institutions in states lagging in technical education39 was a secondary objective of the project. 24 percent of project institutions came from the 19 states/UTs identified as lagging in technical education. States identified as low-income40 were also included in the project and 19 percent of project institutions belonged to these states. Further, the project’s focus on providing additional focused support to first time college goers and students from disadvantaged backgrounds (primarily SC/ST) aimed at improving learning outcomes and employability for this group, thus contributing to poverty reduction and shared prosperity. In the long term, these efforts contribute towards sustaining the country’s economic growth and increasing competitiveness by improving the quality of engineering education and producing higher skilled engineers. Other Unintended Outcomes and Impacts 43. The basic approach of the project, which sets criteria for participation and promotes reform at the institutional and policy level is reflected in other important programs. The Government of India’s national higher education program (Rashtriya Uchchatar Shiksha Abhiyan), launched in 2013 uses outcomes-based funding and focuses on governance reforms to improve quality and relevance across the higher education sector in India. The Good Practice Guide for 38 The All India Survey of Higher Education, 2015-16 39 ‘Lagging in technical education’ implied that these states had 1 engineering institution per million population, as per AICTE’s approved list of engineering degree institutions in 2004. 40 Bihar, Chattisgarh, Jharkhand, Rajasthan and Uttar Pradesh were identified as lagging in technical education and are also low-income states. In addition, Madhya Pradesh and Odisha, low-income states, also participated in the project. Page 22 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Governing Bodies published under the project provides a roadmap on implementing governance reforms at the institutional level and has been disseminated widely among internal and external audiences.41 The design of WB- supported projects for higher education quality improvement within and outside India have also been informed by the TEQIP experience. This includes projects with two low-income states in India – Madhya Pradesh and Odisha – and projects in Sri Lanka and Bangladesh. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 44. TEQIP was envisioned as a series of investments over a 15-year phased period aiming to improve engineering education in India. A Sector Specific Investment Loan (SIL) was the preferred lending instrument over the Adaptable Program Lending instrument or a Development Policy Credit because, (i) it could support policy reforms that affect productivity of the investment, (ii) it offered the option of scaling-up operations via additional financing or another SIL, and (iii) it allowed for embedding hands-on technical assistance and monitoring in the project. 45. Evidence-based design - The technical design of the project drew from 4 globally-recognized approaches for improving the quality of higher education and these were adapted to the Indian context - governance reforms, competitive funding, simple design squarely focused on strengthening selected institutions to improve quality of engineering education, and leveling the playing field through inclusion of states and institutions lagging in technical education. These design elements were well-suited to most institutions participating in the project. 46. Appropriate selection of beneficiary groups to target - Institutions from low income states and those at the lower end of the competitive selection process likely faced different challenges in terms of capacity and quality. The project design included additional implementation support for these institutions. But given their low base and varied challenges, design could have tailored the effort and expectations from these institutions differently anticipating a slower rate of progress. A recently negotiated higher education project in a low-income state in India has adopted this approach. 47. Appropriate selection of stakeholders to engage – TEQIP II leveraged and built key networks with top- ranking engineering and management institutes in the country (IITs and IIMs). These institutions were engaged by way of initiating a range of quality and governance improvement efforts within project institutions. Involvement in the project has also enabled a greater understanding of the sector among the IITs and IIMs and challenges faced by most institutions. These networks have played a critical role in achievement of objectives in TEQIP II and can also be key to sustaining efforts under the project. 48. Appropriate plan for monitoring - The design benefitted from over 10 Bank-financed competitive funds in higher education and from lessons from TEQIP I. Experiences from TEQIP I highlighted the need to build management capacity, incentivize good performance, improve efficiency of the M&E system, and strengthen fiduciary capacity and oversight. The project designed specific activities for capacity building like funds focused 41 http://www.teqipgoodgovernance.in/TEQIP%20GOOD%20PRACTICE%20GUIDE%20FOR%20GOVERNING%20BODIES_DEC%2 02012.pdf Page 23 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) on state governments and affiliating universities and funds for sharing good practices across institutions and states. To incentivize better performance institutes received rewards and sanctions linked to bi-annual performance benchmarks. Implementation arrangements were enhanced – an MIS each for education and procurement was acquired, roles and responsibilities were clarified at all levels and NPIU was significantly strengthened. Several assessment, studies and surveys to track, analyze and assess progress were incorporated in M&E. These could be further strengthened in 2 ways – gathering data to assess effectiveness of activities and building capacity at institutions to collect, analyze and use these data. 49. Adequacy of risk and mitigation measures identification - Key risks and appropriate mitigation measures were adequately identified during preparation. Challenges in the form of political and bureaucratic interference were anticipated to some extent given the ambitious reforms-based nature of the project. This was especially true in the case of obtaining autonomy. Resistance of state governments to autonomy came from equating autonomy with privatization of institutions. Although some mitigation measures were identified, more detailed discussions with state governments sooner in the process might have better mitigated this risk. There were also some shortcomings on fiduciary, governance (lack of decentralization and oversight especially in the case of lagging states) and capacity risks identified at preparation. B. KEY FACTORS DURING IMPLEMENTATION 50. Participation in TEQIP II was voluntary – Interested states first had to meet specific eligibility criteria and institutions in selected states were then competitively selected in the project. About 500 institutional proposals were received across both funding windows – which necessitated detailed evaluation of proposals. MHRD and NPIU maintained an objective, merit-based and transparent selection despite substantial pressure from states and institutions to lower eligibility (reform) criteria. However special consideration was given to low- income states and some eligibility criteria were relaxed for institutions in these states. 51. Human resources and organizational capacity - The pace of implementation was of significant concern during the first 2 years and ‘Implementation Progress’ was rated as ‘Moderately Unsatisfactory’ in 2013. The delays were mainly due to: (i) delays in institution selection42 (ii) inadequate staffing and capacity of NPIU and SPFUs, (iii) lengthy procurement process for the academic MIS, (iv) technical difficulties with MIS, PMSS and e- FMR, (v) delays in flow of funds, which remained a problem through the project, and (vi) lack of incentives and vision in the case of some project institutes. 52. Fiduciary - Fund flow and disbursement delays affected implementation practically throughout the project. Fund releases from state treasuries to institutions took inordinately long – over 100 days in many cases and even 300 days in a couple of cases. Some delays in disbursement were expected given the highly- decentralized nature of the project which required selecting and training decentralized units, building systems and ensuring timely release of funds. The reasons for delays in fund flows varied by state, but in general pertained to constraints in the state treasury. Recommendations to address fund flow issues included ensuring that provisions for expenditures under TEQIP II were adequately reflected in the State Budget each financial year. In addition, a system of monthly monitoring of fund flows at all levels by NPIU was initiated. Page 24 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) 53. Monitoring and evaluation & Adequacy of supervision - The project made a significant and commendable turnaround on implementation owing to the institutional performance assessment system and proactive and enhanced supervision of institutions by NPIU. Performance benchmarks with technical and operational targets43 were set every 6 months and the performance of institutions against these benchmarks was tied to rewards and sanctions. This innovative incentive mechanism resulted in performance improvements as evidenced by the increase in the number of “well-performing� institutions. Institutions were not only motivated by the promise of additional funding but also the potential risk to reputation if associated with lower performance among a group of their peers. Well-Performing Institutions 2nd JRM (Meeting 7 or 8 indicators) 67% 3rd JRM (Meeting 9 or 10 indicators) 75% 4th JRM (Meeting 8 or 9 indicators) 87% 5th JRM (Meeting 8 or 9 indicators) 87% 54. To accelerate implementation progress at the institutional level BoGs were delegated powers and decision making authority to carry out project activities. This worked in many cases, except in those institutions that faced state government interference and/or had passive, ineffective BoGs. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 55. The project had multiple objectives articulated in a compound PDO statement. The PAD clearly identified the main objectives of the project and higher-level objectives to which the project would contribute in the long term. The objectives were ambitious but realistic and supported by a clear Results Framework that was developed after extensive stakeholder consultations44 and reflected the project’s theory of change. 56. Discussions with stakeholders and experts led to identifying relevant and specific PDO and IO indicators. The use of accreditation and autonomy as indicators of institutional strengthening through governance reforms and as indicators of quality improvement has precedence, and have been used in other higher education projects.45 Further, as detailed in footnote 14 and Section I B (#19), accreditation of engineering programs in India is based on, among other parameters, institutions demonstrating improvements in learning and employment outcomes and processes in place to improve the same. The use of accreditation however posed some challenges during implementation which were not fully anticipated during stakeholder consultations (see footnote 14 and para 19 for a discussion of the challenges). 57. The project defined several IO indicators which, in general, adequately captured activities relevant to each 43 Technical targets included key performance indicators like University Grants Commission (UGC) autonomy and accreditation, publication of BoG minutes; and operational targets included expenditures as a percent of releases, data input into MIS and completion of statutory audit. 44 Stakeholders included state governments, representatives from institutions and the private sector. 45 Retrieved from https://openknowledge.worldbank.org/bitstream/handle/10986/26486/113867-WP- PUBLIC.pdf?sequence=5&isAllowed=y Page 25 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) component and linked these to overall project outcomes. Most of the IO indicators were clear and measurable and this allowed for stretching target values midstream to accommodate further progress during the project period. As discussed in Table 1, in the case of two IO indicators, the method of calculation was not clear and these indicators were dropped as part of the first restructuring. 58. Data on progress was collected and entered by project institutions in a web-based MIS – used for the first time in an education project in India. Institutions could use this data for monitoring and decision-making, and the state and NPIU to monitor performance. To avoid technical difficulties, the system was revised to accommodate uploading Excel files to the MIS. The M&E design included checking the quality of data in at least two ways – regular and frequent verification by data auditors and verification by mentors and performance auditors during their visits to institutions. 59. Apart from a system for continuous tracking of outcomes, the project’s M&E design included various qualitative and quantitative assessments and studies to help understand project performance and enable informed decision- making. This included the following – a. Student and faculty satisfaction surveys – three rounds of the survey were planned during the project period to gather information on their satisfaction with services and support provided by institutions b. Review of mentors and performance auditors – designed to get feedback from participating institutions on the usefulness of mentorship and performance audit processes, and suggestions for improvements c. Interim impact assessment – to help monitor progress and identify areas for targeted support and capacity building d. Impact assessment – to evaluate the impact of the project on stated objectives. This was planned to be carried out towards closure of the project e. Bibliometric study – to assess research output and performance in project supported institutions f. Employer studies – to gather feedback from employers on the employability and quality of graduates. M&E Implementation 60. As institutions started generating data on outcomes, it became evident that some indicators were not accurately capturing progress and needed modification to better measure achievement in the project. Modifications made to indicators during the project period are detailed extensively in Section I of this ICR. These modifications did not limit the assessment of achievement of PDO. Further, the indicators on accreditation/applying(e-SAR) and autonomy could also be used to assess the project’s progress towards the longer-term goal of quality. For employability, which is often hard to track and attribute causality to, the project carried out a survey of employers and a tracer study of graduates to gather progress on this longer-term objective. 61. Data auditors were responsible for verifying data in the MIS, and in general, the quality of the data was acceptable. However, the audit process was put on hold for about 6 months when some inconsistencies were observed in the case of a few indicators. The data auditors were thoroughly evaluated and several were let go. Data from the MIS was used extensively at the national level to monitor progress and was regularly reported in the ISRs. At the state and institutional levels however, data use was modest. This was partly due to limited capacity and lack of incentives. Introduction of the performance benchmarking system addressed the lack of incentives and as a result, use of project data at the institutional level to improve results increased. In cases where states/institutions have shown proactivity, for instance in West Bengal one BoG meeting is exclusively for discussion on M&E, not only has this oriented the state and institutions towards outcomes but has also promoted better understanding between the state and BoG. Data from the project will be migrated to the MIS system for TEQIP III leading to sustainability of some M&E functions. Page 26 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) 62. Despite small delays, the project collected additional information as planned. Work on the satisfaction survey was initially contracted to a survey firm and a first round of survey was completed in 2014. But the contract was terminated thereafter due to default on contract obligations. NPIU took on the responsibility of the satisfaction surveys and used SurveyMonkey to develop and administer 3 rounds of the survey. This also enabled NPIU build capacity in survey work. Findings from the satisfaction survey and other studies provided important insights on project needs and performance. For instance, findings from the interim impact assessment suggested the need for continuous training and capacity building on fiduciary and M&E aspects and the tracer study showed that majority of the graduates pursued employment over higher education which had important implications for student activities under the project. The project had also planned for and commissioned a final impact assessment; preliminary findings from which were available after project closing. M&E Utilization 63. Progress on project indicators and findings from various assessments and surveys were shared and discussed during Joint Review Missions and reported in all ISRs. Data from the project was used not only to evaluate progress but highlight lessons learned and in meetings with the NBA and UGC to discuss the status of accreditation and autonomy in TEQIP II institutions. Findings from the satisfaction surveys were shared with project institutions in the form of institutional report cards. Utilization of M&E information at the state/institutional level was modest. 64. Data from the web-based MIS was a key enabler of the innovative performance benchmarking system introduced during the project. Institutional data on technical indicators regularly updated in the MIS were used to identify areas where the project was lagging and where focused efforts were required. The performance-benchmarking system brought about a notable improvement in institutional performance during the project period. Justification of Overall Rating of Quality of M&E 65. Design, implementation and utilization of M&E in the project served the purpose of systematically tracking project progress and providing relevant information for midstream corrections. There were some shortcomings in design and implementation which were addressed without significant delay. The project gathered substantial amount of data which was analyzed and utilized in ways that ultimately improved overall project performance and provides sufficient evidence to assess achievement of objectives. The quality of M&E is thus rated as Substantial. Rating: SUBSTANTIAL B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 66. The Project triggered 2 safeguards policies – Indigenous Peoples (OP/BP 4.10) and Environmental Assessment (OP/BP 4.01). The safeguard policy on Involuntary Resettlement (OP 4.12) was not triggered as there was no land acquisition or requisition as part of the project. 67. On the social safeguards, the project implemented an Equity Action Plan (EAP) to ensure that the needs of students and teachers belonging to tribal communities and other vulnerable groups are adequately addressed and they have equal opportunity to avail the benefits of the project. All participating institutes prepared an action plan outlining ways in which various social risks to the project could be addressed. The plans focused on improving learning and employment outcomes of students from disadvantaged backgrounds, improving gender Page 27 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) friendliness of campuses, providing teachers equal opportunities for development activities and so on. 46 Monitoring indicators were used to assess implementation of various activities in the EAP. 68. No adverse environmental impacts were anticipated from the project. The project was assigned Risk Category B because of the nature and magnitude of potential environmental impacts from academic research works and refurbishment and renovation activities, although these were limited. An Environment Management Framework (EMF) was prepared and integrated with the ‘Refurbishment and Renovation Activities Operational Manual’ to ensure that various environment management aspects were mainstreamed in sub-project activities. This format provided a simplified but comprehensive coverage of issues and was a huge departure from TEQIP I. It was also the first time this was done for an education project in India. Four regional workshops were organized with institutions and SPFUs to discuss implementation of the EMF under the project. These workshops revealed many good practices including relevant environment research being undertaken at some CoEs. However limited efforts towards documenting these good practices (either by institutions themselves or SPFUs and NPIU) hindered dissemination of this information across all project institutions and the potential benefit it may have had. 69. Financial Management: The design of FM arrangements in TEQIP II were informed by TEQIP I and focused on how the decentralized nature of the project could best be accommodated. The project moved to an electronic-FMR system for implementation at institutions for efficient reporting and consolidation of reports. Other arrangements included provisions to enhance disclosure information, a dedicated senior officer responsible for FM in NPIU, strengthening the internal and external audit system, a single audit report for the entire project and report based disbursements. Further, to ensure stronger state commitment to the project the fund flow mechanism was modified to have funds flow through the State Treasuries. 70. Delays in releases of funds from states to institutions and inadequate budget provisions for TEQIP II in some states were major and persistent challenges during the initial stages of implementation. FM capacity at the institution and state levels was also an issue. For instance, the e-FMR system designed for the project could only be partially implemented due to high turnover of FM staff in project institutions. This made using excel- based reporting more viable. Thus excel-based FMRs from institutions formed the basis of state-level FMRs. 71. Despite significant delays and challenges, the project disbursement rate is nearly 100 percent (98.9%). This is based on expenditures incurred before project closing. As noted in paragraph # 17, funds were cancelled due to substantial shift in the exchange rate. Thus, the amount available after cancellation was SDR 134.7 million of which the project disbursed SDR 133.1 million. 72. Procurement: An electronic procurement system – PMSS – was introduced to bring about greater efficiency, transparency and clarity in conducting the decentralized procurement activities required under the project. Institutions valued being able to see what other institutions were procuring. NPIU provided significant support in the form of frequent training and workshops, and prompt on-line help and resolution of queries related to PMSS. The system went offline for about 5 months because the vendor supporting the PMSS system was declared bankrupt. This led to a stalemate in procurement activity during a crucial period of the project (close to the project’s closing date and the academic cycle of institutions). MHRD allowed institutions to temporarily use the manual procedure for procurement until the system was functional. 46 http://blogs.worldbank.org/education/most-effective-services-well-rounded-higher-education-students-india Page 28 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) 73. The team responsible for procurement at the central level, EdCIL, facilitated administrative tasks for NPIU. There were significant delays in NPIU’s administrative processes due to numerous approvals required by EdCIL and MHRD, and a lack of clarity on which tasks required approvals. Several discussions between MHRD and EdCIL on streamlining the approval process and prioritizing delayed and pending approvals however did not fully resolve this issue. Subsequently some powers were delegated to NPIU to carry out limited and well-defined administrative tasks (like minor purchases and consultant contracts, limited staff travel, low cost meetings and workshops) without prior approval from EdCIL and MHRD. C. BANK PERFORMANCE Quality at Entry 74. Bank performance at entry is based on the steps taken during identification, preparation and appraisal to ensure achievement of the PDO – a) The PDO was highly relevant and remained so throughout the project period. The objectives were strongly aligned with country and sector priorities as articulated in the Government’s Five Year Plan and the Bank’s CPS. Design of the project was based on extensive consultations with and participation of stakeholders, incorporating learnings from the previous phase and successful practices in higher education projects in other countries. b) The results framework was comprehensive and indicators were closely linked to main objectives of the project. The formulae for calculating some IO indicators and validity of the employer satisfaction index could have been more clearly agreed upon during preparation. Although these indicators were dropped, the project was able to use alternate measures of employability, like employer surveys, to gather information on employability. c) Change in fund flow mechanism from TEQIP I for greater state commitment seemed like a good idea but did not work as planned. Options for fund flow were deliberated upon during preparation but perhaps more active discussion on state finances earlier in project cycle might have helped anticipate and/or identify ways to address possible issues. Quality of Supervision 75. The quality of supervision provided by the Bank team was proactive and effective. This was evident not only during the initial period of slow progress but even subsequently when the project picked up pace and was on track to meet outcomes. The Bank team established a “strong working relationship with MHRD and NPIU� and were looked up on by the implementing agencies as an “important resource� guiding achievement of outcomes.47 Aspects demonstrating high quality of supervision include – a) Regular and well-planned supervision missions representing specialists from education, procurement, FM, environment and social included visits to participating institutions, exchange of best practices and challenges faced by institutions, setting of performance benchmarks to motivate progress towards technical and operational targets, and discussions with system level bodies like AICTE and NBA to address systemic challenges impeding implementation and capacity building for reforms. b) Effective and timely support from FM and Procurement to build capacity of implementing agencies through multiple regional workshops and handholding. c) Detailed and candid AMs and ISRs reflecting the task team’s keen monitoring and focus on solutions, with changes to implementation ratings to flag aspects that needed focused attention from implementing 47 Direct quote from interviews with members of the National Project Directorate. Page 29 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) agencies. d) Incentivizing better performance through benchmarking of institutions on the basis of technical and operational indicators which enabled higher achievement than originally targeted under the project. End targets for several indicators were stretched as part of the restructuring in September 2014. e) Seamless continuity of supervision without deterioration in quality during transitions in project leadership. Justification of Overall Rating of Bank Performance 76. Based on these assessments, overall Bank Performance is rated as Satisfactory. Rating: SATISFACTORY D. RISK TO DEVELOPMENT OUTCOME 77. The risk that development outcomes will not be maintained is rated as ‘Low’ based on the following assessments – a) The government’s investment in TEQIP III and priority accorded to technical education in the New Education Policy48 is evidence of the country’s continued commitment to improving quality of engineering education. States and institutions participating in TEQIP III will continue and strengthen reform activities. Further, the same NPIU will continue to provide support to and monitor participating states and institutions. b) Sustaining academic and faculty development activities will require strong institutional leadership and commitment with sufficient funds being made available for these activities. This will also require the state leadership’s continued commitment to and support of academic autonomy at the institutional level. c) TEQIP II aimed at building and strengthening sustainable governance structures and processes at the institutional level, which along with investments in equipment and facilities will pay off for several years. Establishment of the BoGs to oversee and guide institutional development, generating internal revenue, creating and maintaining funds for continuous improvement and sustaining gains after project closing are some examples. d) Private colleges (affiliated to universities) make up over 80% of the engineering education sector and therefore any effort to enhance the quality of engineering education in the country must include private institutions. State governments can be reluctant to extend public funds to private institutions. In TEQIP III private colleges will benefit from inclusion of some Affiliating Technical Universities (ATUs) in the project. Risk Rating: LOW V. LESSONS AND RECOMMENDATIONS 78. Measuring student learning is central to the measurement of quality in higher education. The project 48 Draft inputs to the New Education Policy highlight several governance reforms and activities implemented in TEQIP II for the higher education sector including – autonomy and accountability, mandatory accreditation, faculty development, performance-linked funding, promotion and regulation of MOOCs a and so on. The draft can be accessed here: http://mhrd.gov.in/sites/upload_files/mhrd/files/nep/Inputs_Draft_NEP_2016.pdf Page 30 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) used proxies to measure student learning (accreditation, transition rates) and assess quality of institutions. There were two key reasons – there is tremendous variation in examination systems across states and years which make comparisons difficult. Second, designing a credible assessment system and administering it in an environment where students suffer from considerable test anxiety is difficult. While the proxy measures used showed positive movement, valid and standardized student assessments are needed to measure improvements in student learning and use this to improve performance of institutions. Stanford university, in collaboration with ETS, has developed a tool to measure student learning gains in engineering. This assessment will be used in TEQIP III to compare institutions across India and internationally (Russia, China, Brazil and the United States). 79. Quality improvement requires behavior change at different levels. Autonomy and institutional leadership, key mechanisms for institutional reform, worked well in 87% of institutions but not so well in the remaining 13%. There are three key reasons that explain why some institutions faced difficulties achieving and exercising autonomy – (i) limited capacity and/or responsiveness by the UGC in organizing visits to institutes meeting all criteria, (ii) limited initiative by some autonomous institutes to take advantage of autonomy, and (iii) weak state support in some cases. Weak state support could potentially limit the effectiveness of autonomy. The most visible of these reasons pertains to not complying with the precondition pertaining to the independence of the Board of Governors in colleges. 80. Higher education projects can benefit from technical assistance to strengthen country quality assurance systems. Institutions applying for accreditation of programs experienced significant delays during the application process. These delays were partly a consequence of change in accreditation procedures and limited supply of qualified accreditors. Further, as more programs applied for accreditation (as part of their participation in TEQIP II), the capacity of the system was inadequate to meet the additional demand. Integrating capacity building of quality assurance systems in higher education projects, where accreditation is an often-used metric for quality, is important while weighing potential conflicts of interest. 81. Expanding and sustaining student and faculty development activities requires capacity building. Among the academic reforms implemented by institutions were activities like internships and research opportunities for students and trainings for faculty. These activities benefitted few students and institutions did not make systematic efforts to network students or sustain efforts to multiply the benefits. Capacity building that enables institutionalization of these activities can be incorporated in projects while also ensuring that institutional capacity to evaluate these activities and use the resulting information for decision-making is strengthened. 82. Building successful institute-industry collaboration requires a strategic approach and time. Institutions with good industry outreach were notable in how they went about establishing these relationships. These institutions were often located in dense industry areas and initially took up any industry project or problem that came their way. This helped build the trust required for more and fruitful engagements with the industry. Initially, the focus was on ways in which these institutions could benefit the industry and deliver solutions to industry problems. Once the relationship was built, institutions primarily focused on initiatives that also added value to students and the institution. Thus, time, effort and patience of the part of institutions initially led to strong relationships once industry was convinced of the value in the partnership for them. Page 31 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) 83. Sectoral reform in the engineering sector will need to include private institutions. Private colleges make up over 80% of the sector. Improvement in quality of the sector and sustainability of reforms will thus have to include these colleges. In TEQIP III private colleges will be benefitted through their affiliation with ATUs. . Page 32 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Institutional strengthening to produce more employable and higher quality engineers Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Share of supported programs Percentage 30.00 50.00 55.00 65.00 that are accredited or applied for 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Comments (achievements against targets): This indicator was modified in January 2014. The project achieved 118% of the target at closing. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of institutions Percentage 30.00 65.00 70.00 70.00 with academic autonomy 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Comments (achievements against targets): The project achieved 100% of the target at closing Page 33 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage Faculty with at Percentage 45.00 60.00 88.00 92.35 least an M. Tech (regular and contract) 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Comments (achievements against targets): This indicator was modified in January 2014 to include regular and contract faculty. At closing, achievement was at 105% of the target. Objective/Outcome: Producing more graduate students to reduce faculty shortage Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Total number of Master and Number 30000.00 34000.00 41000.00 45658.00 PhD students 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Comments (achievements against targets): This indicator was modified in January 2014. The project achieved 111% of the target at closing. A.2 Intermediate Results Indicators Component: Improving Quality of Education in Selected Institutions Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of Faculty with or Percentage 63.00 73.00 90.00 93.45 Page 34 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) pursuing M. Tech and PhD 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 (regular and contract) Comments (achievements against targets): The project achieved 100% of the target on this indicator. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increase in the satisfaction Number 3.90 0.00 4.30 4.00 index of student and faculty 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Comments (achievements against targets): Achievement on this indicator fell just below the target at closing. The project achieved 93% of the target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Transition rate of students Percentage 45.00 55.00 60.00 61.45 from disadvantaged backgrounds from the first 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 year to second year of undergraduate study Comments (achievements against targets): The project achievement on this indicator exceeded the target by 1.45 percentage points – 102% of the target was achieved. Component: Improving System Management Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 35 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Target Completion Number of governance self Number 0.00 80.00 190.00 186.00 reviews received 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Comments (achievements against targets): The project fell just under the target for this indicator – achievement was at 98% of the target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of Governance Number 0.00 20.00 180.00 186.00 Development plans received 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Comments (achievements against targets): The project achievement exceeded the target laid down for this indicator. 103% of the target was achieved. Unlinked Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of publications in Number 7032.00 9000.00 18000.00 18163.00 refereed journals (within the field of Engineering) 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Comments (achievements against targets): Achievement on this indicator exceeded the target by 163 publications. 101% of the target was achieved. Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Formally Revised Completion Page 36 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Target Number of faculty members Number 0.00 1000.00 2900.00 3750.00 that have benefitted from the teaching effectiveness 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 training (under sub- component 1.3) Comments (achievements against targets): The project exceeded the target on this indicator by a margin of 850 faculty members trained. Achievement at closing was 129% against the target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of externally Percentage 6.00 20.00 12.00 14.04 funded research and development projects and 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 consultancies in total revenue Comments (achievements against targets): The project achieved 120% against the target on this indicator. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Share of TEQIP Supported Percentage 14.00 20.00 20.00 23.50 Institutions from states in lagging in technical 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 education Page 37 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Comments (achievements against targets): This indicator target was exceeded – the project achieved 117% of the target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Direct project beneficiaries Number 300000.00 350000.00 405000.00 443970.00 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Female beneficiaries Percentage 26.00 0.00 30.00 29.40 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Comments (achievements against targets): Achievement on this indicator exceeded the target by about 39,000 beneficiaries. 110% of the target was achieved at closing. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Fully functional MIS -Number Number 0.00 150.00 180.00 181.00 of institutions reporting at least 70% of the indicators 30-Nov-2009 31-Dec-2014 31-Mar-2017 10-Mar-2017 Comments (achievements against targets): Achievement on this indicator exceeded the target and was at 100.5% at closing. Page 38 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) A. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: To strengthen selected institutions 1. Share of supported programs that are accredited or applied for Outcome Indicators 2. Percentage of institutions with academic autonomy 3. Percentage of faculty with at least an M.Tech (regular and contract) 1. Increase in the overall satisfaction index of faculty and students on quality of education Intermediate Results Indicators 2. Transition of students from first year to second year by social group 3. Increased student internships with industry 4. Increase in partnerships between industry and academia 1. Upgraded UG curricula 2. Equity Action Plans Key Outputs by Component 3. Training and Placement offices operating in all institutions (linked to the achievement of the Objective/Outcome 1) 4. Pedagogical and subject area training for teachers 5. Institutional reforms on governance Objective/Outcome 2: To prepare more post-graduate students Outcome Indicators 1. Total number of Masters and PhD students enrolled 1. Percentage of revenue from externally funded R&D and consultancies in total revenue Intermediate Results Indicators 2. Increase in the number of publications in refereed journals 3. Number of co-authored articles in refereed journals 1. Institutional reforms including increased academic autonomy and increase in percentage of NBA accredited programs Key Outputs by Component 2. Upgrading faculty from masters to PhD (linked to the achievement of the Objective/Outcome 2) 3. Industry collaborations for research 4. Feedback from mentors and performance auditors Page 39 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Tara Beteille, Francisco Marmolejo Task Team Leader(s) Satyanarayan Panda Procurement Specialist(s) Alpana Saxena Financial Management Specialist Elfreda Vincent Team Member Maria Pinto Team Member Karthika Radhakrishnan Team Member Ritu Sharma Team Member Neha Pravash Kumar Mishra Environmental Safeguards Specialist Kurt Larsen Team Member Ling Jessica Diana Lee Team Member Rudraksh Mitra Team Member Harjot Kaur Social Safeguards Specialist A. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY10 0 262,767.43 FY11 0 56,127.94 Page 40 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Total 0.00 318,895.37 Supervision/ICR FY10 0 35,202.72 FY11 0 167,269.00 FY12 0 356,444.17 FY13 0 473,083.38 FY14 44.678 440,867.84 FY15 33.385 338,211.75 FY16 38.249 268,041.76 FY17 13.951 102,151.81 FY18 1.700 2,805.11 Total 131.96 2,184,077.54 Page 41 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Closing Percentage of Approval Components (US$M) (US$M) (US$M) Improving Quality of Education 453.00 334.2 74 in Selected Institutions Improving System Management 38.00 15.7 41 Unallocated Amount 9.0049 0.0 0.0 Total 500.00 349.9 70 Note: SDR 51.7 M (USD 72.9 M equivalent) was cancelled at the time of the first restructuring bringing down the total available amount for the project. The final disbursement rate for the project is 98.9% based on expenditures incurred under the project up to March 31, 2017. See paragraph #71 for details on available and disbursed funds in SDR. 49 The unallocated funds could be used for either component based on performance and need. This amount was utilized and is reflected in the totals shown for ‘Actual at Project Closing’ and ‘Percent of Approval’. Page 42 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ANNEX 4. EFFICIENCY ANALYSIS The economic feasibility of the project is examined through internal rate of return calculations across a variety of scenarios. The returns are measured compared to baseline projections such that the benefit-cost analysis and rate of return calculations are incremental over the counterfactual state of the world in which the project does not take place. The computations are based on the following assumptions some of which we were made at project appraisal: 1. Zero rate of inflation and no secular productivity trend growth so that the wage-experience profiles estimated at one point in time can be used to estimate life-time wage experience profiles for graduates from engineering education institutions, both beneficiaries and non-beneficiaries of the project. 2. The discount rate is assumed to be 5 percent per annum which approximates the real interest rate. 3. Project beneficiaries and the comparison groups earn wages over a period of 35 years. The difference in the life-time stream of the two groups is the benefit attributable to the project. We make the reasonable assumption that the incremental wage is Rupees 32,500 per year/ USD 500 for our baseline case. This is consistent with the assumptions made in the Project Appraisal Document. 4. Based on project data we can categorize about 85 percent of participating institutions as “well- performing� in that these institutions made better progress towards project outcomes than the remaining 15 percent of institutions. Average salaries earned by students will vary across institutions of different quality. To incorporate this wage differential in our analysis we assume that students from well-performing institutions have, on average, 50 percent higher earnings than students from institutions that did not perform as well. This assumption is based on available data on the range of earnings estimates for undergraduate engineering students in India and in TEQIP project institutions.50 Thus, 85 percent of beneficiaries from well-performing institutions will earn about 50 percent more compared to the other students under TEQIP II. 5. We have also assumed three different scenarios of placement rate - 40%, 50% and 60%. These placement rates are arrived at by assuming an average placement rate across TEQIP II institutions. Although some well performing institutions may have higher placement rates (~80 percent), we consider two facts - a) placement rates vary considerably across institutions of different quality51 and b) they also vary by department within the field of engineering (usually IT and Electrical engineering have higher placements compared to civil and chemical engineering52). Since TEQIP II catered to all streams of engineering it is not possible to estimate the average placement rate for all beneficiaries without error. 50 Data on wages of undergraduate engineering students has been estimated using a nationally-representative th household survey (NSS, 68 Round, 2012). Additionally, data on UG earnings from the project MIS, tracer study and impact study have also been referenced to estimate the range of earnings. 51 National Employability Report of Engineers, project MIS and case studies of TEQIP institutes conducted by IIT Delhi 52 ibid Page 43 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Therefore, we make a conservative assumption about the placement rates to be 40%, 50% and 60%. Table 1 below describes the assumptions made for the benefit-cost analysis. Table 1: Benefit-Cost Analysis Costs (in million USD) Project Costs $191.91 Opportunity Costs 0 Total $191.91 Benefits Average Wage Rate USD 6000 (INR 300,000) / year Additional Information Number of student beneficiaries under the project53 FY 2011 318279 FY 2012 333358 FY 2013 354811 FY 2014 369735 FY 2015 379130 FY 2016 369642 Incremental Wage (Base case) USD 500 (Rupees 32,500) / year Beneficiary Working Life 35 years Discount Rate (Base case) 5% per annum We have done a multi-dimensional sensitivity analysis by assuming different values for: (a) the discount rate; (b) incremental wages; and (c) placement rate (%). Internal Rate of Return (IRR) for the Project under different scenarios Table 2 below shows the IRR for the project under different assumptions for the extent of benefits and the discount rate. The IRR ranges from 7% to 72%. If we assume that the average rate of return to alternative investment in the economy is approximately 11-12%, then the project is always justified except when in case 1 with a 10 percent discount rate when the incremental wage is USD 250 and the placement rate is 40 and 50 percent. In the base (or moderate) case, when the incremental wage is USD 500, the project is justified in all cases. The seemingly high IRR of the project compared to that estimated at appraisal could be attributed to (a) significantly higher number of beneficiaries (more than 300,000 students) than those estimated at appraisal54 and (b) lower project costs (US$ 191.91 million) compared to estimates at appraisal (US$ 250 million). 53 Post Graduate (PG) students are not included in the IRR analysis. Lack of reliable data on average wages and placement rates at the PG level would lead to erroneous assumptions about differences in placement rates for UG and PG students and average wages between the two groups. 54 The number of beneficiaries considered at appraisal was 150,000 and is far less than the baseline for TEQIP II which was about 300,000 students. The Economic Analysis in the PAD therefore underestimated the IRR for TEQIP II. Page 44 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) The rates of return in Table 2 underestimate the full range of returns to the project. In computing the returns we have taken into account only the labor market benefits accruing to undergraduate students exposed to the project over the four year period. Project benefits will accrue to students graduating from project institutions for at least 5-20 years. With this in view, combined with the picture from the rates of return in alternative scenarios, the project is robust to future variations. Table 2: Internal Rate of Return Placement Rate Placement Rate Placement Rate (40%) (50%) (60%) Case 1: Incremental wage for students at well-performing institutions: USD 250 Incremental wage for students at other institutions: USD 125 Discount Rate 5% 12% 16% 21% Discount Rate 10% 7% 11% 15% Case 2: Incremental wage for students at well-performing institutions: USD 500 Incremental wage for students at other institutions: USD 250 Discount Rate 5% 29% 38% 46% Discount Rate 10% 23% 31% 40% Case 3: Incremental wage for students at well-performing institutions: USD 750 Incremental wage for students at other institutions: USD 375 Discount Rate 5% 46% 59% 72% Discount Rate 10% 40% 52% 64% Note: Based on the assumptions detailed in the note, 15 percent of beneficiaries earn 50 percent lower wages than the incremental wage assumed for students from well-performing institutions. In the 3 cases shown in the table, the incremental wage for students from well-performing institutions is USD 250, 500 and 750. The benefits that accrue in the future are discounted using the discount rates shown. Page 45 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS This has reference to the Implementation Completion and Results Report of TEQIP-II project, draft of which has been shared with the Ministry. The report adequately captures the various aspects of implementation and has also brought out areas which require strengthening. These lessons and recommendations are very crucial since they guided the formulation and follow-up project TEQIP-III. The TEQIP-II project was considered a successful project because of the structural changes brought-in in the institutions’ governance. The report has captured the outcomes which were measured based on the Project Development Objectives. However, the TEQIP-II project has supported institutions which have come forward voluntarily to becoming part of the project and therefore, has addressed areas which are relatively more developed. Based on the outcomes which are achieved in TEQIP-II, the TEQIP-III has been posed as a focus area project to provide undivided attention on the backward States especially the tribal, the North-east and hilly States. The NPIU at the national level has learnt valuable lessons in organising large programmes and for processing the proposals received from the field without any delay. The project activities have been organised in a transparent manner and there were proactive disclosures of various progress reports at the project level. The Ministry agrees largely with the overall findings of the report, and thank you for a well-written report. Regards Subrahmanyam Page 46 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) SUMMARY OF BORROWER’S ICR 1. Project Description and Objectives The second phase of TEQIP had 191 technical institutions (one institution dropped out as it did not receive the funds), competitively selected from several applicants. In order to boost research in the participating institutions, 30 centres of excellence (COEs) were set up in some of the participating institutions – involving creation of thematic research infra-structure and groups across emerging areas of relevance. 1.1 Project Objectives: Phase II of TEQIP intended to continue the good work of Phase I, and address the concerns and lacunae encountered, while adhering to the over-arching goals of educational reforms to transform India into a knowledge super-power and a hub for skilled manpower required to power the growth engine of the economy. The primary objectives were: a) Strengthening institutions to produce high quality engineers for better employability. b) Scaling up postgraduate education and demand-driven research, development and innovation. c) Establishing centers of excellence for focused thematic research. d) Training of faculty for effective teaching. e) Enhancing institutional and system management effectiveness. f) Capacity building to strengthen management. 2. Achievement of Project Objectives The stated objectives of the project were ambitious and demanding. Academic and systemic reforms require change in mind-sets, overcoming systemic inertia and a willingness to reform. For this synergy amongst the participants – MHRD/NPIU, State and Institutions was of utmost importance. Varying regional constraints, perspectives and ambitions had to be overcome to get a holistic satisfaction of the stated objectives. This required a clear enunciation of performance indicators, and a periodic assessment of performance against these indicators. The project saw several initiatives being undertaken to effectively achieve greater success in areas that were cause for concern from TEQIP I. 2.1 Component 1: Improving quality of education in selected institutions This component targeted improvement of overall academic quality and ambience in the selected institutions. The outcomes have been measured in terms of three broad categories, as discussed below. Subcomponent 1.1: Strengthening institutions to improve learning outcomes and employability of graduates. (a) Number of institutions with autonomy Page 47 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) There has been a concerted effort to give autonomy to the academic institutions so that they can innovate in the design of curriculum, have more effective and learning-centric evaluation processes, introduce flexibility through credit based system with possibility of earning credits from other institutions, continuous evaluation that is not examination-centric. All these initiative are possible if the institutions are not forced to follow a fixed structure imposed either by the State Technical Universities, or other affiliating university. Following Table 1, it can be seen that a large number of institutions have become autonomous (by UGC), as compared to the pre-project data. Table 1 Percentage of institutions with academic autonomy Baseline Actual (Previous) Actual (Current) End Target Value 30.00 69.00 70.00 70.00 Date 30-Nov-2009 15-Sep-2016 10-Mar-2017 31-Mar-2017 (b) Number of accredited UG/PG programs NBA accreditation is outcomes-based accreditation, with key parameters for accreditation including student performance and processes for continuous improvement of outcomes. Applying for NBA accreditation is voluntary and time-consuming as processes and outcomes must be in order for applications to be accepted. Getting an NBA accreditation is difficult. Following the data in table 2, it can be seen that there has been a significant increase in the number of programs accredited (or applied for) by NBA. Table 2 Share of supported programs that are accredited or applied for Baseline Actual (Previous) Actual (Current) End Target Value 30.00 65.00 65.00 55.00 Date 30-Nov-2009 15-Sep-2016 10-Mar-2017 31-Mar-2017 There has been a remarkable increase in the number of accredited programs state-wise. From a poor 10% accredited programs in 2010, the number has risen to more than 55% in general with high values of greater than 70% in Himachal, Maharashtra, Tamil Nadu and Puducherry. Overall, this is a very good achievement. This means that more students, across several disciplines have access to good class-room instruction, laboratory experience and other academic enablers. A cause for concern is the decrease in percentage of accredited programs in Uttarakhand (from around 70% to 65%). Similarly, the percentage accreditation of Haryana, Punjab, Rajasthan, Karnataka and Telangana is still not good (less than 50%). (c) Modernization of laboratories, classroom, library and e-resources: In order to achieve the overall learning objectives, it is important that the laboratories are well- equipped, current and relevant to the demands of the curriculum. Similarly, modernization of the class- Page 48 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) room (e.g. smart class-rooms) and upgradation of library and internet connectivity play a vital role in the learning process. TEQIP II provided resources for the modernization and upgradation of all these facilities. A study on the utilization of assigned resources was carried out via a report titled “Study to assess utilization of equipment & other resources purchased under TEQIP II�. Some salient features and outcomes of this report are: i. Under sub-component 1.1 institutes got 55 per cent of total available amount of Rs 10 Crore, i.e., Rs 5.5 Crore for purchase of equipment and soft resources. ii. Institutes spent 58% of total expenditure on laboratory equipment, 14% on software, 10% on books, 7% each on office equipment and connectivity and 1-2% each on furniture, civil work and smart classes. This shows a major emphasis on modernizing the teaching laboratories. However, many institutions (especially the non-CFTI institutions) still have poor wi-fi connectivity in common areas and classrooms. iii. Institutions benefitted immensely from purchase of high-value technical software like matlab, Ansys, Labview, Cadence and Adobe. Similarly, institutions could buy specific engineering software, digital equipment, high precision manufacturing and visualization equipment and software which helped immensely in enriching the learning experience. iv. Most colleges utilized more than 75% of the allocated funds. Some colleges in some states like Punjab (52%) and Uttarakhand (56%) could not effectively use the assigned resources. v. Instead of prioritizing purchases, the institutions typically divided money equally across several departments. Cooperation across departments was low. Overall, this approach led to sub- optimal utilization of resources vi. Hands-on learning has been emphasized under TEQIP II, and several institutions have tried to implement it with various degrees of success. Often, due to fear of damage to the equipment, demonstration experiments were conducted – hence denying the student the experience of working with a high-end equipment. vii. Overall, the TEQIP grant was a boon for the colleges, as they could achieve global upgradation of laboratories and other facilities – which has improved the academic ambience and learning of the students. (d) Quality of graduates i. Quality and quantity of placement: Gross placement figures are given, in terms of tracer studies (source: National Report on Tracer Study of Alumni of TEQIP II Institutions) with 5548 alumnus from across the country, at around 75% graduating students finding employment. Placement data can vary across specializations, e.g. 100% for mechanical engineering and 83% for civil engineering for one of the selected colleges. Page 49 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ii. However, the indicators for quality of jobs (core or non-core; software; finance and marketing) and the average gross annual pay are difficult to come by. Following the impact study report it can be concluded that general gross salaries are in the range of Rs. 3 lakhs to Rs. 6 lakhs. A similar representative study reflects an increase in employer satisfaction from 33% (pre-2011) to 75% current. iii. A common comment came up is with respect to competence in soft-skills. Interestingly, the tracer study indicates that the institutions have not built strong bonds with industry; have not provided for and encouraged industrial internship for UG students (many do not have the mandatory 8-weeks window in summer for this activity); and have not provided for effective finishing school training. The study further emphasizes the need to create a good alumni network, which is missing in most of the project institutions. iv. Job Quality and Satisfaction: The tracer study indicates that there has been an improvement in the quality and quantity of placement – possibly through improved attention to technical education, placement support and soft-skill development. The study strongly emphasized better industry-institution network, strong alumni network and engaging in current research through interactions/collaborations with industry and leading academic institutions. v. Quality of graduates: An indicator that has been used to reflect improvement in quality of graduates is the percentage of students majoring from first year to second year of the program. The outcome is better than the target set. The result indicates that several institutions took up programs to help weak students (academically weak and category students), either general or from SC/ST. Table 3: Transition rate of UG students from first to second year (source: 8th JRM report) Baseline 2010-11 2011-12 2012-13 2013-14 2014-15 Target (2009-10) All 64.2 69.82 69.50 71.04 68.71 65 SC/ST 45 60.21 64.99 63.41 64.92 62.96 60 vi. Stakeholders’ Satisfaction: Overall, as compared to a baseline score of 3.9, the student satisfaction score has increased to 4.0 (target set at 4.3). Sub-component 1.2: Scaling up post-graduate education and demand-driven research, development and innovation a) Number of PG programmes and courses introduced: Several institutions have either started new branches, and complied with the goal of starting new post-graduate programs that are demand- driven and industry relevant. Page 50 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) b) Number of Masters and PhD students; papers published in reputed journals; external research grants received; consultancy projects, patents; inter-institution collaboration: There has been a substantial increase in the quantity of graduate students – both Masters and PhD in the selected institutions (from 30,000 in 2010 to 45000+ in 2016 against a target of 40,000). Table 4: Progress in postgraduate education, number of publication and externally funded research at TEQIP institutions Indicator 2009- 2010-11 2013-14 2015-16 Project End 10 Target Actual Target Actual Target Actual Target Number of publications in 7032 7500 11370 9000 19914 14500 18163 15000 refereed journals (within the field of Engineering) Percentage of externally 6 7 8.28 10 13.76 12 14.04 12 funded research and development projects and consultancies in total revenue c) Publications: An important indicator of quality of research is the publication record of the institutions. Towards this progress has been made, as the combined publication is 18,000+ as compared to the target of about 15,000 (see Table 3.6 for details, Source 8th JRM report). While publications in conferences are important, and has seen a healthy increase, the publication in refereed journals is important. This parameter has seen an increase (national average) from 29 to 57 per institute. A further review of the outcomes of the Bibliometric study suggests that a large percentage of the publications are in conferences and non-Scopus journals. On a positive note, funding from TEQIP II allowed the selected institutions to recruit more graduate students (especially PhD scholars) and part-time faculty members. This helped many research groups to improve their output. d) Knowledge Incubation for TEQIP Centres (KITs): All the KITs have put in place a mechanism by which graduate students in a TEQIP II institution can access library and laboratories at these institutions. This is done through several enabling channels. These are: i. Short and long term visits: Through KIT students can get in touch with domain experts as mentors. KITs have helped students access specialized equipment, software that may be required for their research work. The faculty at IITs have been extremely supportive and have allowed graduate students to spend time in their laboratories. Several PhD and Masters students have benefitted from this exercise. These students also benefit from the research environment and the ambience – going back to their colleges with renewed vigor. Page 51 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ii. Mentorship: Several PhD scholars have approached KIT, which has helped them contact desired mentors. These students have worked closely with their mentors on a research topic of interest (relevant to their thesis). Normally these students spend one to six months at the IITs, to interact and learn from their mentors. iii. Library facility: Several students have visited the libraries of the IITs, through KIT, and have thus been able to access books, journals and thesis documents – which otherwise would be impossible to do. All these initiatives have helped the interested graduate students immensely. It has also helped the colleges give a better training to their graduate students. A report on the initiative of Centres of Excellence (COEs): A major initiative of TEQIP II was the establishment of 30 COEs in 27 selected institutions. The aim was to encourage establishment of world-class research infra-structure and activity in some institutions, building on the existing expertise. All the 30 CoEs were selected through a competitive process by distinguished domain-experts. Senior academics also functioned as peer reviewers and mentors for the research teams at the CoEs. The peer mentor/reviewing process has provided valuable feedback to the research teams in order to enable them to revise and improve their research projects – hence greatly enhancing both the overall quality of the CoEs and the credibility of decision making. The CoE model has been one of the star outcomes of the TEQIP II initiative. It has provided a mechanism for focussed intervention, with direct interaction with domain experts in the institutions. By engaging research groups, a recipe for spawning successful research has been established. This is a model that can be aggressively pursued in future initiatives. Table 5: Outcome of activities of COEs No. of PhD No. of patents No. of Technology No. of No. of MOUs students enrolled obtained and transfer programs international with industry and awarded applied for publications 30 Centers of PhDs enrolled: Obtained: 9 6 Excellence 403 (3 each by BMS 315 59 supported in PhDs awarded: 44 Applied: 74 Bangalore and NIT TEQIP II Rourkela) Sub-component 1.3: Faculty Development for Effective Teaching (Pedagogy Training) Page 52 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) a) Qualification of Faculty: Technical education needs a critical pool of good teachers in specific subject areas. There has been substantial progress on this front, as seen from tables 3.7 and 3.10. The student to faculty ratio has improved in several of the selected institutions. TEQIP II has achieved its set targets effectively. A major bottleneck that remains is the large number of contractual and ad- hoc faculty in some cases. Table 6: Percentage of Faculty with or pursuing MTech and PhD (regular and contract) Baseline Actual (Previous) Actual (Current) End Target Value 63.00 92.74 93.45 90.00 Date 30-Nov-2009 15-Sep-2016 10-Mar-2017 31-Mar-2017 b) Training of faculty: i. In-house: All selected institutions conducted several workshops, conferences, curriculum review sessions and courses aimed at bringing in current knowledge – through lectures by peers and domain-experts from across the country. Generally, in a period of five years, each institution has conducted about 50-100 such courses. ii. Effect of KITs: The KITs carried out detailed evaluation of the undergraduate and graduate curriculum of the institutions through several conclaves and charted out individual plans for specific interventions depending on the needs of colleges in their Quality Circle (QC). More than 9000 faculty members have been trained in the past four years (see table 7 for yearly break-up). Table 7: Number of faculty members that have benefitted from the teaching effectiveness training Baseline Actual (Previous) Actual (Current) End Target Value 0.00 2,816.00 3,750.00 2,900.00 Date 30-Nov-2009 15-Sep-2016 10-Mar-2017 31-Mar-2017 Overall, the performance in component 1.2 has been rated Satisfactory. The enablers/initiatives have proved to be extremely useful in achieving the initial goals of the project. 2.2 Component 2: Improving System Management In TEQIP II, a lot of effort has been put in streamlining the administrative mechanism, as well as the monitoring mechanism. Subcomponent 2.1: Capability building to strengthen management: The objective of this sub-component is to build capacity of technical education policy planners, administrators and implementers at the central, state, and institutional levels. This is crucial for institutes to translate autonomy and accountability effectively to the grass-roots level during the project Page 53 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) period and beyond. Almost all institutes have undertaken governance self-reviews, completed governance development plans, and provided access to these reports by publishing these documents on their respective websites. The percentage of institutions that obtained autonomy by UGC has increased significantly from 30 percent in 2009-10 to 70 percent in 2015-16. A key endeavour under TEQIP II has been to share the learnings from good governance practices widely, so that non-TEQIP institutes can benefit as well. This has been accomplished through the website: http://www.teqipgoodgovernance.in/. Table 8: Growth in the percentage of institutions with UGC autonomy, governance self-reviews and governance development plans (source: Report of 8th JRM) Indicator 2009- 2010-2011 2013-14 2015-16 Project 10 Target Target Actual Target Actual Target Actual Percentage of 30 40 51 65 63 68 70 70 institutions with UGC autonomy Number of governance __ __ __ 80 166 190 186 190 self- reviews received Number of Governance __ __ __ 20 38 170 186 180 Development Plans i. Role of Board of Governors (BOGs): TEQIP II laid out a model administrative structure, with the Board of Governors (BoG) as the prime administrative authority. The BoG continuously reviews the performance of the institution on various fronts, and recommends policy changes. The BoG also ensures autonomy is practiced at all levels, ensuring that all arms of an academic institution are enabled to work freely and to their potential. Institutions that have recognized this are doing well. ii. Leadership and management development training by IIMs: A total of 2418 faculty/officials have attended the trainings of the Management Capacity Enhancement Programme (MCEP) conducted by seven IIMs. A large majority of participants, in various capacity building activities, indicated that these activities were very effective. The head of the departments or the institution heads of the institutions saw a positive change in the academic and administrative activities performed by these participants. The institution Heads noticed positive changes especially with respect to interpersonal relationship, team work and time management. Most of them shared that the improvement in the participants were discernible and resulted in more engaging student activities. Subcomponent 2.2: Project monitoring, evaluation and management Page 54 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) a) MIS system: TEQIP II supported the development and implementation of a web-based MIS system. This web-based MIS, the first of its kind for an education project in India provided timely, sufficient, precise and reliable information to improve and assess the performance of the selected institutions. By closing, 191 institutions were reporting on at least 70% of indicators which is above the revised end-of-project target of 160. b) Performance Benchmarking: The first Joint Review Mission (JRM), held in February 2013, recommended the introduction of performance benchmarking to be achieved in six months by linking performance to release of funds to institutions. This innovative mechanism enabled sound decision-making under the project and helped in improving the project outcomes. c) Assessments and Studies: Two types of assessment studies were carried out. One set focused on project impact and the other set was more qualitative, to continuously monitor and understand project performance as well as enable informed decision-making. d) Impact Assessment:  An interim impact assessment, carried out by Administrative Staff College of India (ASCI), was based on interviews of senior management and a survey of staff and students in 30 of the TEQIP institutions. It helped the project agencies monitor progress, fine tune monitoring and support systems, and target support and capacity development to where it is most needed. A final impact assessment of the project has been completed by IIT Delhi.  A Tracer Study was introduced in the project during implementation and was successfully carried out by NPIU using SurveyMonkey. Results from the tracer study of students that graduated in 2014-15 and 2015-16 suggest that 75% were in full time employment and 6% had opted for higher education.  Other studies carried out TEQIP II include: (i) Bibliometric study carried out by the Indian Institute of Science, Bangalore; and (ii) Resource Utilization study carried out by Confederation of Indian Industry. e) Reviews and Surveys:  Student and Faculty Satisfaction survey was designed in the project with the objective of measuring students and faculty satisfaction with quality of teaching-learning, faculty, infrastructure, academic and other support provided by their institutions. Institution-wise report cards were also prepared and shared with all institutions. It should be noted that NPIU has now developed the capability to carry-out these surveys in-house.  Reviews of Performance Auditors and Mentors: NPIU conducted a survey of mentors and performance auditors, in order to obtain their general feedback about the success in implementation and areas for future improvement of the mentorship and performance auditing components of the project.  Assessment of COEs: An assessment of the performance of CoEs was conducted by senior academics/peer reviewers from IITs using a Progress Review Sheet. The assessment served as Page 55 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) guidance for the CoEs in identifying areas where they are on track as well as issues that need improvement. Detailed outcomes have been discussed above.  Assessment of IITs and IIMs: An evaluation on the impact of the IIT and IIM training on teaching and research quality, student learning and employability, as well as the efficiency of leadership and management of the institution was carried out under the project.  QEEE Feedback: Feedback from student and faculty on the Quality Enhancement in Engineering Education (QEEE) initiative, which was spearheaded by IIT Madras, was continuously carried out during the implementation of the initiative to ensure that feedback obtained are used to enhance the effectiveness of the program.  Report on implementation of EMF: A report was prepared by NPIU to highlight the findings/ suggestions from workshops conducted to monitor the compliance of Environment Management Framework in all project institutions and to ensure implementation of corrective actions. (e) Joint Review Missions: The project was monitored through bi-annual JRMs conducted jointly by MHRD and the World Bank. The review missions were used to evaluate progress in achieving the project outcomes and to report on lessons learned during implementation. 3. Project Finance Management The project was declared effective on 6th Aug., 2010 and closed on 31st July 2017. Original Credit Amount SDR 186.4 m Revised Credit Amount SDR 134.6 m Amount Disbursed SDR 133.146 m %age achieved 98.92 Funds released upto July 31, 2017 : 2167.40 Crore Expenditure upto July 31, 2017 : 2187.31 Crore 4. Summary and closure: The implementation of TEQIP II was a gigantic effort, a one of its kind initiative, which brought together 191 technical institutions, thousands of faculty, premier technical and management institutions in India, industry-pundits, MHRD and world-bank officials in one seamless initiative. Through continuous monitoring and review; mentoring and intervention by IITs and IIMs; setting up of specialized research hubs via the CoE initiative; centralized MIS and survey mechanisms; enforcing good-governance principles and several other collaborative and individual initiatives, a serious attempt was made to upgrade the quality of technical education in the selected institutions. The outcomes are promising, and learnings were several. Overall, definite positive outcomes were achieved – especially for those institutions that honestly pursued the project goals. The project designers envisaged such an outcome Page 56 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) and set realistic targets, which were achieved or surpassed in most cases. Reviews have thrown up several areas that need to be addressed in subsequent projects (TEQIP III). Page 57 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) 1. Andreas Blom & Hiroshi Saeki (2011). “Employability and Skill Set of Newly Graduated Engineers in India�. Policy Research Working Paper No. 5640, World Bank. 2. Government of India, Ministry of Human Resource Development (2012). “The Good Practice Guide for Governing Bodies�. Retrieved from http://www.teqipgoodgovernance.in/TEQIP%20GOOD%20PRACTICE%20GUIDE%20FOR%20GOVERN ING%20BODIES_DEC%202012.pdf 3. Government of India, Planning Commission (2008). Eleventh Five Year Plan, 2007-12, Volume II – Social Sector. Retrieved from http://planningcommission.nic.in/plans/planrel/fiveyr/11th/11_v2/11th_vol2.pdf 4. Government of India, Planning Commission (2013). Twelfth Five Year Plan, 2012-17, Volume II. Retrieved from http://planningcommission.gov.in/plans/planrel/12thplan/pdf/12fyp_vol2.pdf 5. World Bank (2009). Country Strategy for the Republic of India for the Period 2009-2012. Report no. 46509-IN. 6. World Bank (2013). Country Partnership Strategy for India for the Period FY 2013-2017. Report no. 76176-IN. 7. World Bank (2009). Project Appraisal Document. Report no. 48470-IN. 8. Restructuring Papers 9. Implementation Status and Results Reports 10. Aide Memoires Page 58 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ANNEX 7. ACHIEVEMENTS BY COMPONENTS Progress against select intermediate outcomes and outputs is discussed here. Overall, the project was successful in meeting all IO indicator targets. Some institutions excelled over others -- this was expected, and results from the competitive selection process as well as state-specific factors. a. Component 1 – Institutional strengthening to improve quality of education in selected institutions : Reform-based investments with activities focused on student and faculty development and on building the institution’s research capabilities were key mechanisms adopted here. a. Equity Action Plans: Institutions were required to develop an Equity Action Plan (EAP) to help students underprepared for college. These plans included a variety of activities offering students additional academic support (through bridge or remedial courses), peer learning groups, faculty mentors, ensuring access to study spaces and enhancing communication and presentation skills.55 i. Institutional efforts towards adhering to EAPs were assessed by the NPIU and SPFUs. Progress on quantitative indicators like transition rates and student placements were gathered by the project and in some states, such as Maharashtra, activities undertaken as part of the EAPs were compiled for dissemination of best practices. ii. Effect on student transition rates: Transition rates at project institutions improved overall (68.71%) and specifically among disadvantaged students (62.96%) exceeding the end target by about 3 percentage points. The experience from TEQIP I suggested that improvements in transition rates signaled that students were learning enough to pass exams and continue their college learning journey. b. Placement Cells and Student Internships: All project institutions set up placement cells to assist students secure internships and benefit from pre-placement activities like informational sessions by employers/alumni, campus interviews for graduating students, and personality development workshops to develop necessary soft skills. These types of activities organized by placement cells are associated with better learning outcomes among students.56 i. Placement and training cells were set up in all project institutions and over 60% of students surveyed as part of the tracer study indicated that placement cells provided soft skills training and personality development workshops in addition to other activities like organizing campus interviews. Strong linkages between institutions and industry however depended on the state’s industrial base and institutions in states with a considerable base were able to provide students better internship and employment opportunities. 55 Examples of good practices for supporting first-generation college goers or students underprepared for college have been highlighted in this blog post: http://blogs.worldbank.org/education/most-effective-services-well-rounded-higher-education- students-india 56 Mitra et al. (2016). Page 59 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) ii. The project leveraged partnerships with the best engineering institutions in the country – the Indian Institutes of Technology (IITs) –to offer students internship and learning opportunities. Senior faculty at the IITs offered summer and winter internship and learning opportunities for students at TEQIP institutions. Partnerships with international organizations like MITACs for research opportunities were also established. About 200 students were competitively selected in 2014-15 and 2015-16 to participate in a 12-week research internship in Canadian universities as part of this initiative. iii. The benefits of these activities are available in the form of student feedback and indicate their clear satisfaction with resources and opportunities available at their institutions. 57 Yet, they reached only a minority of students. Institutionalizing these opportunities for students and ensuring students share their learnings will ensure sustainability of these efforts and contribute to the project’s long-term objectives of improved quality and employability. c. Mentors and performance auditors: The role of mentors and performance auditors was key to ensuring that project institutions implemented reforms and carried out activities based on their development plans, identified remedial actions, and made continuous progress towards project goals. Mentors, who were senior academics from the field, served project institutions – including Centers of Excellence - in an advisory capacity and institutions could reach out to them anytime. Performance auditors periodically evaluated institutional progress. Over time, a core group of experienced mentors took on more of an advisory role in the project and trained other mentors for institutional visits. i. Institutions were surveyed on the quality of the mentoring and performance auditing and the vast majority of them felt that they benefitted significantly from feedback by mentors. ii. Mentors provided valuable feedback to research teams at CoEs that enabled them to improve and revise research projects thereby enhancing the quality of the CoE and the credibility of decision-making. iii. In addition to institutional support, mentors were also links between project institutions and the state implementing agencies (SPFUs) which enabled a common understanding of the project’s key elements and priorities and its progress among key stakeholders. d. Building and Expanding Research Capabilities: Project institutions received support to scale-up demand-driven R&D in collaboration with industry or other research organizations. A smaller subset of project institutions (27 in all) meeting higher eligibility criteria were supported to establish 30 Centers of Excellence (CoEs) focused on cross-departmental research supported by the private sector. These centers had adequate financial autonomy through the project; they set up state-of- the-art facilities and brought on board PhD scholars to undertake high-end research. Mentors and performance auditors provided research teams with valuable feedback and recommendations to revise and improve research projects. This feedback was translated in to Action Plans by CoEs that included plans for sustainability of the centers after project closing. Table 1. Number of publications and externally funded research at TEQIP institutions Indicators 2009-10 2010-11 2013-14 2015-16 End Target 57 This blog post showcases student experiences and satisfaction with a competitive summer internship program in Canada: http://blogs.worldbank.org/education/supporting-india-s-next-generation-bright-tech-and-science-minds Page 60 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Indicators 2009-10 2010-11 2013-14 2015-16 End Target Number of publications in refereed journals (within the field of Engineering) 7032 11370 19914 18163 15000 Percentage of externally funded research and development projects and consultancies in 6 8.28 13.76 14.04 12 total revenue i. These activities have led to a strong R&D culture as evidenced by the increase in the number of publications in refereed journals in the field of engineering and the percentage of externally funded projects and consultancies in the institution’s total revenue. As seen in Table 1, the number of publications in refereed journals went from about 7,032 at baseline to over 18,163 by 2015-1658. Externally funded research and consultancies made up about 14 percent of the institution’s total revenue surpassing the end target of 12 percent for the project period. ii. Among institutions that had many publications the CFIs outperformed other project institutions in terms of number of publications and the impact of the same. Institutions with more and diverse postgraduate departments published more and their impact was larger. Publications were also related to faculty strength and so institutions with several contractual faculty focused on teaching at the UG level had modest publications. iii. In the CoEs specifically, significant accomplishments are noted; across the 30 centers, 59 MoUs have been signed with industry, 74 patents were applied for and 9 obtained, faculty and students produced 315 international publications, and a large number of PhDs were enrolled (403 students) and awarded (44 students) during the project period. iv. Reviews of CoEs indicate an increase in confidence and collaborative spirit among faculty and students. The CoEs provided project institutions the necessary incentive to pursue industry collaborations especially where institutions are located near a considerable industrial base (for instance, Mumbai, Pune and Durgapur). Outside the COEs, however, industry collaborations are weaker. This could partly be due to the time and strategic approach required to build collaborations. Table 2: Number of PhD students enrolled and awarded, international publications, patents obtained and applied for, and MoUs with industry under the CoEs in October 2016 Indicators No. of PhD students No. of patents No. of Technology No. of No. of MoUs enrolled and obtained and transfer programs international with awarded applied for publications industry 30 Centers of PhDs enrolled: 403 Obtained: 6 Excellence 9 (3 each by BMS 315 59 supported PhDs awarded: Applied: Bangalore and NIT under TEQIP II 44 74 Rourkela) e. Faculty development: This area received significant attention as limited professional development opportunities are a major constraint to improving the quality and relevance of engineering education. The project leveraged the expertise of premier engineering institutions in the country to improve teachers’ pedagogical and subject domain skills. Knowledge Incubation Cells (KITs) and 58 The bibliometric study carried out as part of the project indicates that many publications are in conferences or non-Scopus journals. Page 61 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) Quality Circles (a cluster of 20-25 institutions) were set up at 8 IITs. Based on a thorough evaluation of the curriculum at project institutions, the KITs developed specific modules and workshops. Activities included teaching workshops in general pedagogical techniques, for specific content areas, workshops on content development, seminars, short-term visits for students and longer visits for faculty from TEQIP institutions focused on upgrading content knowledge as well as pedagogical skills to teach that knowledge. In addition to the KITs, IIT Madras designed and implemented a blended MOOC program called Quality Enhancement in Engineering Education (QEEE). Through this program, professors from the IITs taught a portion of the syllabus remotely and supported faculty to teach the remaining syllabus. The penetration of QEEE was remarkable and provided invaluable learning to students and faculty alike. i. By the end of the project period, over 7,000 faculty members had benefitted from teaching effectiveness training under sub-component 1.3, surpassing project targets (4,682 teachers). An additional 9,500 teachers received training in subject-domains. ii. Satisfaction surveys of faculty and follow up surveys for each KIT suggest overall satisfaction with the training. Students’ responses to surveys also reveal satisfaction with the quality of teaching at their institution. iii. Qualitative feedback from the KIT survey and feedback on QEEE were regularly collected and used to inform and improve subsequent training programs. For instance, the feedback revealed the need to strengthen planning and academic leadership required to identify faculty candidates for training, their training needs, and to avoid mismatch between participants and training. f. Component 2 - Improving system management: A key mechanism through which the project envisioned change was by strengthening governance at the level of institutions, the centre and state. Improvements in institutional governance practices was conducive to academic autonomy and contributed to results under Component 1. Activities for improving management of the system focused on capacity building for effective implementation of reforms in project institutions and knowledge sharing between TEQIP states and institutions. a. Good Governance: A governance guide59 was developed as part of the project which provided institutions an easy-to-use roadmap on how to implement governance practices. The guide played a big role in moving institutions from ‘inaction to action’ on governance. 186 institutions each completed reviews of their governance practices based on which they developed Governance Development Plans that were published on the institute’s website. Good governance practices were also posted on the TEQIP Good Governance website (http://www.teqipgoodgovernance.in/) so non- TEQIP institutions could benefit. As part of the governance reforms, institutions were mandated to establish Board of Governors (BoG) or similar body with a specific composition and without political interference. While this happened in many cases, in a few cases institutions were found to establish a temporary BoG for the purposes of TEQIP or set up an ineffectual BoG with limited decision- making powers. 59 The TEQIP Good Practice Guide for Governing Bodies can be found here: http://www.teqipgoodgovernance.in/good-practice-guide-governing-bodies.html Page 62 of 60 The World Bank India: Technical Engineering Educational Quality Improvement II ( P102549 ) b. Leadership and Management Training: As the IITs were leveraged for their technical expertise in faculty and student development, the project leveraged India’s leading management institutes – the Indian Institutes of Management (IIMs) – to conduct Management Capacity Enhancement Programs for faculty/officials in project institutions. Topics covered strategic planning and implementation to infrastructure and people management, corporate governance and finance and accounts. 2,418 participants across all project institutions were trained across 87 batches. This number comprises about 10 percent of faculty at project institutions. i. Evidence of training effectiveness is available from the self-assessment studies conducted by the IIMs. These indicated positive changes in participants’ interpersonal relationships, team work, time management and engagement with student activities. However, the IIMs also noted that institutions needed to develop better strategies for long-term financial planning and deployment of funds in a more decentralized way to benefit a larger number of departments. Page 63 of 60