Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) Report Number : ICRR0020955 1. Project Data Project ID Project Name P102549 IN: Tech Engr Educ Qual Improvement II Country Practice Area(Lead) India Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-46850 31-Dec-2014 500,000,000.00 Bank Approval Date Closing Date (Actual) 18-Mar-2010 31-Mar-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 300,000,000.00 0.00 Revised Commitment 217,910,320.90 0.00 Actual 191,908,912.05 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg John R. Eriksson Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives According to the Financing Agreement (p. 4), the project's objectives were "to produce more employable and high quality engineers, and prepare more post-graduate students to reduce faculty shortage." Key outcome targets were increased at a January 2014 restructuring. Because all of the revised targets were more ambitious, and these targets were met or exceeded, this ICR review assesses efficacy in relation to the revised targets only, and therefore this ICR review does not perform a split rating. The project was the second phase of an envisioned 15-year phased program. The first phase, the Technical Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) Engineering Educational Quality Improvement project (US$ 315 million, 2002-2009), supported 109 engineering education institutions in 13 states as well as 18 central institutions. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 06-Jan-2014 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project contained two components: 1. Improving Quality of Education in Selected Institutions (appraisal: US$ 453 million; actual US$ 334.2 million). This component was to support around 200 competitively selected engineering education institutions, through two funding windows, to improve learning and outcomes and employability of graduates, and to scale up research, development, and innovation. Selection of the 200 institutions was to take place in two rounds, in 2009 and 2010, according to a pre-determined set of eligibility criteria. The first funding window was to strengthen around 140 institutions to improve the competencies of undergraduates through the implementation of reforms and investments focusing on accreditation, academic autonomy, faculty qualifications, increase in faculty and student satisfaction with quality of education, and transition rate of students from first year to second year by social group. Institutions in states "lagging in technical education" were to receive preferential treatment (PAD, p. 7) in order to prioritize institutional strengthening in those states, and institutions were to be asked to improve social and academic assistance to "weaker students" to improve their learning and employment outcomes (PAD, p. 10). The second funding window was to increase enrollment in post-graduate education and produce engineering research, development, and innovation in collaboration with the private sector, focusing in particular on enrollment in PhD programs in around 60 advanced institutions in order to address faculty shortages. In addition to the two funding windows, this component was also intended to carry out pedagogical training of about 10,000 engineering faculty members, in around 500-750 batches of 20 faculty members each, conducted in project-supported institutions. 2. Improving System Management (appraisal US$ 38 million; actual US$ 15.7 million). This component was to build capacity of technical education policy planners, administrators, and implementers at the central, state, and institutional levels, and to support project monitoring, evaluation, and management. According to the Project Appraisal Document (PAD, p. 8), there was also initially US$ 9 million of unallocated funds to invest in project activities that were producing the best results, and/or had to Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) overcome unforeseen challenges. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: Total planned project cost was US$ 500 million. Actual total costs were US$ 349.9 million. The project experienced savings from exchange rate fluctuations, changes in financing terms, and effective use of an e-procurement system. Financing: The project was financed by a US$ 300 million Specific Investment Loan (SIL), with US$ 280 million of the loan designated toward the first component, US$ 17 million for the second component, and US$ 3 million for the initially unallocated category. US$ 191.9 million was actually disbursed. At a January 2014 restructuring, US$ 72.9 million was cancelled, at the government's request, due to significant savings accruing from exchange rate fluctuations. Borrower Contribution: The Borrower was to contribute US$ 200 million, with state governments responsible for 25% of the cost of activities at the state level, and the central government responsible for 75% of the cost of activities at the national level (this represented an overall center:state funding ratio of 60:40). The total actual contribution was US$ 160 million. Dates: The mid-term review was held in December 2013. There were three restructurings. On January 6, 2014, approximately US$ 80 million of Bank financing was cancelled due to considerable shifts in the exchange rate, and the project's results framework was modified with revision of key intermediate outcome and outcome indicators and targets. On September 30, 2014, the closing date was extended from May 1, 2014 to October 31, 2016, to allow for further development of project institutions and additional motivation toward better performance. On September 26, 2016, the closing date was again extended, to March 31, 2017, to allow for further utilization of project funds, and the project's co-financing arrangement between India's central and state governments was shifted from 60:40 to 50:50. The project closed on March 31, 2017. 3. Relevance of Objectives Rationale As this was the second phase of a 15-year program, the project's objectives were appropriately ambitious and framed in terms of the desired outcomes of high-quality, employable engineering graduates, and alleviation of engineering faculty shortages. The objectives were substantially relevant to country conditions and to government and Bank strategy at the time the project closed. The ICR does not discuss country context in 2017, but a 2016 report cited by the ICR (p. 18, footnote 31) notes that there had been no significant improvements in employability of engineers over the previous four years, and that there remained a "huge" demand for manpower in the technology sector that was unmatched by supply. The Government of India's 12th Five-Year Plan (2012-2017) focuses on sustainable and inclusive growth specifically through Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) increasing the supply of skilled workers to drive the economy while ensuring that low-income states receive special emphasis to catch up. Similarly, the World Bank's Country Partnership Strategy (2013-2017), under the engagement areas of integration and inclusion, has a strategic outcome to increase the production of high-quality workers for growing sectors of the economy and for improving demand-driven skills for productive employment. It is worth noting that, although there were project activities focused on lagging states, "weak" students, and gender considerations, the project's objectives did not include equity. The PAD (p. 2) identified inequitable access to technical education as an important issue, with particular focus on enrollment inequities between rich and poor, rural and urban areas, men and women, disadvantaged groups and the general population, and between states, and it appears that this continues to be an important element of country context. The follow-on project (Technical Education Quality Improvement Project III, P154523, approved in 2016) has objectives to enhance both the quality and equity of engineering education. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Produce more employable and high quality engineers Rationale Outputs: The project supported 190 higher education institutions from 23 states and union territories to implement their specific improvement plans, reaching nearly all the targeted 200 institutions. The ICR does not directly address the coverage represented by these institutions, though the PAD (p. 1) notes that there were approximately 2,400 engineering institutions in 2010, and the ICR (p. 6) links in a footnote to an All-India Council of Technical Education report citing growth from 2,972 institutions in 2009-2010 to 3,384 in 2013- 2014. Of the 190 higher education institutions supported, 26 were centrally funded, 126 were public or public-aided, and 38 were private. Thirty Centers of Excellence were established to create thematic research infrastructure and groups across emerging areas of relevance, supported by the private sector. Mentors (senior academics in their fields) and performance auditors were put in place to ensure that project institutions implemented their development plans, identified necessary remedial actions, and made progress toward specified goals. There were 4.44 million direct project beneficiaries, exceeding the original target of 3.5 million and the revised target of 4.05 million. The percentage of female beneficiaries increased from 26% in 2009 to 29.4% in 2017, almost reaching the target of 30%. 58% of funds for purchases at the institute level were spent on laboratory equipment, 14% on software, 10% on books, 7% on office equipment, 7% on connectivity, and 1-2% each on furniture, civil works, and smart Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) classrooms. The ICR (p. 49) points out that these allocations represent an emphasis on modernizing teaching laboratories, but that there is still poor wireless connectivity in many institutions. The project strengthened and reformed institutional governance and management in preparation for increased academic autonomy. A Good Practice Guide for Governing Bodies was developed to provide institutes with a roadmap for development and implementation of good practices. Institutes were mandated to establish Boards of Governors or similar bodies without political interference. About 10% of faculty at project-supported institutions underwent training in strategic planning, infrastructure and personnel management, corporate governance, and finance/accounts. 186 governance self-reviews were completed, exceeding the original target of 80 and almost meeting the revised target of 190. 186 governance development plans were submitted, exceeding the original target of 20 and the revised target of 180. Resulting at least in part from these interventions, the percentage of supported institutions with academic autonomy (defined as ability to make independent decisions about academic and curricular reforms, evaluation systems, staffing, financial management, sustainability, and private sector collaborations) increased from 30% in 2009 to 70% in 2017, exceeding the original target of 65% and meeting the revised target of 70%. As part of project activities, institutes were encouraged to spend 10% of their budgets on faculty and staff development, including conducting a training needs analysis and sending faculty to other institutes to upgrade their qualifications. The percentage of regular and contract faculty with at least a Master of Technology degree increased from 45% in 2009 to 92.35% in 2017, exceeding the original target of 60% and the revised target of 88%. The percentage of regular and contract faculty either holding or pursuing at least a Master of Technology degree increased from 63% in 2009 to 93.45% in 2017, exceeding the original target of 73% and the revised target of 90%. The number of annual publications in refereed journals in the engineering field increased from 7,032 in 2009 to 18,163 in 2017, exceeding the original target of 9,000 and the revised target of 18,000. The percentage of total revenue from externally funded research and development projects increased from 6% in 2009 to 14.04% in 2017, not meeting the original target of 20%, but exceeding the revised target of 12%. Given the project's focus on infrastructure and faculty development, it is plausible that these achievements were, at least in part, a result of the project. 3,750 faculty members received teaching effectiveness training, exceeding the original target of 1,000 and the revised target of 2,900. Scores on a satisfaction index of students and faculty increased from 3.9 in 2009 to 4.0 in 2017, not meeting the target of 4.3. Knowledge Incubation Cells and Quality Circles were set up at eight institutes to develop specialized modules and teaching workshops, and one institute designed and implemented a blended Massive Open Online Course on Quality Enhancement in Engineering Education. The share of project-supported institutions located in the 19 states lagging in technical education (defined as states having only one engineering institution, or less than one engineering institution per million population) increased from 14% in 2009 to 23.5% in 2017, exceeding both the original and revised target of 20%. The project required all participating institutions to complete Equity Action Plans to support lower- performing students, focusing on first-year students with programs including peer mentoring, open access to campus facilities, and orientation programs. As a result, the transition rate of students from disadvantaged backgrounds from the first to second years of undergraduate study increased from 45% in 2009 to 61.45% in Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) 2017, exceeding the original target of 55% and the revised target of 60%. According to the ICR (p. 14), the private sector was involved in key project activities to ensure that engineering education was demand-driven. Industry representatives sat on many institutions' Boards of Governors and were closely involved in curriculum revisions, lectures and faculty consultancies, and establishment of research partnerships. The ICR does not provide specific data on the strength and pervasiveness of these industry ties. Outcomes: The ICR makes convincing arguments that accreditation and autonomy are strong indicators of quality education. The project used institutional accreditation as a proxy for quality and employability of students. The project team provided detail on the metrics used for accreditation, which include (among many other factors) quality of student projects and assessments, initiatives related to industry interaction, and student performance and placement in employment or graduate-level education. The ICR reports that the percentage of supported programs/institutions that are accredited or have applied for accreditation increased from 30% in 2009 to 65% in 2017, exceeding the original target of 50% and the revised target of 55%. Importantly, project-supported programs received higher average accreditation scores (by about 30 percentage points) than the national average (ICR, p. 15). All institutes were required to set up placement cells and track the rates of student job placement over the project period. The methodology for calculating these rates varied among institutes and was standardized only in 2015. A tracer study of 2013 and 2014 graduates from project-supported institutions (with significant methodological caveats well explained in the ICR, including a relatively small sample size of 5548 students) found that about 75% of surveyed graduates were employed, and over 60% had found employment through campus placements. This compares favorably to a 2015 study of engineering graduates across the country (presumably covering both project and non-project institutions) that reported only a 20-30% job placement rate. Data on 2015 or 2016 graduates are not reported. The ICR cites project data indicating that the average annual salary for graduates of project-supported institutions increased by nearly 20% during the project period, a possible indicator of greater demand for higher quality graduates. It is not clear, however, how inflation may have impacted these salary figures, and the ICR does not address this issue. The ICR reports data from a survey indicating that employer satisfaction with newly recruited engineers increased from 33% in 2009-2010 to 75% in 2014-2015, but this survey covered graduates from both project and non-project institutes. It is therefore not possible to draw conclusions about project-specific outcomes from these data. The project did not collect and/or report more robust or recent data directly on the quality and employability of engineering graduates from participating institutions, and these graduates were not compared to those from non-participating institutions in order to assess the specific impact of the project. The ICR (p. 31) notes that there is variation in examination systems across states and years, making comparisons difficult, and that "test anxiety" among students complicates the design and administration of a credible assessment system. Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) However, there appear to be available options that were not used. The "Aspiring Minds" report cited by the ICR (p. 18) is based on results of the "Aspiring Minds Computer Adaptive Test," noted in the report as "India's largest and only standardized employability test" for engineers. This test is used by 3500 Indian companies -- including seven of the top ten information technology companies -- for assessment and recruitment. This instrument may have been useful for assessing the project's impact on production of employable and high quality engineers; it represents a missed opportunity for the ICR. In fact, some of the ICR's evidence calls into question the extent to which even achievement of targets for the project's key indicators can be attributed primarily to project-financed activities. Specifically, the ICR (p. 18) notes that progress on increased academic autonomy and accreditation varied according to the level of proactivity and policy environment at the state level, and that even non-project institutes made progress on quality improvement in states that implemented effective reforms. Similarly, employment of graduates varied by state according to a state's industrial base. However, despite these caveats and the absence of direct information on quality and employability of graduates specifically of project-supported institutions, the accreditation scores of project-supported institutions compared with non-project institutions -- scores that incorporate student achievement and employment, among other factors -- are indicative of project impact. Achievement of this objective is therefore rated Substantial. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Prepare more post-graduate students to reduce faculty shortage Rationale Outputs: The outputs on faculty/staff development and support for institutional development and research infrastructure cited under the first objective are also relevant here. The total number of Masters and PhD students at project-supported institutions increased from 30,000 in 2009 to 45,658 in 2017, exceeding the original target of 34,000 and the revised target of 41,000. Outcomes: Presumably, some of the graduate students trained under the project pursued academic careers and therefore have begun to alleviate faculty shortages, and will continue to do so in the longer run. The ICR does not report the number or percentage of graduate students trained in project-supported institutions who Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) were placed in academic jobs. The project team later added that institutions reported between 65% and 80% of their graduate students having taken teaching positions after completing their degree programs, with most institutions closer to 80%. The extent to which these new faculty have alleviated faculty shortages, however, is not clear. The ICR (p. 18) reports small improvements in the faculty-to-student ratio at project- supported institutions, as well as an overall increase of "about 40%" in the total number of faculty at these institutions, but this is not a measure of achievement of the project's objective, since graduate students trained by project-supported institutions could and should have moved into faculty positions at both project and non-project institutions of higher education across the country. The project team explained that faculty vacancy data are difficult to ascertain, given the complex political economy of the sector, and especially variance in departmental and institutional reporting on positions and vacancies. This explanation, however, calls into question the framing of the objective in these terms; if there was enough clarity around the existence of faculty shortages to state alleviation of these shortages as an objective, then it should be possible to measure whether those shortages lessened. Although the ICR and project team presented sufficient information to support a conclusion that project-supported institutions did prepare post-graduate students for faculty positions, and that these institutions saw an increase in the total number of faculty, achievement of this objective is rated Modest given the absence of information on actual alleviation of faculty shortages due to project activities. Rating Modest PHREVDELTBL PHOVRLEFFRATTBL Rationale The PAD (p. 5) states that the project's objective was "strengthening of institutions," with production of more employable and high-quality engineers as a "longer-term program objective." The ICR (p. 7) similarly argues that "improving quality, employability and reducing faculty shortages" were "longer term goals." The Financing Agreement, however, does not contain this logic; it clearly specifies the project objectives as outlined above. IEG/OPCS guidelines state that a project is to be assessed on achievement of its objectives, not its key performance indicators. In this case, although the ICR makes a strong case that institutional strengthening occurred, it does not provide direct measures of achievement of the project's objectives. Data on student achievement, employment of graduates, and faculty hiring, broken down by state, project versus non-project institution, and pre-project versus during-project, would usefully address achievement and attribution questions, but the ICR does not provide this information. The ICR itself, in its first "lesson and recommendation" (pp. 30-31), acknowledges this need, observing that "measuring student learning is central to the measurement of quality in higher education," and that "valid and standardized student assessments are needed to measure improvements in student learning." However, based on additional evidence provided by the project team on the metrics used for institutional accreditation, and the comparison of accreditation scores of project versus non-project institutions, there is sufficient evidence to conclude that the project had a Substantial impact on the quality and employability of Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) students. There is not similar evidence presented in the ICR on reduction in faculty shortages due to project activities. Given the strong empirical evidence on institutional accreditation, the achievement of the first objective is weighted slightly higher than the second in deriving the overall Efficacy rating. Overall Efficacy is therefore rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency The PAD's economic analysis (pp. 68-74) examined the project's cost-benefit elements, the rationale for public investment in this area, the empirical evidence on the impact of two key project activities (competitive funding and increased institutional autonomy), the premium realized from increased skills and research/development, and estimated rates of return. It demonstrated that institutions with more competitive funding and increased autonomy score higher in international rankings. It also showed that the Indian labor market had been increasingly rewarding higher education for at least two decades. The PAD's cost-benefit analysis found an internal rate of return ranging from 3% to 43%, depending on the impact of the project on skills and therefore wages of graduates. Its assumptions, including a carefully detailed description of the pathways through which inputs (costs) would lead to benefits, discount rates of 5% and 10%, and a range of specifications of the average and incremental wage, working lifetime, and total number of potential student beneficiaries, were reasonable and well justified. A brief financial analysis demonstrated that the project contributed about 1% of available resources for India's 11th Five-Year Plan for higher and technical education. The ICR's calculated ex post internal rate of return ranges from 7% to 72% under different scenarios, using assumptions similar to those used in the PAD, with the higher result than estimated at appraisal resulting from the higher than planned number of beneficiaries and lower US dollar costs due to exchange rate shifts (ICR, pp. 43-45). Neither the PAD's nor the ICR's analysis takes into account the economic benefit to society of well-trained engineers (for the first objective), or assesses the economic costs of engineering faculty shortages versus the benefits of reducing those shortages (for the second objective). Implementation efficiency contained some positive elements, with the project able to cancel some loan funds due to savings from effective use of an e-procurement system. Similarly, technology was used to create a performance-based benchmarking system to track implementation progress and incentivize project institutions toward improved performance. The ICR (pp. 19-20) also notes that most institutions successfully fulfilled their procurement plans for laboratory equipment, software, books, and equipment. However, there were some implementation inefficiencies. According to the ICR (p. 12), the first round of selection of institutions took about a year to complete, with the process drawn out by: (a) excessive inquiries from state secretaries, institutional directors, and parliamentarians about the possibility of lowering eligibility criteria; (b) the long and cumbersome nature of required institutional development plan proposals; (c) requests from several institutions for reconsideration of proposals, requiring a review process; and (d) formal expressions of disagreement with selection decisions from private institutions. The second round of selection, however, Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) experienced more rapid turnaround. Flow of funds and disbursement delays were a challenge throughout the project period (see Section 10b), mostly due to constraints in state treasuries and shortcomings with procurement approval processes; the National Project Implementation Unit contributed significantly to addressing these challenges. Finally, spending at the institute level tended to be divided equally across participating departments, with low cooperation between departments; this approach led to "sub-optimal utilization of resources" (ICR, p. 49). Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of Objectives is rated Substantial, as the objectives were aligned with country context, Bank strategy, and government strategy at the time of project closing and were appropriately ambitious for a second-phase project. The project clearly made significant contributions to institutional strengthening that led to improved accreditation scores for project versus non-project institutions, and the metrics for accreditation encompassed student quality and employability. Achievement of the objective to produce more employable and high-quality engineers is therefore rated Substantial. However, although the project increased the number of engineers with graduate-level training, and a large percentage of those graduates took up faculty positions, there is not sufficient evidence to demonstrate an alleviation of faculty shortages. The Modest rating for the second objective therefore emerges not from evidence of low achievement, but from insufficient evidence of achievement of the objectives as stated in the Financing Agreement. On balance, the first objective is weighted slightly higher than the second, due to the convincing empirical evidence on institutional accreditation, leading to an overall Efficacy rating of Substantial. Efficiency is rated Substantial based on the internal rate of return and purported savings from the use of electronic procurement and other systems. Taken together, these ratings are indicative of minor shortcomings in the project's preparation and/or implementation, resulting in an Outcome rating of Satisfactory. a. Outcome Rating Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) Satisfactory 7. Risk to Development Outcome Although propensity and energy for reform varies from state to state, overall political support for engineering and technical education is solid and stable. The institutional strengthening achieved under the project appears to be robust and unlikely to be reversed. Boards of Governors for each institution continue to oversee and guide strategic decisions. The project contributed only about 1% of total spending for India's 11th Five-Year Plan for higher and technical education, indicating that financial sustainability is likely in terms of available government resources. In addition, institutions participating in the project were required to set up funds in four areas -- Corpus Fund, Faculty Development Fund, Equipment Replacement Fund, and Maintenance Fund -- specifically to continue reform activities and sustain gains after the project period. The ICR (p. 22) reports that US$150 million has been accumulated in the Corpus Fund alone. A US$ 201.5 million third phase of the Bank- supported project was approved in 2016, intended to support quality and equity in engineering education in 200 institutions across several focus states. The National Project Implementation Unit for the project will continue to provide support to and monitor participating states and institutions. 8. Assessment of Bank Performance a. Quality-at-Entry The lending instrument, a SIL, was appropriately chosen to allow for potential further scale-up through additional financing or another SIL, and because of the embedded hands-on technical assistance and monitoring. Key lessons were learned from the earlier project, centered on the following needs: focus on capacity building to ensure effective implementation of reforms; increased availability of pedagogical training; focus strongly on project implementation; and enhanced monitoring and evaluation. Sixteen background studies and reports (11 by the government and 5 by the Bank) were completed to inform project preparation. Overall risk was rated Moderate, with some Substantially risky elements, including reluctance of participating states and institutions to introduce reforms, insufficient implementation capacity due to high staff turnover, inadequate decentralization and oversight, and financial management/procurement challenges. Mitigation measures included building on policies and procedures established under the previous project, hiring new staff in areas critical for capacity enhancement, investing in new management information systems, outsourcing much of the project's training and oversight so that the project implementation unit's focus remained on more strategic tasks, and building evaluations into every level of project implementation. However, there were shortcomings. The project's key outcome indicators were inadequate and imprecise measures of achievement of its development objectives (see Section 9a). The ICR (p. 24) notes that more detailed discussions earlier in the preparation process with state governments (from whom there was resistance based on the mistaken idea that institutional autonomy was the same as privatization) might have better mitigated the risk of political resistance to reform. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Supervision missions were regular, well planned, and staffed with appropriate specialists. Effective support was provided to build capacity of implementing agencies on financial management and procurement. Implementation Status Reports were candid and focused on identification of solutions that would produce results. The introduction of performance benchmarking for individual institutions was an effective innovation. Restructuring to modify financing arrangements and revise indicators and targets was carried out when necessary. The ICR (p. 30) notes that there was continuity of supervision during transitions in project leadership, though no details are provided. The project team did not, however, identify the lack of indicators sufficient to measure achievement of the project's objectives as stated in the legal agreement. The ICR does not provide information on adequacy of supervision of compliance with safeguard policies. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project's objectives were clearly stated, and the theory of change, building on the predecessor project, was sound and well reflected in the results framework. However, the key performance indicators did not measure the development objectives as stated in the financing agreement; instead, they measured outputs toward the achievement of those objectives. There were no indicators for employability or quality of engineers who were trained by the institutions supported by the project. Similarly, there were no indicators for reduction of faculty shortages. The ICR (p. 25) states that institutional accreditation is based in part on student learning and employment outcomes, and therefore is a legitimate proxy for quality and employability of graduates, but no specific data are offered in the ICR to support this claim. The project team later provided detailed information on the metrics used during the accreditation process, convincingly tying those metrics to achievement of the objective on employability and quality of engineers. The output indicators that were used were specific, measurable, achievable, and time-bound. Baselines and targets were appropriately specified in the PAD. Systematic monitoring was to be achieved through a web-based Management Information System (MIS) with access at the national, state, and institutional levels. Sampling methods, data collection methods, and analysis were appropriate for all indicators. Verification was to be provided by third parties on a sample basis. Academic data was to consist primarily of information that each educational institution routinely provides to regulatory bodies (PAD, p. 11). Student and faculty surveys were planned for each institution, as was a Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) bibliometric study summarizing each institution's national and international publication records. Surveys requesting feedback from employers on the quality and employability of graduates were to be conducted every two years. M&E capacity at the national and state levels was to be strengthened by designating staff specifically for project M&E activities. b. M&E Implementation Data on implementation progress and achievement was collected and entered by project institutions into the MIS. 181 participating institutions reported at least 70% of formal project indicators at closing, exceeding the original target of 150 and the revised target of 180. Reported data appear to be reliable and of satisfactory quality. Modifications were made to some specific indicators (particularly the indicator on accreditation) as appropriate. A performance benchmarking system was put in place to incentivize attention to data at the state and institutional levels. Most additional surveys and studies were carried out as planned. Given that the bulk of data collection took place through routine educational institutions to regulatory bodies, it seems likely that core M&E functions and processes are likely to be sustained after project closing. However, significant initial shortcomings in M&E design, most importantly the specification of results indicators that directly measured achievement of the project's objectives, were not corrected during implementation. Furthermore, the project did not take advantage of testing data that already existed, e.g. the "Aspiring Minds Computer Adaptive Test" mentioned under Objective 1 (Section 4). c. M&E Utilization Progress as reported in the MIS and from the various assessments and surveys was shared and discussed during Joint Review Missions and used to assess the status of accreditation and autonomy in project institutions. Institutional report cards were issued to incentivize corrective actions where necessary, and although the ICR (p. 27) reports that performance benchmarking brought about a "notable improvement in institutional performance," it also states that "utilization of M&E information at the state/institutional level was modest." Overall, since indicators were designed only to provide evidence of achievement of outputs, M&E data were not sufficient for full demonstration of achievement of outcomes, either incrementally during project implementation or at closure. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as environmental category "B" because of the nature and magnitude of potential environmental impacts from academic research and from limited renovation and refurbishment activities. Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) Safeguard policies OP/BP 4.01 (Environmental Assessment) and OP/BP 4.10 (Indigenous Peoples) were triggered. No adverse environmental impacts were anticipated, and therefore an environmental assessment was not required (PAD, pp. 16-17). To ensure positive impacts from upgrading of infrastructure, an Environmental Management Framework was prepared and its provisions incorporated into relevant project operational manuals. An Equity Action Plan (PAD, pp. 85-88) was also prepared to ensure that all students, especially those from Scheduled Castes, Scheduled Tribes, Other Backward Castes, and other disadvantaged groups, would have equal opportunity to benefit from project activities. Each participating educational institution was to describe an Equity Action Plan in its Memorandum of Understanding with the project, including diagnostics to identify student weaknesses at the time of entry, plans to address those weaknesses, provision for bridge courses/remedial teaching, and appropriate pedagogical training for faculty. Gender-specific needs were to be taken into account for facility upgrades and training. The ICR (p. 28) notes that project monitoring indicators were used to assess implementation of various activities in the Equity Action Plans. It also states that although four regional workshops were organized to discuss implementation of the Environmental Management Framework under the project, there were only limited efforts toward dissemination of this information across project institutions. The ICR does not state whether there was compliance with the Bank's safeguard policies; the project team later confirmed that the project was in full compliance. b. Fiduciary Compliance Financial management: The project instituted an electronic Financial Management and Reporting system for efficient reporting and consolidation of reports. Funds flow was through state treasuries, to ensure strong commitment to the project at the state level. There was a dedicated senior officer responsible for financial management in the National Project Implementation Unit. However, there were challenges at both the state and institutional levels. Delays in releases of funds from states to institutions and inadequate budget provisions for the project in some states were "major and persistent challenges" during the initial stages of implementation (ICR, p. 28). High turnover of staff in project institutions hindered the implementation of the electronic reporting system. Despite these shortcomings, however, the project disbursement rate was very close to 100%. The ICR does not report on the timeliness or results of audits. Procurement: An electronic procurement system was introduced to bring about greater efficiency and transparency. There were significant challenges in the operation of this system, including a five-month period during which manual procedures were used because the vendor supporting the electronic system went bankrupt. Approval processes were hindered by excessive requirements for and a lack of clarity around those requirements; attempts at streamlining these processes were only partially successful. The ICR (p. 9) states that there were cost savings to the project due to the use of electronic procurement, but these are not quantified. Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Use of a proxy indicator for the first objective leads to a rating of Substantial rather than High on that objective; insufficient evidence of Outcome Highly Satisfactory Satisfactory achievement of the objective as stated in the Financing Agreement leads to a rating of Modest for the second objective. There were shortcomings in the results framework and risk mitigation at preparation. The Moderately shortcomings related to Bank Performance Satisfactory Satisfactory inadequacy of key performance indicators were not addressed during implementation. The project's key performance indicators did not capture fully the contribution of the project's activities toward achieving Quality of M&E Substantial Modest outcomes as expressed in the objectives, a shortcoming that was not rectified during implementation. Quality of ICR Substantial --- 12. Lessons The ICR (pp. 30-32) offers a number of useful and important lessons, modified slightly here: Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) Quality improvement involving increased autonomy and institutional leadership can stem from a variety of factors. Support must therefore be based on a thorough and specific assessment of the challenges and constraints impacting each individual institution: limited capacity, limited initiative, weak state support, or other considerations. Accreditation mechanisms depend on prerequisite capacity building for quality assurance mechanisms. In this case, some institutions applying for accreditation experienced significant delays because of changes in accreditation procedures and limited supply of qualified accreditors. IEG offers an additional lesson: Focus on outcomes other than those in the project objectives as stated in the legal agreement can result in inadequate measurement of progress and achievement. In this case, the project defined its objectives in terms of institutional strengthening, even though the Financing Agreement committed the project to more outcome- oriented objectives. Consequently, the project did not collect and report data sufficient to demonstrate high efficacy. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is generally well written, internally consistent, and consistent with the new ICR guidelines. The evidence on achievement of outputs and progress along the project's formal indicators is strong, credibly sourced, and appropriately referenced. There are clear links between the evidence and findings on achievement of outputs. The visualization of the project's results chain (p. 7) clear and effective. It was useful to compare results from a 2015 study of engineering graduates across the country with results of the tracer study to check employment status of 2013 and 2014 graduates from project-supported institutions. However, there are shortcomings. An important structural shortcoming is that the ICR presents an assessment of achievement based on indicators, rather than using those indicators as the basis for an assessment of achievement of the project's objectives. The lessons presented, while highly interesting and potentially useful, seem in some cases not to be derived from experiences described earlier in the document. The ICR does not indicate whether there was compliance with the Bank's safeguard policies. On balance, however, the ICR's strong presentation of the results chain, evidence, and implementation experience is sufficient to merit a rating of Substantial. a. Quality of ICR Rating Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Tech Engr Educ Qual Improvement II (P102549) Substantial Page 17 of 17