Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) Report Number: ICRR0021744 1. Project Data Project ID Project Name P106216 BD: Higher Education Quality Enhancement Country Practice Area(Lead) Bangladesh Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-45440,IDA-53320 31-Dec-2013 191,938,252.75 Bank Approval Date Closing Date (Actual) 17-Mar-2009 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 81,000,000.00 0.00 Revised Commitment 205,812,964.42 0.00 Actual 196,426,533.23 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Project Appraisal Document (PAD, page 4) and the Financing Agreement (page 5), the project objectives were as follows:  To improve the quality and relevance of the teaching and research environment in higher education institutions, through encouraging innovation and accountability within universities and by enhancing the technical and institutional capacity of the higher education sector. Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) The project objectives remained the same throughout the project period, although key outcome targets were briefly revised downward during a January 2013 project restructuring, due to slow implementation progress. However, the targets were subsequently revised upward at the time of Additional Financing (AF, within one year), and therefore a split rating is not applied. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 03-Dec-2013 c. Will a split evaluation be undertaken? No d. Components 1. Promoting Academic Innovation (Appraisal: US$ 50.2 million; AF: US$ 48.7 million; Actual: US$ 116.1 million): This component aimed to introduce a demand-driven funding mechanism to allocate public funds to higher education institutions, with an emphasis on innovation and accountability. An Academic Innovation Fund (AIF) was to provide grants to eligible public and private universities on a competitive basis, according to clearly identified selection criteria and procedures. The three funding windows were as follows: improvement of teaching and learning; enhancement of research capabilities; and university-wide innovations. The overall aims of the AIF were to improve the quality of academic activities and outcomes, promote voluntary self-assessments, and strengthen universities' linkages with national development efforts. 2. Building Institutional Capacity (Appraisal: US$ 4.4 million; AF: US$ 4.6 million; Actual: US$ 6.8 million): This component aimed to reinforce the strategic and institutional capacity of the higher education sector, both at the central level and at the institutional level. Activities were to support both the University Grants Commission (UGC) and individual universities, and included the following: capacity building for strategic planning and management; development of a Higher Education Management Information System (HEMIS); communication campaigns about the AIF; and technical assistance in preparing subprojects for the AIF. 3. Raising the Connectivity Capacity of the Higher Education Sector (Appraisal: US$ 34.0 million; AF: US$ 44.2 million; Actual: US$ 45.9 million): This component aimed to integrate universities with the global knowledge community. Activities included: establishment of the Bangladesh Research and Education Network (BdREN); and establishment of a digital library network. 4. Project Management, Communication and Monitoring and Evaluation (Appraisal: US$ 3.2 million; AF: US$ 11.6 million; Actual: US$ 16.4 million): This component was to support project management through the Project Management Unit (PMU). AF was approved to scale up activities, as well as to meet the financing gap for AIF. The project objectives were unchanged, while the original components were scaled up and one new component was added. Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) According to the Interim Impact Assessment conducted in June 2013, the project was on track to achieve its development objectives despite initial delays, including achieving targets for satisfaction levels of stakeholders. The study recommended areas of support to scale up impact, including conducting an awareness-raising campaign to increase utilization of facilities and digital services, developing university- industry linkages, and establishing a separate entity to oversee quality assurance. The following additions were made to the components: 1. Promoting Academic Innovation: A third round of grants from the AIF was created, with a new window named the "Innovation Fund" to support stronger university-industry linkages. 5. Establishment of Quality Assurance Mechanism (AF: US$ 37.0 million; Actual: US$ 15.8 million): This component aimed to ensure quality of higher education through the establishment of quality assurance mechanisms. Activities included: institutional strengthening of the Quality Assurance Unit in the UGC; and establishment of quality assurance cells at selected higher education institutions. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost  The original project was appraised at US$ 91.8 million. With the approval of AF, the total appraised cost was US$ 216.5 million, and the actual total cost was US$ 216.8 million.  Funds were reallocated to Component 3 (Raising Connective Capacity) from other components to meeting additional funding needs for the BdREN. Financing  The original project was financed largely by an International Development Association Credit of US$81.0 million.  AF in the amount of US$ 125.0 million was approved in December 2013, in order to scale up activities and to meet the financing gap for uncompleted activities. Borrower contribution  At appraisal, the Borrower contribution was expected to be US$ 10.5 million. Following project restructuring, the total Borrower contribution was appraised at US$ 21.3 million. The actual amount was US$20.4 million. Dates  January 2013: The project was restructured to streamline activities and modify the results framework (including key outcome targets) due to the slow pace of implementation. The closing date was extended from December 2013 to October 2015. At the time, US$54.5 million of the Credit, or 67.3%, had disbursed.  December 2013: Following the January 2013 project restructuring, the pace of implementation improved significantly and all funds under the original Credit were disbursed. Additional Financing in the amount of US$ 125.0 million was approved to build on the reforms and programs initiated during Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) the original project period. At the time, US$ 76.0 million, or 93.8%, had disbursed. The closing date was extended to December 2018. 3. Relevance of Objectives Rationale Bangladesh has made notable progress in poverty reduction and improved human capital outcomes in recent decades, particularly in comparison to countries at similar per capita income levels. In education, primary and secondary enrollments have increased significantly, along with gender parity. At the time of project appraisal, the higher education sector included 81 universities, of which 30 were public and 51 were private, accounting for 17% of total enrollments in the higher education sector (colleges accounted for 75%, and Open University accounted for the remaining 8%). The Ministry of Education (MOE) is responsible for policy, strategy, and budgeting for public funding. Within the MOE, the UGC was created in 1973 as the oversight apex body, serving as the intermediary between the government and universities. However, the sub-sector is marked by low quality, limited access, low level of research, inadequate governance, and weak institutional capacity. These challenges have been a significant impediment to economic growth and favorable investment climate. The Government's Higher Education Strategic Plan for 2006-26 was prepared with participation of multiple stakeholders and was viewed as a significant shift by the government to commit to investing in higher education and addressing long-term challenges. The Bank's Country Partnership Strategy for FY16-20 identified quality improvement in education, including increasing relevance of higher education to respond to labor needs, as a priority area. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve the QUALITY of the TEACHING environment in higher education institutions Rationale According to the project's theory of change, the provision of grant funding to schools through a competitive grant mechanism, the networking of institutions through a research network and digital library, and the introduction of a quality assurance framework, all supported by institutional strengthening activities, were likely to contribute to the outcomes of improved quality and relevance. The emphasis on demand driven Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) activities (Academic Innovation Fund), in particular, was likely to have immediate impact on relevance, as strong demand indicates the areas of most need or interest. With regard to quality, project activities (Academic Innovation Fund and capacity building) were also likely to improve quality in the short term for teaching and research environment/conditions and for promoting innovation and accountability. Outputs Academic Innovation Fund Provision of financing to 439 subprojects in 38 private and public universities through the AIF. The AIF used a transparent competitive funding mechanism, which the post-evaluation found had 100% of fund allocations adhering to the transparent procedures and judged by "renowned experts and competent authority." The subprojects led to the following improvements: upgraded labs and education infrastructure, increased connectivity to high speed internet, enhanced teaching capacity and tools, and modernized curricula. Specifically, the ICR reported the following outputs:  27,051 students and staff trained in academically-oriented areas such as research methodologies, proposal writing, statistics, technical knowledge for specialized equipment, curriculum development, and pedagogy  15,350 pieces of information and communications technology equipment for teaching installed across 38 universities  41,184 books and journals procured for offices/libraries  1,653 classrooms/labs/offices renovated  12,012 faculty members faculty receiving additional training in various disciplines  618 Masters degree students enrolled  9 libraries modernized and automated with modern library management system  103 curricula updated Digital Connectivity  Digital library platform with subscriptions to over 3,000 e-journals  Establishment of the BdREN, connecting 40 universities (target: 60)  Establishment of the HEMIS, with annual statistical yearbooks being produced Quality Assurance  Creation of Quality Assurance Unit within the UGC to promote good practices and governance  Establishment of 69 Institutional Quality Assurance Cells (IQAC) at 28 public and 41 private universities (target: 15), producing 810 self-assessment reports, which set the benchmark on quality and propose strategies for attaining those quality goals.  Drafting of a National Qualifications Framework, which provides the foundation of an accreditation process Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) Outcomes  The mean satisfaction level among students* increased on the five-point Likert scale from 3.3 in March 2009 to 3.7 in December 2018. This fell short of the revised target of 4.0. The ICR (page 20) suggested that the shortfall was due to the fact that student cohorts sampled were, by nature of the survey, always new cohorts who have had only a few years in university and thus can only make relatively short-term comparisons about improvements in the learning environment. As such, their satisfaction levels are more likely to be modest compared to faculty members and employers who tend to base their perception on longer-term comparisons. Furthermore, it is also possible that undergraduate students have limited research activities and thus did not directly benefit from improvements in the research environment where much of the project investments were directed.  The mean satisfaction level among faculty increased on the five-point Likert scale from 2.9 in March 2009 to 4.0 in December 2018. This achieved the revised target of 4.0. The ICR (page 18) cited particularly high levels of satisfaction regarding access to online journals and e-resources, quality of internet, and availability of modern lab equipment and facilities. Annex 8 of the ICR presented evidence, largely qualitative, from the Final Round Satisfaction Survey, Graduate Tracking Survey, and Final Impact Assessment. The findings included the following: the AIF-financed technology improved the learning environment for students and enhanced faculty's ability to teach more advanced concepts. Faculty members in AIF departments had higher satisfaction levels than the control group of faculty from non-AIF departments, owing to the improved quality of infrastructure and materials. However, one finding was that "students in AIF departments did not survey higher survey satisfaction across a number of common indicators of pedagogy (such as quality of instruction), but AIF departments were found to more effectively use technology in instruction" (ICR, page 64).  17 institutions submitted institutional improvement plans, in addition to the 810 self-assessment reports produced by the 69 institutional quality assurance cells, which led to the implementation of a range of quality improvement activities.  The Bangladesh Accreditation Council Act was passed by National Parliament in 2017, establishing a statutory autonomous entity for accreditation of universities and programs. The project team confirmed that the Endowment Fund for the Council has been established to provide operating budget and that the Council is now operational and preparing for the initiation of the accreditation procedure. * The survey methodology was described in ICR Annex 8; this description included numbers of departments and individuals surveyed but lacked specific information on response rates (or on actual numbers or responses received either through the survey or in focus groups). Achievement is rated Substantial. There was improved satisfaction of faculty and the delivery of numerous outputs that likely improved the quality of the teaching and learning environment. Although there were moderate shortcomings in the implementation of national quality assurance framework, quality improvements were still carried out at the institutional level. Rating Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) Substantial OBJECTIVE 2 Objective To improve the RELEVANCE of the TEACHING environment in higher education institutions Rationale Outputs See outputs reported above on subprojects approved through the AIF and digital connectivity. Outcomes  The mean satisfaction level among employers increased on the five-point Likert scale from 3.0 in March 2009 to 4.1 in December 2018. This nearly achieved the revised target of 4.2. Employers noted satisfaction with customer service skills, critical thinking and analytical skills, communication in English, advanced computer skills, and willingness to learn. The graduate tracer survey reported that AIF projects were effective in improving skills, job search prospects, and employability; however, "students from AIF-supported institutions reported a high rate of unemployment (38%) among all university graduates in Bangladesh" (ICR, page 66).  However, there was a "considerable lack of academia-private industry collaboration. Less than half of surveyed employers maintain collaboration with universities and even less in any sort of sustained way. However, from the academic institution side, most mentioned maintaining some kind of relationship with industries" (ICR, page 67). Achievement is rated Substantial due to improved employer satisfaction and increased utilization of digital resources, although a minor shortcoming is noted in the limited academia-industry collaboration. Rating Substantial OBJECTIVE 3 Objective To improve the QUALITY of the RESEARCH environment in higher education institutions Rationale Outputs See outputs reported above on subprojects approved through the AIF and digital connectivity. Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) Outcomes  As reported above, the mean satisfaction level among faculty increased on the five-point Likert scale from 2.9 in March 2009 to 4.0 in December 2018. This achieved the revised target of 4.0. The ICR (page 18) cited particularly high levels of satisfaction regarding access to online journals and e- resources, quality of internet, and availability of modern lab equipment and facilities. Annex 8 of the ICR presented evidence, largely qualitative, from the Final Round Satisfaction Survey, Graduate Tracking Survey, and Final Impact Assessment. The findings included the following: the AIF-financed technology invigorated the research environment for faculty and provided researchers with greater capacity to conduct research and disseminate findings. Faculty members in AIF departments had higher satisfaction levels than the control group of faculty from non-AIF departments, owing to the improved quality of infrastructure and materials.  The original indicator on the percentage of students and faculty with access to advanced internet connectivity was dropped. Instead, the project reported that the monthly average volume of inbound education/research data traffic in BdREN increased from 10 terabytes in 2013 to 1,362 terabytes in 2018.  778 academic publications were produced through AIF support, of which 181 were papers published on conference proceedings. This surpassed the target of 150.  The number of doctoral level enrollments increased by 368% to 170 students, surpassing the target of 100%. Achievement is rated Substantial due to increased satisfaction among faculty, as well as other evidence of increased volume and productivity in research. Rating Substantial OBJECTIVE 4 Objective To improve the RELEVANCE of the RESEARCH environment in higher education institutions Rationale Outputs See outputs reported above on subprojects approved through the AIF and digital connectivity. Outcomes Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) See outcomes reported above for Objectives 2 and 3. Achievement is rated Substantial due to increased satisfaction among faculty and among employers as well as other evidence of increased volume and productivity in research. A minor shortcoming is noted in the limited industry-academia collaboration. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Achievement of the objective to improve QUALITY of the teaching environment is rated Modest. Achievement of the remaining three objectives to improve RELEVANCE of the teaching environment and the QUALITY and RELEVANCE of the research environment is rated Substantial due to evidence of improved faculty and employer satisfaction. Overall Efficacy is therefore rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency At appraisal (PAD, Annex 9), a cost-benefit analysis was conducted to compare the expected stream of incremental wage gains for project beneficiaries with the project costs. The estimated internal rate of return, under the base case scenario, was 17.9% for the AIF component. A cost efficiency analysis was also conducted for the BdREN activity, with the BdREN providing high-speed and high-performance connectivity to institutions at a significant cost savings compared to the current spending by the institutions on the current system (US$ 9- 10 million for 50 Mbps vs. US$ 4 million for 100 Mbps through BdrEN). The ICR (Annex 4) updated the cost-benefit analysis, under the assumption that 20% of total eligible university graduates had benefited from the project. Benefits were identified as: additional wage premiums for university graduates (the wage differential in entry-level salaries) alongside growth in university enrollments. The net present value was calculated at US$ 728.0 million, and an internal rate of return of 52.8%. However, there was limited evidence on the extent to which the project actually improved the skills of the graduates that were to have led to improved job prospects and increased wages. Other indications of efficiency in the use of project resources were suggested: the use of specialized expertise (for technical work in the digital network, quality assurance, and intellectual property) contributed to project Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) efficiency. In comments on the draft ICR Review, the project team reported that 2.7% of total project costs was used for technical assistance and that the ambitious reforms supported by the project would have likely experienced setbacks (and hence further implementation delays) had this specialized expertise not been available; and the digitization of annual data collection from universities replaced time-consuming manual data collection. Also, despite a project closing date extension of almost two years for the original project period, US$ 76.0 million of the original credit (94%) had been utilized at the time of the original closing date. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 54.70 Appraisal  17.90  Not Applicable 100.00 ICR Estimate  52.80  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the project objectives is High due to strong alignment with country conditions and Bank and country strategies. Overall Efficacy is rated Substantial due to evidence of improvements in quality and relevance of the teaching and research environment. Efficiency is rated Substantial. Therefore, overall outcome is rated Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The two primary risks to development outcome are institutional capacity and financial sustainability. With regard to institutional capacity, the skills to implement the innovation fund and develop quality assurance mechanisms have been introduced into the MOE and UGC, although it is unclear whether quality assurance, despite the passage of the Accreditation Act, will be carried out by the individual tertiary institutions without continued support. With regard to financial sustainability of the numerous activities funded by the AIF, each entity was supposed to create a maintenance plan for equipment and facilities to ensure sustainability, although the final surveys highlighted the beneficiaries' concern for continued financing to sustain benefits of the project. Endowments were created for the BdREN and Accreditation Commission, as well as a dedicated Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) budget line for quality assurance activities at publicly funded universities. Lastly, a follow-up operation, the Higher Education Acceleration and Transformation project (P168961), is currently under preparation and aims to support similar outcomes. 8. Assessment of Bank Performance a. Quality-at-Entry This project represented the first Bank operation in the higher education sub-sector in Bangladesh, a sub- sector in which few other development partners had interest and/or capacity to provide effective support. Despite being the first operation of its kind in the country, the Bank drew upon its significant experience in other countries, particularly in establishing innovation funds to promote research and innovation. As noted in the ICR (page 6), the project objectives and design were intentionally designed to be a "low reform-intensive project avoiding controversial measures... it was anticipated that opportunities and consensus for pursuing reforms in higher education will arise" once sufficient wins and strong engagement were demonstrated under this project. Therefore, government and higher education institution commitment was strong throughout the project period, although some time was needed to develop support for the demand-driven funding approach. The risk assessment was candid and realistic, identifying capacity constraints as the main risk. The results framework was overall satisfactory, with opportunities for improvement in terms of more specific and explicit outcomes/indicators for "innovation" and "accountability" rather than the more general outcome of "improved quality." Quality-at-Entry Rating Satisfactory b. Quality of supervision The project experienced significant implementation delays at the start, due to the need to shift the mindset of key stakeholders from supply-driven to demand-driven funding mechanisms. There was also significant learning-by-doing once the new demand-driven approach was established, such as selecting competitive grantees. Also, there were numerous technical and logistical difficulties in establishing the BdREN across multiple institutions. Implementation Supervision Report ratings during this period, however, appeared to lack some candor, as "Implementation Progress" was rated in the satisfactory range in all but one supervision report. However, proactive supervision support helped to improve the pace of implementation, and AF was secured to significantly scale up the project's activities and impact. The first project restructuring (October 2013) revised targets that were unlikely to be achieved due to the slow start and also revised indicators themselves to make them more measurable. The second restructuring/additional financing accounted for the improved performance and adjusted targets upward again. Initial fiduciary challenges were addressed sufficiently to ensure delivery of outputs and project completion, although some shortcomings remained. Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The ICR (page 18) noted that the results framework opted to use satisfaction ratings to track improvements in quality and relevance to "overcome inherent difficulties of measurement of quality of higher education services. Assessing quality and relevance of higher education directly is known to be difficult due to the complex and diverse nature of higher education institutions and programs. High-level cognitive skills development of students is also extremely difficult to capture quantitatively." Opportunities for improvement remained in terms of identifying more specific and explicit outcomes/indicators for the aspects of "innovation" and "accountability. Also, baseline data was not established until two years into the project period. Notably, the M&E design included establishing an M&E Unit to support M&E implementation, which was critical for oversight given the newly introduced innovation fund mechanism. b. M&E Implementation The M&E Unit conducted project monitoring activities (with regular reporting on project progress, implementation issues, and possible solutions), carried out validation surveys of AIF sub-projects, and completed three rounds of satisfaction surveys. There were initial delays and inadequate documentation, which led to a downgrade in the M&E rating. However, these challenges were subsequently addressed. Restructurings were used to enhance the results framework to make targets realistic and indicators more measurable. For example, one of the original key outcome indicators - "Number of universities (public and private) connected to BdREN" - was revised to "Monthly average volume of inbound education/research data traffic in BdREN" to better measure impact. Six evaluative studies were also conducted, including the three rounds of the satisfaction surveys, interim impact assessment, university graduate tracer study, and project impact assessment. c. M&E Utilization The ICR (page 29-30) reported the following examples of M&E utilization: regular M&E reports were used to support implementation supervision missions by identifying key issues, and evaluative studies informed the MOE's strategic plan for the sector and design of a follow-up operation. The National HEMIS developed under the project produced data to inform areas of interest such as enrollment, teacher numbers, and budgeting. Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as an Environmental Category "B" project due to minor civil works/rehabilitation. Under the AF, the safeguard policy on Indigenous Peoples (OP/BP 4.10) was also triggered due to the potential for ethnic minorities to participate in project interventions (as beneficiaries). A Social Management Framework was prepared accordingly. At the time of the AF, environmental safeguards compliance was rated Satisfactory. There is no further information in the ICR about the level of compliance with the environmental and indigenous peoples safeguards and whether they remained satisfactory by project closing, although the project team subsequently confirmed that the project was in full compliance as of the final supervision mission. b. Fiduciary Compliance Financial management: There were challenges in financial management performance throughout the project period, including delayed, inaccurate, and ineligible financial reporting, ineligible expenditures, a resurfacing of hiring needs, and a slowdown in field-level fiduciary support. According to the ICR, although these fiduciary issues were never fully mitigated, they were addressed sufficiently that financial management performance was rated moderately satisfactory by project closing. Procurement: Procurement performance was generally satisfactory, although there were some delays in procurement actions and staffing. In 2015, episodes of civil unrest disrupted procurement for AIF subprojects as well as for the BdREN. Specific challenges included: an outdated procurement plan; and delayed procurement of key items due to the failure to hire requisite procurement specialists and necessary staff in the UGC to implement the HEMIS and BdREN, thereby delaying overall implementation of the BdREN, the campus network, and the digital library. Actions were undertaken to strengthen procurement capacity, and performance improved by project closing. c. Unintended impacts (Positive or Negative) Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Modest 12. Lessons Lessons drawn by IEG:  Difficulties in measuring higher education sector outcomes (i.e. improved skills of university graduates) can be addressed through multiple evaluative sources. In the case of this project, the M&E plan employed student, faculty, and employer satisfaction surveys (three rounds), supplemented by findings from a tracer study and an impact assessment.  Introducing a competitive grant mechanism can be an effective first step to engaging autonomous institutions, prior to attempting more complex sector reforms. In the case of this project, the grants aimed to spur innovation and accountability, while deeper quality assurance mechanisms (drafting of national qualifications framework and creation of quality assurance cells in each participating higher education institution) were introduced at a later stage. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BD: Higher Education Quality Enhancement (P106216) This project would make a good candidate for further project assessment, given the use of a innovative approaches -- in particular the AIF and the quality assurance mechanisms. In addition, a project assessment would provide an opportunity to observe any future impacts on the quality and relevance of tertiary education. 14. Comments on Quality of ICR The ICR presented a range of evidence, although the results framework created challenges for providing a clear analysis of the outputs and outcomes at different points on the results chain. Output achievements were clearly reported. Assessment of outcome achievements was based on "improved satisfaction" as a proxy of improved quality. The Efficiency analysis would have been strengthened with a stronger evidence base, specifically impact on wages for university graduates to verify benefits. Also, there were some negative findings reported in Annex 8 that were not reported in the main text but had a bearing on Efficacy: Annex 8 reported survey findings on university-industry collaboration that indicate less engagement from industries compared to universities, and less satisfaction from students regarding quality of instruction. In comments on the draft ICR Review, the project team noted that historically there was zero collaboration and therefore any level of engagement should be viewed as a positive improvement. There was also a lack of candor about missing baseline data, which ISRs reported were not established until two years into the project period. Lessons were presented as recommendations, without drawing out the specifics of what was learned from the project experience. Given the shortcomings noted (inconsistent findings in annexes not explained in the ICR, lack of candor about missing baseline data, and weak articulation of lessons), ICR Quality is rated Modest. a. Quality of ICR Rating Modest Page 15 of 15