IMPLEMENTATION COMPLETION AND RESULTS MEMORANDUM Thailand IDF Grant TF-99480 for Strengthening quality assurance and performance excellence in Thai higher education – P125572 Data Sheet A. Basic Information ICM Date: 01/04/2016 Region : EAST ASIA AND PACIFIC Country: Thailand Trust Fund Name: Strengthening quality assurance and performance excellence in Thai higher education Project ID: P125572 TF Number: TF-99480 Grant Amount: USD 200,000.00 Disbursed Amount: USD 102,664.93 Cancellation Amount: USD 97,335.07 Recipient: GOVERNMENT OF THAILAND Implementing Agency (ies): Office of Higher Education Commission Other Donors/External Partners: B. Key Dates Process Original Date Revised / Actual Date(s) Approval: 02/15/2011 03/31/2011 Effectiveness: 01/30/2015 Restructuring(s): Operational Closing: 01/30/2015 01/30/2015 C. Project Team Members Title At ICM At Approval Country Director: Constantine Chikosi Ulrich Zachau Practice Manager/Manager: Harry Anthony Patrinos Harry Anthony Patrinos Project Team Leader: Lars M. Sondergaard Lars M. Sondergaard ICM Regional Coordinator: Xiuru Huang Procurement Specialist : Sirirat Sirijaratwong Sirirat Sirijaratwong Financial Management Specialist : Malarak Souksavat Malarak Souksavat LOA Specialist : Legal Counsel : D. Ratings Summary D.1 Performance Rating (ICM) Overall Grant Performance Relevance Moderately Satisfactory Efficacy Satisfactory Efficiency Moderately Unsatisfactory Overall Outcome Performance: Moderately Satisfactory Risk to Development Outcome (Sustainability): Low or Negligible Overall Outcome Performance Bank Performance Moderately Satisfactory Recipient Performance Moderately Satisfactory D.2 Bank and Recipient Performance Rating Bank Ratings Recipient Ratings Ensuring Quality at Moderately Satisfactory See Overall Outcome Government: Entry: Performance Quality of Supervision: Moderately Satisfactory Implementing See Overall Outcome Agency/Agencies: Performance E. Results Framework Analysis Program Development Objective (from Program Document): Link to CAS and/or ESW (core diagnostics, e.g., CPAR, CFAA, PER, CEM, etc.), and relationship to Bank operations The development objective of this grant is to enhance the capacity of the Office of Higher Education Commission (OHEC) in the area of quality assurance and performance excellence in higher education. Revised Program Development Objective (if any, as approved by the original approving authority): PDO Indicator(s) PDO Indicator 1 Operational Manual for the assessment of performance excellence Measurements (Number or Text) Date Baseline Value Only an English version of the Baldrige 09/30/2011 Performance Excellence Criteria existed. Original Target Values The Operational Manual was translated 07/15/2014 (from approval into Thai in February 2013. OHEC is documents) updating it using lesson learns during initial workshop. The manual was used to conduct workshops and guide institutions to conduct their self- assessment reports. Formally Revised Target Values Actual Value Achieved Final EdPEx operational manual 12/30/2014 at Completion or Target approved by OHEC and fully Years operationalized Comments (incl. % achievement) PDO Indicator 2 OHEC staff and assessors trained in the new performance excellence framework. Measurements (Number or Text) Date Baseline Value None of the OHEC's staff and assessors 09/30/2011 are trained in using this the new performance excellence framework. Original Target Values More than 70 OHEC staff and assessors 07/07/2014 (from approval trained in the new performance documents) excellence framework apply skills in assessing 4-5 pilot public research universities. Formally Revised Target More than 70 OHEC staff and assessors 07/15/2014 Values trained in the new performance excellence framework apply skills in assessing 4-5 pilot public research universities. Actual Value Achieved 71 OHEC staff and assessors trained in 01/30/2015 at Completion or Target the new performance excellence Years framework apply skills in assessing 4-5 pilot public research universities. Comments (incl. % achievement) PDO Indicator 3 Public universities assessed using the new performance excellence framework. Measurements (Number or Text) Date Baseline Value EdPEx is only known to a few medical 09/30/2011 schools in Thailand. Original Target Values 4-5 public universities use EdPEx 12/31/2013 (from approval framework. documents) Formally Revised Target Values Actual Value Achieved 21 Faculties in 5 public universities use 01/30/2015 at Completion or Target EdPEx. Years Comments (incl. % achievement) F. Ratings of Project Performance in Progress Reports: No. Date DO IP Actual Archived Disbursements (USD) 1 02/28/2012 Moderately Moderately 0.00 Satisfactory Satisfactory 2 03/19/2013 Satisfactory Moderately 40000.00 Satisfactory 3 03/03/2014 Satisfactory Moderately 63893.05 Satisfactory 4 08/27/2015 Satisfactory Moderately 108600.95 Satisfactory G. Restructuring (if any) H. Disbursement Graph IMPLEMENTATION COMPLETION AND RESULTS MEMORANDUM Thailand IDF Grant TF-99480 for Strengthening quality assurance and performance excellence in Thai higher education – P125572 1.Grant Description, Development Objective and Design 1.1 Original Development Objective (DO) (as approved) DO: The development objective of this grant was “to enhance the capacity of the Office of Higher Education Commission (OHEC) in the area of quality assurance and performance excellence in higher education.” Three results indicators were designed to track this objective: 1) A full operational manual for the assessment of performance excellence prepared and approved by OHEC; 2) 50 OHEC staff and assessors trained in the new performance excellence framework.; and 3) Public Universities assessed using the new excellence framework. Short description: Since Thailand's Education Reform in 1999, universities have been required by law to put in place internal and external quality assurance systems. With the rapid increase in numbers of universities, OHEC has been focusing on putting in place internal quality assurance, focusing on “minimum standards” for universities. However, there has been a long overdue need to provide universities with more comprehensive tools to strengthen their internal quality assurance, and helping them to identify their strengths and weaknesses. With this in mind, OHEC did some groundwork a few years ago and selected the world-renowned Malcolm Baldrige Quality Framework, a tool to help universities strengthen the way they run their institutions by focusing on seven key dimensions of management (e.g. Leadership, Strategic Planning, results, etc.). In Thailand, this tool is known as EdPEx. In 2011, OHEC approached the World Bank (WB) to help introduce this tool in the Thai higher education system: (i) focusing on providing an operational manual for how to use the tool and (ii) by training university staff in the use of the tool as well as“assessors” to assess universities using the criteria provided in the tool. It was agreed that a US$200,000 three-year Institutional Development Grant (IDF) could help provide funding to train assessors through hands-on/direct experience with approximately 4-5 universities from a pool of nine leading research universities selected by OHEC. The grant became effective on January 30, 2012 and closed three years later. Upon completion of the project, it was envisioned that OHEC would share best practices, conduct a benchmarking exercise, set up a special unit for performance excellence in higher education, and provide awareness raising activities for university senior management in the area of quality assurance and performance excellence. Furthermore, it was envisioned that the performance excellence efforts would create stronger links between research universities and industry. 1 1.2 Revised Development Objective (as approved by RIC) and reasons/justification The development objective was not revised. 1.3 Link to Country Assistance Strategy (CAS) The IDF grant supported the second pillar of the Interim Strategy Note FY2011-2012 CAS (FY03- 05) to support Thailand's effort to broaden its economic competitiveness and strengthen institutional capacity, in this case OHEC and the universities). Thailand's level of human capital development was (and still is) viewed as a constraint to increasing competitiveness. In the area of education, there are indications that the overall quality and relevance of education in Thailand is declining relative to other Asian countries. This is borne out in notable shortages of skilled labor, a general mismatch between labor supply and demand, and low levels of innovations among Thai firms. Activities funded by this grant were designed to respond directly to the Government's desire to sustain its competitiveness. Thailand needs an economy where science, technology and engineering are integrated into the production process and where creativity, knowledge and design capability are embodied in well-educated skilled workers who are the main source of national prosperity. Making this transition will require improving the overall effectiveness of the university and performance excellence is at the heart of this improvement. As such the proposal was to contribute to improving quality of higher education, promoting accountability and good governance. An enhanced self-assessment process to identify maturity levels of universities according to their specific contexts would also allow for better targeted World Bank support. 1.4 Main Beneficiaries (Original and revised, briefly describe the “primary target group”, as well as any other individuals and organizations who benefited from the Grant) The main beneficiaries were the staff at OHEC and the 21 faculties (and the institutions they represented) and the individuals (approximately 70) involved in the training supported by this grant. 1.5 Grant Components 1) Component 1: Strategic Technical Assistance (cost US$ 140,000) This component was designed to train OHEC staff, assessors and 4-5 research universities in the effective use of EdPEx (using IDF funds to hire local and international consultants). Expected outputs and outcomes:  Trained individuals understand and support the use of the EdPEx framework  An operational manual developed, tested and formalized as an instrument to eventually be used by all universities in carrying out a self-assessment.  Number of OHEC staff and assessors trained in the new framework  Number of research universities assessed under the EdPEx performance excellence framework. 2 2) Component 2: Awareness Raising to Promote Performance Excellence Framework and Self Assessment: (Cost US$60,000) This component was designed to build on activities under Component 1 to allow OHEC (with the help of consultants hired by the IDF grant) to help raise awareness of the EdPEx performance excellence framework and the self-assessment process amongst Thai universities. This awareness-raising was envisioned to be done through a series of workshops with management and staff of both public and private universities. Expected outputs and outcomes:  Establishment of a performance excellence system with faculties initiating “self- assessments” as an integral part of that system. 1.6 Revised Components (if restructured) None 1.7 Other significant changes (In design, scope and scale, implementation arrangements, schedule, and funding allocations) The biggest design change was not hiring local consultants, as envisioned at the design stage. This decision resulted in less-than-planned disbursements. One local consultant was hired and worked as a project coordinator at OHEC for four months but when this person quit, no new person was hired as a replacement. As a result, by the end, only 54 percent of the grant was used – the bulk of which was used to finance an international specialist. The decision to only use half of the IDF resources was OHEC’s and did not affect the achievement of the development objectives. Early in the project, OHEC decided that it could meet the objectives laid out in this grant while relying more on its own resources and that of the universities who participated. More specifically, OHEC decided to use less IDF resources because it struggled to identify and hire two local consultants envisioned to be hired using IDF resources: a consultant who was to have worked as project coordinator in OHEC and a local expert who was expected to work hand- in-hand with the international consultant. After several unsuccessful recruitment attempts, OHEC decided to move forward without a project coordinator (using instead its own staff) and without a local expert, setting up instead a local “committee”, consisting of eminent experts providing their time for free, or for small “speaker fees” at events. While this implied spending less of the IDF grant, as will be discussed below, both decisions turned out to have helped make the project a success. 2. Key Factors Affecting Implementation and Outcomes (Describe) Two factors played an important role in affecting both implementation and outcomes: 1) The design of the grant was strong, having centered it on the preparation of an “operational manual” (for how to use EdPex) that would gradually improve during 3 training. This ensured that an internationally renowned instrument could be tailor-made to Thailand while, from the outset, making clear where the project was heading. 2) The second factor was the strong interest and ownership of OHEC and the faculties involved. One example of this high level of ownership was that OHEC more than doubled the resources for this work (e.g. to finance part of the workshop costs) by using its own (non-IDF) resources. Throughout the project, OHEC staff were intimately involved in managing this project (possibly as a result of not being able to identify a local consultant to work as a project coordinator), providing leadership and ensuring that the activities were fully aligned with OHEC’s other work. Moreover, because a local expert consultant was not hired, OHEC, instead, had to rely on academics providing their inputs and leadership on a voluntary basis. This seem to have resulted in a better outcome, reflected in more ownership of the new tool. 3. Assessment of Outcomes (Ratings are available in ICM Data Sheet) Rating Summary 1. Performance Rating (ICM) Overall Grant Performance Relevance Moderately Satisfactory Efficacy Satisfactory Efficiency Moderately Unsatisfactory Overall Outcome Performance: Moderately Satisfactory Risk to Development Outcome (Sustainability): Low or Negligible Overall Outcome Performance Bank Performance Moderately Satisfactory Recipient Performance Moderately Satisfactory 3.1 Relevance of Objectives (Relevance of grant objectives in relation to country needs/priorities and Bank strategies) Comments: Given Thailand’s need to upgrade its higher education sector, the PDO remained highly relevant throughout the project (and continues to be relevant). In terms of relevance for the Bank’s strategies: a Systematic Country Diagnostics exercise has recently been launched to inform a new Country Partnership Framework. 3.2 Efficacy (Extent to which the development objectives have been achieved) Comments: The development objective was “to enhance the capacity of the Office of Higher Education Commission in the area of quality assurance and performance excellence in higher education.” The PDO was fully met to the extent that the three results indicators (tracked during the course of the project) captured that objective. All of these indicators were met, or exceeded: 4 First indicator: a fully operational manual for the assessment of performance excellence has been prepared and approved by OHEC. Fully achieved. Second indicator: 50 OHEC staff and assessors trained (and shown to apply skills learn) in the new performance excellence framework. Fully achieved: Throughout this project, a total of 5 workshops (for each cohort of universities) have been held, training more than 70 assessors (in this new approach). A final workshop was conducted in December 2015. Third indicator: Lessons learned from the use of EdPEx framework in 4-5 public research universities documented for further improvement prior to sharing with the remaining universities. Achieved: Reflecting lessons learned from workshops, OHEC has revised the “Operation Manual” which will be used to conduct future workshops and guide future institutions in using the EdPEx tool. In addition to the stated results, the IDF grant proposal also hinted at more ambitious results related to “enhancing the capacity of OHEC in the area of quality assurance and performance excellence”, namely: OHEC would conduct a benchmarking exercise, and set up a special unit for performance excellence in higher education. And it was envisioned that the performance excellence efforts would create stronger links between research universities and industry. Since no indicators were identified to track progress on these dimensions, it is difficult to judge what progress was made during the IDF. However, by the end of the IDF, there were no “benchmarking exercises” and there was no special unit established; rather the work on EdPex is being done alongside other core activities. 3.3 Efficiency (Cost effectiveness, e.g. benefits and costs compare with the expectations when grant was made, to what extent the grant is adding value to the client country and other partners work programs in the country/region) Comments: The IDF grants role was more important by the knowledge and outside experience it provided as opposed to the money it brought on the table. Specifically, the grant’s resources helped provide international experience (the tool itself, and the international expert who was hired to help train OHEC staff and assessors). Looking back, though, it would have been far more efficient to bring that knowledge and consultancy expertise to OHEC through the use of OHEC’s own resources, and possibly complemented with regular World Bank resources (e.g. through the use of World Bank’s regular budget and through the use of a technical assistance task), or by contracting the World Bank directly (under a fee-for-service arrangement). In the end, OHEC only ended up spending a total of US$102,664.93 (from the IDF grant) over three years, a process which required an inordinate amount of additional work on their side (to get training in Client Connection, learn Bank procurement rules, learn how to set up a designated account, make withdrawals, replenish the account, prepare interim financial reports, learn how to account for bank transaction fees and the exchange rate, get auditor signoff, etc.) and, correspondingly, work to provide that support on the World Bank’s side. To the extent that 5 this work built “capacity” in OHEC to improve their own project management, this was useful (and in line with the PDO of the grant). However, given the differences in systems, more often than not the capacity build was on how the Bank’s systems worked. For a small, one-time engagement of US$102,664.93, this time could have been better spent on the client and on the World Bank’s side. 3.4 Overall Outcome Performance Rating (Combining relevance, efficacy and efficiency) Comments: Overall, the outcome was moderately satisfactory. All the activities under the grant were carried out successfully and the results indicators – as spelled out at the outset – were fully achieved. It was not, however, an efficient way of providing this knowledge and technical assistance. Asking an upper middle income client to become proficient with World Bank procedures to execute a one-time grant of US$200,000 caused a lot of scarce time being spent learning and discussing World Bank procedures – time that should have been spent discussing how to improve the capacity of OHEC in quality assurance and performance management. 4. Risk to Development Outcome Comments: The risk of non-sustainability is rated low. Although OHEC choose not to establish a unit focusing exclusively on “performance excellence” (as envisioned at the outset), there are strong signs that EdPex will expand further in the years to come. Below some of these signs are discussed in more details. OHEC has prepared a future plan for EdPex for the years 2016-2020. This plan was presented and discussed twice at various sub-committee meetings on EdPex (in January and February of 2015). And, in April of 2015, OHEC announced this plan to expand EdPex to all 150 universities in the countries. Moving forward, OHEC’s aim is to roll out the tool to 10-15 faculties/universities per year, hoping to reach 75 faculties over five years. More broadly, OHEC has split up the 150 universities into separate groups (reflecting varying capacity) and plan on supporting them accordingly. Moreover, OHEC has prepared for the end of the strong hand-holding provided by the international consultant (hired by the IDF grant). Specifically, the sub-committee which was established to work with the international consultant now has a sufficient number of people who fully understand the tool and will be able – together with OHEC-- to play the role which the international consultant did during the past three years. 5. Bank and Recipient Performance (Please describe performance as ire relates to design, implementation and outcome issues consistently with the ratings assigned in the ICM Data Sheet) 6 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry (i.e., performance through lending phase) Description: Moderately satisfactory. The grant was well-designed: it was simple in design, with a manageable but relevant set of results indicators, and it was well-aligned with CAS and OHEC’s needs. (b) Quality of Supervision (Including fiduciary aspects) Description: Moderately satisfactory. No big issues arose during implementation, and OHEC decided to substitute its own resources for part of the IDF grant so it was a relatively easy task to supervise. At mid-term, there was a discussion as to whether to cancel the remainder of the grant (in the face of the difficulties OHEC faced with Bank procedures). However, at that stage, most of the training had already been provided, the main consultant contract to the international consultant had already been issued – and the remaining bureaucratic hurdles (submitting IFRs, audits, closing down account) had to be cleared irrespective of whether the remainder of the grant was cancelled. As such, the team and counterparts decided to continue with the project, and, instead opened up for the possibility of a study tour to Singapore (which was completed shortly before the grant closed). (c) Rating for Overall Bank Performance Description: Moderately satisfactory. The TTL and core team (of procurement and FM) were all based in the field and provided feedback and support to OHEC, as needed. 5.2 Recipient (a) Recipient Performance Description: No issues. (b) Implementing Agency or Agencies Performance Description: OHEC staff (tasked with implementing the grant) did so with high professionalism, smoothly and with few issues along the way. A key challenge was getting interim financial reports and audits prepared and submitted on time. 7 (c) Rating for Overall Recipient Performance Description: Overall rating is moderately satisfactory 6. Lessons Learned (Both project-specific and of wide general application)  Strong ownership by the government and, in particular, OHEC facilitated smooth implementation.  Ensuring that an international tool (such as EdPex) is adapted to local circumstances worked well by gradually revising the “operational guidelines” of the tool, taking on board experience from training and early implementation. Having a results indicator that focused on the importance of producing these guidelines was a wise decision and worked well.  For an upper middle income country such as Thailand and where there are no other planned lending engagements with OHEC (where the client could have benefitted from learning World Bank procedures), a small IDF grant makes little sense as a vehicle to share global knowledge and build capacity. Instead, more upfront discussions about whether it made sense to bring EdPex to Thailand using an IDF grant would have been useful. Most likely, such conversations would have revealed the high costs of using World Bank procedures (relative to the small amounts involved) and may have led to discussions of alternative ways of providing this support,. e.g. having OHEC pay for the costs out of its own resources, or using World Bank regular budget (for TA), or hiring the World Bank to provide such services using a fee-for-service arrangement. 7. Follow up Activities. As discussed above under “Risk to Development Outcome”, OHEC will continue working on EdPEx with an ambitious plan for further expanding the use of the tool to another 75 faculties/institutions in the coming five years, providing opportunities for the World Bank to continue to provide support and learn from this experience (for the benefit of other countries to learn from OHEC’s experience). 8. Other Comments None 8 Annex 1. Grant Costs and Financing by Category, Disbursement profile (a) Project Cost by Expense Category (in USD) Category Category Description Allocated Disbursed Undisbursed Totals 200,000.00 102,664.93 97,335.07 1 TF99480 - CONS SERV 140,000.00 60,920.35 79,079.65 2 TF99480 - TRAINING & WS 60,000.00 41,662.60 18,337.40 3 Designated Account – A (waived) 0.00 81.98 -81.98 9 10 Annex 2. Outputs by Component (Components are listed in Request for IDF Financing) (Please describe the outputs for each activity under the grant) See discussion on section “1.5 Grant Components” 11 Annex 3. List of Supporting Documents 1. Operation manual 12