1 ! STRENGTHENING QUALITY ASSURANCE AND PERFORMANCE EXCELLENCE IN THAILAND’S HIGHER EDUCATION Report on Lessons Learned and Recommendations Dr. Luís Ma. R. Calingo, Consultant 1. Overview of Engagement The Office of Higher Education Commission (OHEC) has engaged Dr. Luís Ma. R. Calingo, principal consultant to the Thailand Quality Award (TQA) Program, as the international consultant for the World Bank-funded project, “Strengthening Quality Assurance and Performance Excellence in Thailand’s Higher Education” (hereafter referred to as “EdPEx”). Dr. Calingo’s local counterpart has been OHEC’s Subcommittee on the EdPEx Fast-Track Project (hereafter referred to as the “EdPEx Subcommittee”), which was involved in the planning, training, group work, and review of lessons learned. To implement this three-year project, OHEC has identified about 30 prospective pilot universities throughout Thailand. After several initial consultations and preparation, the Consultant has accomplished the following as of June 30, 2014: 1. Designed, conducted, and continuously improved the EdPEx Organizational Self- Assessment Workshop for four cohorts consisting of a total of 26 participating schools from 20 universities. 2. Led site visit reviews of Mahidol University, Faculty of Science on 21 December 2012 and Chiang Mai University, Faculty of Medicine on 5 July 2013. 3. Conducted Skype videoconferences with all participating schools to guide them in interpreting the EdPEx Criteria and in preparing their Self-Assessment Reports (SARs). 4. Conducted face-to-face consultations with the participating schools from Cohorts 1-3 to guide them in interpreting their EdPEx Feedback Reports, in prioritizing their opportunities for improvement, and in refining their Self-Assessment Reports. 5. Provided ongoing consultative advice to OHEC as regards the continuing development of the EdPEx Program. Cohort and Workshop Dates Participating Schools 1 (5-7 June & 18-20 1. Chiang Mai University, Faculty of Medicine December 2012) 2. Hatyai University 3. Khon Kaen University, Faculty of Medicine 4. Mahidol University, Faculty of Science 5. Suan Dusit Rajabhat University 6. Ubon Ratchathani University, Faculty of Pharmaceutical Sciences 7. University of the Thai Chamber of Commerce ! 1 2 ! Cohort and Workshop Dates Participating Schools 2 (2-4 July 2013) 8. Kasetsart University, Bangkhen Campus, Faculty of Engineering 9. Mahidol University, Faculty of Dentistry 10. Naresuan University, Faculty of Science 11. Phuket Rajabhat University, Faculty of Humanities and Social Science 12. Rajamalanga University of Technology, Faculty of Industrial Textiles and Fashion Design 13. Suranaree Institute of Technology, Institute of Social Technology 14. Walailak University, School of Nursing 3 (18-20 December 2013) 15. Chiang Mai University, Faculty of Economics 16. Dhurakij Pundit University 17. Kasetsart University, Faculty of Engineering at Si Racha 18. Stamford International University 19. Suranaree University of Technology, Institute of Medicine 20. Walailak University, School of Pharmacy 4 (30 June-2 July 2014) 21. Assumption University, Martin of Tours School of Management 22. College of Research Methodology and Cognitive Science, Burapha University 23. King Mongkut University of Technology Thonburi, Faculty of Information Technology 24. National Institute of Development Administration 25. Prince of Songkla University, Faculty of Fine and Applied Arts 26. Thammasat University, Faculty of Medicine The EdPEx Organizational Self-Assessment Workshop included the following topics: • Review of EdPEx Criteria Core Values and Concepts • Overview of the Criteria Categories: Best Practices and Basic & Overall Item Requirements • Category-by-Category Evaluation of the Self-Assessment Reports (SAR I) • Identifying Cross-Cutting Strengths and Priority Opportunities for Improvement • Action Planning for Improvement Priorities The format of the course was about 30 percent lecture-discussion and about 70 percent of facilitated self-assessment workshops. There was ample opportunity for the individual participants to learn from each other and for the participating universities to receive and provide feedback from other universities about their practices, systems, and processes. ! 2 3 ! 2. Lessons Learned The Cohort 1 participants agreed upon a common set of key performance measures/indicators that the universities might use to track organizational performance and to benchmark each other’s performance outcomes. The OHEC staff has possession of the list of these measures. Although having a uniform set of key performance measures has a lot of merit from a comparability standpoint, educational institutions still need to have sufficient flexibility in determining their most appropriate key performance measures given the diversity of missions and strategic orientations in Thai higher education. The training for Cohort 1 by the Consultant was conducted over a total period of six days in June and December 2012. Training efficiency was improved by having pre-training sessions prior to the Consultant’s arrival. Starting with Cohort 2, members of the EdPEx Subcommittee led by Professor Rachavarn Kanjanapanyakom (hereafter referred to as the “EdPEx Training Team”) delivered a three-day EdPEx criteria training to the participating schools, which enabled them to complete their prework assignment for the Consultant’s workshop. The prework assignment for this pre-training was a self-assessment report (SAR I) that is responsive to the basic requirements of the EdPEx criteria. Starting with Cohort 2, the prework assignment for Consultant’s workshop consisted of an Organization Profile (5 pages) and a 40-page Self-Assessment Report (SAR II) that was responsive to the overall requirements of the 2013-2014 Baldrige Award Education Criteria for Performance Excellence. The facilitated self-assessment workshops provided the participating schools an opportunity to identify their most important strengths and most significant OFIs, which should be prioritized and addressed in their improvement planning. At the conclusion of the course, the participating schools identified their top five priority OFIs and action plans for addressing these OFIs during the next 6-12 months. Two process redesigns that have been implemented include (a) Skype videoconferences conducted with participating schools about 2-3 months after their training by the Consultant and (b) 50-minute consultative meetings with the participating schools about six months after their training by the Consultant. The videoconferences provided the participating schools an opportunity to report on their action plans for addressing priority OFIs and the Consultant to respond to the schools’ questions about the EdPEx criteria and to provide them further guidance as they prepare or improve their SAR reports. The consultative meetings provided the schools an opportunity to have the Consultant clarify and, when appropriate, prioritize the opportunities for improvement (OFIs) contained in their respective feedback reports. Each meeting lasted about one hour and ended with the Consultant offering specific recommendations as to next steps for the school. The quality of a Feedback Report may be demonstrated by its accuracy in terms of strengths and OFIs in response to the overall requirements of the EdPEx criteria. ! 3 4 ! Although judging accuracy is a subjective determination, it is still possible for an experienced Baldrige examiner who has read both SAR II and its feedback report to conclude whether the EdPEx assessment team was more prone to commit Type I errors (false positives) or Type II errors (false negatives). After reviewing the initial Feedback Reports (specifically for Cohort 1), the Consultant’s determination is that the EdPEx assessment teams were more likely to have committed Type II errors, which is evident in the greater instances of missing strengths than of missing OFIs and the tendency to assign lower scoring ranges than higher scoring ranges to the criteria items. The commission of Type II errors makes it very likely that more participating schools, especially those from Cohort 1, might have been inaccurately assessed as not meeting the EdPEx 200 mark. This type of non-calibration—elevation inaccuracy—is a very likely occurrence among first-year assessors (whether in TQA, Baldrige or similar programs) because first-year assessors are more likely to treat the performance excellence criteria as an audit checklist. 3. Plan for Final Consultant Visit on December 2014 The EdPEx Subcommittee and the Consultant agreed to the following schedule for the Consultant’s final visit in December 14-29, 2014: Activity Dates EdPEx assessor calibration course (preparation and December 15-17 delivery) – Consultant will prepare assessor calibration course materials (presentation and participant manual) and recommend an appropriate case study. Site visit review of a Cohort 4 participating school December 18-19 (tentatively National Institute of Development Administration) – Consultant will review and improve the site visit team’s feedback report prior to presentation to the school leadership. Executive briefing of university presidents of Cohorts 1-4 December 19 institutions; consultative meetings with selected Cohorts 1-4 schools – Consultant will prepare presentation for university presidents and review progress reports of schools. Travel to Mueang Chiang Rai December 20 Working session with OHEC staff and EdPEx December 21-23 Subcommittee to plan next steps and prepare report to World Bank Holiday December 24-25 Exit meeting with OHEC Secretary General December 26 In preparation for the December 2014 visit, the EdPEx Subcommittee will organize Skype videoconferences with Cohort 4 schools on October 2 and 7, 10:00-11:00 a.m. ICT. ! 4 5 ! Although this Report on Lessons Learned and Recommendations constitutes the Consultant’s final deliverable, the Consultant will produce additional documents in the course of the December 2014 visit. 4. Recommendations 4.1 Leveraging the Experience of the EdPEx Pilot Program The EdPEx Program should identify potential “success stories” among Cohort 1 schools and mentor them to undergo not only the EdPEx formal assessment, but also the journey to achieve the Thailand Quality Class (TQC) and eventually TQA recognition. The EdPEx Program should deploy to all participating schools the common set of key performance measures/indicators that were initially developed with Cohort 1 schools and continually refined over the cohorts. Included in this report (Appendix A) is a copy of this common set of KPIs. The EdPEx Program should enter into a formal partnership with the TQA Program so as to involve more TQA assessors, especially those from the education sector, as team leaders, backup team leaders, and coaches of EdPEx assessment teams. The EdPEx Program should develop an EdPEx assessor preparation and calibration course that is suited to the process improvement and consultation orientation of EdPEx. Although experienced TQA assessors employed in the higher education sector might be the initial pool of EdPEx assessors, they will need to be retrained or reoriented to provide feedback that: (a) is responsive to the basic or overall requirements (as opposed to the multiple requirements) of the EdPEx Criteria, and (b) includes examples of possible tools that will enable the schools to address more fully the EdPEx Criteria requirements. The EdPEx Program should consider improving the design of the EdPEx assessor preparation courses by using an actual or hypothetical EdPEx Self-Assessment Report as case study or learning materials. A comprehensive set of learning materials would include: • Self-Assessment Report—For example, see Nightingale College of Nursing (Baldrige 2009 Case Study), which is included as Appendix B and which may be downloaded from: http://www.quality.nist.gov/PDF_files/2009_Nightingale_Case_Study.pdf • Feedback Report—For example, see the Nightingale College of Nursing Feedback Report, which is included as Appendix C and which may be downloaded from: http://www.nist.gov/baldrige/publications/upload/2009_Nightingale_Feedback_Re port.pdf ! 5 6 ! • Action Plan—Create a template based on the Action Plan submitted by the Mahidol University, Faculty of Science, a copy of which is included in this report as Appendix D. The goal of these Action Plans should be that, collectively, they would lead to a 50-point increase in total score (or an average 5% increase in Item scores). • Revalidation Report—An annual report, in tabular format, showing the (i) strengths and opportunities for improvement (OFIs) from the Feedback Report, (ii) actions taken to sustain each strength or address each OFI, and (iii) evidence of each action taken, such as references to new procedures or supporting documents. Appendix E presents a proposed template for this Revalidation Report. 4.2 National Deployment of the EdPEx Program The OHEC website lists a total of 171 higher education institutions in Thailand as of December 2012, consisting of 80 public HEIs, 71 private HEIs, and 20 community colleges. OHEC will need to think “outside the box” and create a system to deploy the EdPEx Program to the more than 100 colleges and universities that are not beneficiaries of a significant infusion of resources to bring them to world-class status. The Consultant proposes a system that has two major elements: (a) institutional mentoring, and (b) organizational machinery. 4.2.1 Institutional Mentoring The EdPEx Program should encourage and establish educational partnerships with first- and second-tier institutions for technical assistance in upgrading educational programs. The best and most sustainable of these partnerships occur when all partner-institutions benefit from the partnership, thereby advancing their respective missions. An example of a mutually beneficial linkage here might be a regional university working with one of Thailand’s 20 community colleges so that associate’s degree holders of that community college can seamlessly transfer to the university to complete their bachelor’s degrees. Institutional mentoring might be jumpstarted by having each of the Cohorts 1-3 participating schools “adopt” a not-yet-participating school as a mentee. 4.2.2 Organizational Machinery OHEC should create new EdPEx Institute that will be responsible for the overall management of the evaluation function and oversight (but not direct responsibility for) of the consultancy function. The activities of the EdPEx implementing organization should be aligned and integrated with the Office for National Education Standards and Quality Assessment (ONESQA), the Thailand Qualification Framework (TQF), the OHEC-mandated Internal Quality Assessments, and other components of the Thai Government’s quality assurance system for higher education. Ideally, the new EdPEx Institute should be operational during the 2014/2015 fiscal year. ! 6 7 ! 4.2.3 Business Model for the EdPEx Institute In overseeing the national implementation of the EdPEx Program, the proposed EdPEx Institute might adopt the business model (including the use of volunteer peer reviewers) of the Academic Quality Improvement Project (AQIP) of the Higher Learning Commission of the North Central Association of Schools and Colleges.1 The AQIP pathway to accreditation requires participating institutions to first complete a Systems Portfolio2 and then annual Action Projects3 that would be the continuous improvement plans that the institutions will execute. The recommended adaptation of the AQIP approach to Thailand might be an EdPEx Self-Assessment Report prepared every five years, an annual improvement planning day to identify action items to address opportunities for improvement identified in the EdPEx Feedback Report, and an annual submission containing an Action Project similar to AQIP and a progress report of actions taken during the past year to address opportunities for improvement. The EdPEx Institute might initially be formed and organized along the lines of the now-defunct US Federal Quality Institute (FQI).4 FQI was created in June 1988 as a temporary organization to advance and promote quality management programs in the US federal government, which had 3.8 million employees at that time. It was initially staffed with four people and no budget, with all equipment either on loan or rented, and with its 34 staff members on loan from various federal agencies (usually for 1-2 years at a time). FQI coordinated and produced the annual National Conference on Federal Quality and the President’s Quality Award. In September 1995, FQI was closed and its functions transferred to the US Office of Personnel Management. The difference between the proposed EdPEx Institute and the US Federal Quality Institute is that, unlike the FQI, the EdPEx Institute is not meant to be a temporary organization without a budget. There are two options for organizing the EdPEx Institute, each with its own governance and staffing implications: • The first option is to create the EdPEx Institute as a government-chartered Public Organization (PO) that will be autonomous yet organizationally related to OHEC. The founding staff of the EdPEx Institute might be professionals seconded from various government agencies and public universities, with additional staff recruited from among higher education professionals. A role model for the Public Organization form for the EdPEx Institute might be the Healthcare Accreditation Institute of Thailand (HA), !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 1 http://aqip.org/AQIP/aqip-home.html 2 https://www.ncahlc.org/AQIP-Systems-Appraisal/aqip-portfolio.html 3 https://www.ncahlc.org/AQIP-Action-Project-Directory/aqip-action-projects.html 4 Brad Stratton, “Federal Quality Missionaries,” Quality Progress, May 1991, 67- 69. ! 7 8 ! which has a quite similar mission as the proposed EdPEx Institute.5 For reasons that will be provided later, the Public Organization is the more preferred option. • The second option is to create the EdPEx Institute as a not-for-profit foundation under contract with OHEC. This is similar to the organizational relationship between the U.S. Baldrige Performance Excellence Program (BPEP) and the American Society for Quality (ASQ), which is the contract administrator of the Baldrige Award). 6 This is somewhat similar to the relationship between the Foundation for Thailand Productivity Institute (FTPI)7 and the Thai Government in the administration of the Thailand Quality Award (TQA), except that, unlike the BPEP-ASQ relationship, the Office of the TQA is actually a financial responsibility center within FTPI. 4.2.4 Organization for the Consultancy Function Design the organizational machinery for the delivery of the consultancy function under the EdPEx Institute’s oversight. Whatever organization design OHEC decides to utilize for the national implementation of the EdPEx Program, there should be a “firewall” between the units or personnel involved in assuring higher education institutions’ (HEIs) compliance with EdPEx and other standards (evaluation function) and those involved in assisting HEIs address the EdPEx criteria requirements (consultancy function). Limiting the EdPEx Institute’s scope to the evaluation function and oversight of the consultancy function ensures that there would be a firewall between evaluation and consultancy. The consultancy function might be jumpstarted with a set of about 5-10 founding consultants and trainers from among selected Thailand Quality Award assessors in the higher education sector and participants from the advanced EdPEx cohorts. There are two options for the continuing delivery of the consultancy function beyond the set of founding consultants and trainers: • The first option is for the EdPEx Institute to subcontract the consultancy function to not-for-profit, non-governmental organizations such as the Foundation for Thailand Productivity Institute. For example, the business scope of FTPI might be expanded to include the provision of consultancy and training to Thailand’s HEIs on the EdPEx framework and its associated tools and methods. These training programs might be provided in collaboration with Thailand’s business and education schools that might be familiar with the EdPEx framework and its implementation. The surplus net income resulting from this expanded business scope might be shared between FTPI and the !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 5 http://www.ha.or.th/ 6 http://www.prweb.com/releases/2013/11/prweb11371791.htm 7 http://www.ftpi.or.th ! 8 9 ! EdPEx Institute, thereby leading to a non-governmental source of funding to partially support the EdPEx Institute. • The second option is for the EdPEx Institute to create a process for certifying new consultants and trainers beyond the founding set. The certification program might be based on a validation of the candidate’s education and documented EdPEx assessment process experience; completion of TQA Assessor, EdPEx Assessor or Certification training programs; and performance in a written examination on the EdPEx criteria and processes. A role model for this certification process might be the Baldrige Institute of the Missouri Quality Award.8 After completion and certification, the consultants and trainers might reimburse the EdPEx Institute for their training costs in the form of either a forgivable loan (via service as a peer reviewer over a 2-3-year period) or installment payments through withholding of a nominal percentage of their consulting/training fees. 4.2.5 Institutionalization of a Peer Review System The U.S. model of a voluntary system of accreditation based on peer review, as opposed to government regulation, has succeeded because of the mass base of peer reviewers who participate in accreditation reviews not for compensation, but as a contribution to the profession. Over time, the success of the national deployment of the EdPEx Program hinges on the development of a peer review culture in Thai higher education. If such a voluntary system of peer review is not feasible in the Thai context, then the cost of engaging peer reviewers as paid professionals will need to be incorporated into the program budget. 5. Acknowledgments Dr. Calingo would like to express his gratitude to the staff of the Office of Higher Education Commission (OHEC), particularly its subcommittee on the EdPEx Fast- Track Project, and OHEC’s partners from the Thailand Quality Award Program. Dr. Calingo would also like to recognize (1) Professor Avudh Srisukri who has taken the lead to move the whole EdPEx initiative forward and (2) Dr. Nantana Sirisup who has been the main project coordinator overseeing overall planning and implementation. Their commitment, dedication, and passion for raising the level of academic quality and organizational excellence in Thailand’s higher education institutions is commendable and inspiring. It has always been a pleasure working with them. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 8 http://www.baldrigeinstitute.org/Pages/Home.aspx ! ! 9