Document of The World Bank Report No: ICR00003977 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-81260) ON A LOAN IN THE AMOUNT OF US$40 MILLION TO THE REPUBLIC OF CHILE FOR A TERTIARY EDUCATION FINANCE FOR RESULTS PROJECT III APRIL 27, 2017 CURRENCY EQUIVALENTS April 19, 2017 Currency Unit = Chilean Peso CLP646 = US$1 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS CPS Country Partnership Strategy CRUCH National Council of Rectors of Chilean Universities DFI Department of Institutional Financing DIVESUP Division of Higher Education GoC Government of Chile ICT Information Communications and Technology IP Professional Institute IPPF Indigenous Peoples Planning Framework MECESUP (3) Tertiary Education Finance for Results Project (III) MINEDUC Ministry of Education NPV Net Present Value OECD Organization for Economic Cooperation and Development OP/BP Operational Policies and Bank Procedures PA Performance Agreement PAD Project Appraisal Document PDO Project Development Objective PM Improvement Plan or Small Project (Plan de Mejoramiento) PMI Institutional Improvement Plan (Plan de Mejoramiento Institucional) RF Results Framework R&D Research and Development SIES Tertiary Education Information System TEI Tertiary Education Institution TTC Technical Training Center WBG World Bank Group Senior Global Practice Director: Jaime Saavedra Chanduvi Practice Manager: Reema Nayar Project Team Leader: Diego Ambasz ICR Team Leader: Diego Ambasz i CHILE TERTIARY EDUCATION FINANCE FOR RESULTS PROJECT III CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................................. 11 2. Key Factors Affecting Implementation and Outcomes ............................................................ 16 3. Assessment of Outcomes .......................................................................................................... 21 4. Assessment of Risk to Development Outcome ......................................................................... 28 6. Lessons Learned........................................................................................................................ 30 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners........................... 31 Annex 1. Project Costs and Financing .......................................................................................... 32 Annex 2. Outputs by Component.................................................................................................. 33 Annex 3. Economic and Financial Analysis ................................................................................. 54 Annex 4. Bank Lending and Implementation Support/Supervision Processes............................. 56 Annex 5. Beneficiary Survey Results ........................................................................................... 58 Annex 6. Stakeholder Workshop Report and Results ................................................................... 59 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................................... 60 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders ...................................... 70 Annex 9. List of Supporting Documents ...................................................................................... 71 Annex 10. Evaluation of Overall Outcomes Using a Split Ratings Approach (Pre- and Post- Restructuring) ............................................................................................................................... 72 Annex 11. The Causal Chain for the MECESUP 3 Project .......................................................... 77 MAP .............................................................................................................................................. 78 ii Data Sheet A. Basic Information Tertiary Education Country: Chile Project Name: Finance for Results Project III Project ID: P111661 L/C/TF Number(s): IBRD-81260 ICR Date: 09/08/2016 ICR Type: Core ICR GOVERNMENT OF Lending Instrument: SIL Borrower: CHILE Original Total USD 40.00M Disbursed Amount: USD 40.00M Commitment: Revised Amount: USD 40.00M Environmental Category: C Implementing Agencies: Division of Higher Education (DIVESUP), Ministry of Education Co-financiers and Other External Partners: N/A B. Key Dates Revised/Actual Process Date Process Original Date Date(s) Concept Review: 06/21/2011 Effectiveness: 06/20/20131 09/16/14 and Appraisal: 11/29/2011 Restructuring(s): 07/13/2015 Approval: 03/13/2012 Mid-term Review: 03/16/2015 03/16/2015 Closing: 10/31/2016 10/31/2016 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Low or Negligible 1 While the Tertiary Education Finance for Results Project III was approved by the Bank in March 2012, it did not go into effect until June 2013. Effectiveness was delayed because: 1. the Bank signed the Loan Agreement in August 2012, thereby requiring it to go into effective within 120 days, but the Loan Agreement was not countersigned by the Treasurer and Controller General of Chile until March 2013. Thus the loan had to be reinstated by the Bank, which occurred in June 2013; 2. The project code for the lo an was incorrectly recorded in Chile’s Budget Law, which could only be corrected once the GoC requested a supplemental budget, thus further delaying implementation post-reinstatement; and 3. a new Minister of Education entered office in April 2013, which led to an overall slowdown in work at MINEDUC as staffing changes took effect. Nonetheless, Project implementation was able to begin in late August 2012 using GoC funds to execute the selection, negotiation and award of Performance Agreements. 1 Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance by ICR Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Moderately Satisfactory Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of No None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Public administration – Education 5 5 Tertiary education 95 95 Theme Code (as % of total Bank financing) Education for the knowledge economy 100 100 E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Hasan A. Tuluy Country Director: Alberto Rodríguez Susan G. Goldmark Practice Manager/ Reema Nayar Chingboon Lee Manager: Project Team Leader: Diego Ambasz Michael Crawford ICR Team Leader: Diego Ambasz ICR Primary Author: Nicole Mammoser 2 F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The Project Development Objective was to improve quality and relevance for students in tertiary education by strengthening the link between funding of tertiary education institutions and accountability for performance. Revised Project Development Objectives (as approved by original approving authority) No revisions were made to the PDO. PDO Indicators Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values documents) Target Years Number of full-time equivalent (> or = 44 hours/week) faculty members who Indicator 1 hold PhDs at all accredited Tertiary Education Institutions (TEIs) Value Original: 4,148 4,900 7,500 (quantitative or 8,332 Revised: 5,109 (at 12/31/2015) (at 10/31/2016) qualitative) Date achieved 5/15/2011 (Revised) 12/31/2011 12/31/2015 10/31/2016 Original target surpassed; 344% of target achieved. (Uses actual value at Restructuring #1: 6,734.) Revised target surpassed; 135% of target achieved. Indicator is calculated as the sum of total working hours of academics holding a Comments PhD divided by 44 (the number of hours in a full-time work week). At Project (inc. % close, the number of full-time equivalent faculty members who hold PhDs at achievement) TEIs supported by the MECESUP 3 Project account for 95% of the actual value achieved, or 7,915 faculty members. Baseline and target values were revised at the Project’s first Restructuring on September 16, 2014 to reflect more recent and complete data in the Tertiary Education Information System database (SIES). Retention rate measured as first-year undergraduate students that remain at the Indicator 2 institution in the second year Value (quantitative or 68.4% 72% 71.3% qualitative) Date achieved 4/30/2011 N/A 10/31/2016 Target partially achieved; 80.6% of target achieved. Comments Indicator is calculated as first-year students in Year t who continue to study at (inc. % the same institution in Year t + 1, divided by the first year students in Year t, achievement) multiplied by 100. Indicator considers students in programs at Technical Training Centers that are 4 semesters or longer, students in programs at Professional Institutes that are 6 semesters or longer, and students in university 3 bachelor’s degree programs that are 8 semesters or longer. Note that the first-year retention rate increases to 73.8% (surpassing the 72% target) for TEIs that received support from the MECESUP 3 Project, compared to 67.4% for TEIs that did not receive Project support. Further, the retention rate increases to 79.5% when taking into account first-year students who transfer to a different TEI to pursue their second year of study (i.e., who are retained in the system at a different TEI). Percentage of students in teaching degrees with redesigned curricula, including Indicator 3 practical training, at TEIs with Performance Agreements Value (quantitative or N/A 25% 70.5% qualitative) Date achieved N/A 12/31/2015 10/31/2016 Target surpassed; 282% of target achieved. This PDO Indicator was introduced at the July 2015 Restructuring in order to better reflect improvements in relevance for students enrolled in teacher training programs at TEIs implementing Teacher Training Improvement Institutional Improvement Plans (PMI). 2 Indicator is calculated as the number of students pursuing teaching degrees in Comments programs with redesigned curricula that include a teaching practicum at TEIs (inc. % implementing a PMI divided by the number of students pursuing teaching achievement) degrees at TEIs implementing a PMI. A very high percentage of students are studying in teacher training programs that have added a teaching practicum to their curriculum (70.5% versus the targeted 25%). This is due to the fact that the majority of TEIs that implemented a Teacher Training Improvement PMI included a practicum requirement for all matriculated students, not just those entering as first-year students once a PMI was underway. Indicator 4 Percentage of students in technical and professional degrees3 with redesigned curricula at TEIs with Performance Agreements 2 Institutional Improvement Plans (called PMIs, for the Spanish “Plan de Mejoramiento Institucional�) are Performance Agreements that aim to improve and reform TEIs on a larger scale than Improvement Plans (called PMs, “Plan de Mejoramiento�), which are more targeted in scope. All PMIs fall into one of three categories: Teacher Training Improvement, Technical/Professional Training Improvement, or Academic Innovation. PDO Indicator #3 measures the percentage of students pursuing teaching degrees at TEIs that included a teaching practicum in their redesigned teacher training curricula as part of a Teacher Training Improvement PMI. 3 Technical Training Centers (TTCs) and Professional Institutes (IPs) both offer vocational training programs at the higher educational level (in computer science, engineering, contracting services, medical support services, etc.) specifically designed to meet the needs of the labor market. Performance Agreements for “Technical/Professional Training Improvement� PMIs are awarded to TTCs and IPs. 4 Value (quantitative or N/A 50% 89.4% qualitative) Date achieved N/A 12/31/2015 10/31/2016 Target surpassed; 179% of target achieved. This PDO Indicator was introduced at the July 2015 Restructuring to better reflect improvements in relevance for students enrolled in technical and professional degree programs at TEIs implementing Technical/Professional Training Improvement PMIs. Indicator is calculated as the number of students pursuing technical and Comments professional degrees in programs with redesigned curricula at TEIs (inc. % implementing a PMI divided by the number of students pursuing technical and achievement) professional degrees at TEIs implementing a PMI. A very high percentage of students are studying in technical or professional degree programs that have redesigned their curricula (89.4% versus the targeted 50%). This is due to the fact that the majority of TEIs that implemented a Technical/Professional Training Improvement PMI integrated enhanced curricula into the programs being pursued by all matriculated students, not just those entering as first-year students once a PMI was underway. (b) Intermediate Outcome Indicators Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values documents) Target Years Indicator 1 Number of signed Performance Agreements (supporting both PMs and PMIs) For Teacher For Teacher For Teacher Training: 12; Training: 6; Training: 19; for Academic for Academic for Academic Innovation: 12; Innovation: 6; Innovation: 24; Value for Technical/ for Technical/ for Technical/ (quantitative or N/A Professional Professional Professional qualitative) Training: 10; Training: 5; Training: 17; for Small Projects for Small Projects for Small Projects (PMs): 24 (PMs): 12 (PMs): 119 (58 total (29 total (179 total 5 Performance Performance Performance Agreements) Agreements) Agreements4) Date achieved N/A 12/31/2013 12/31/2015 12/31/2015 Original target surpassed; 309% of target achieved. Revised target surpassed: 617% of target achieved. Due to incomplete data at Project appraisal in early 2012, target values for signed Performance Agreements by project category were only set through 2013. Target values were revised at the September 2014 Restructuring (through 12/31/2015) to reflect more detailed information collected subsequent to Comments appraisal. (inc. % achievement) The total number of signed Performance Agreements significantly exceeded the revised target for this indicator (179 versus the target of 29), due to the large number of Performance Agreements for Improvement Plans (PMs, referred to in the PAD as “Small Projects�) signed (119 versus the target of 12). Revised targets for signed Performance Agreements for the three PMI categories were exceeded as well: 19 for Teacher Training Improvement, 24 for Academic Innovation, and 17 for Technical/Professional Training Improvement. Indicator 2 Rate of fulfillment of indicators included in Performance Agreements Value (quantitative or N/A N/A 70% 71.3% qualitative) Date achieved N/A N/A 10/31/2016 10/31/2016 Target surpassed; 102% of target achieved. In mid-2014, data for Performance Agreement indicators showed a 55% fulfillment rate. As a result, the target value for the fulfillment rate of Performance Agreement indicators at Project end was set at 70% at the first Restructuring in September 2014. Comments (inc. % The 71.3% fulfillment rate for the indicators included in all Performance achievement) Agreements incorporates both Performance Agreements that had completed implementation and those that were still in execution at 12/31/2016. Indicator is the average percentage of fulfillment of indicators included in Performance Agreements. It is calculated as the sum of percentages of fulfillment of Performance Agreement indicators, divided by the number of indicators. (Each indicator can range from 0 to 100%.) Indicator 3 Direct Project beneficiaries (number), of which female (percentage) (Core Indicator) 4 Twenty-three of the total 60 PMI Performance Agreements were signed during 2012 (seven in the Teacher Training Improvement category, four in Technical/Professional Training Improvement, and 12 in Academic Innovation), even though the MECESUP 3 Project did not formally become effective until June of 2013. 6 Set at September 2014 Restructuring: 275,000 beneficiaries/ 279,883 Value 51% female beneficiaries / (quantitative or N/A N/A Revised at 51% female qualitative) July 2015 Restructuring: 280,000 beneficiaries/ 51% female Date achieved N/A N/A 10/31/2016 10/31/2016 September 2014 target surpassed; 102% of target achieved. July 2015 revised target substantially achieved; 99% of target achieved. Due to incomplete data at Project appraisal in early 2012, baseline and target values were not set. The target value was set at the September 2014 Restructuring to reflect more detailed information collected subsequent to appraisal. Comments The target value for the number of Project beneficiaries was revised at the July (inc. % 2015 Restructuring to reflect the fact that a higher number of Performance achievement) Agreements were signed by TEIs than had been anticipated at the September 2014 Restructuring. Although the Project fell just 117 beneficiaries short of the revised target, it met the goal of 51% female Project beneficiaries. This is due to the high number of females impacted by the PMs and PMIs underway in 2014 and 2015, many of which focused on teacher training efforts that predominantly affected women. Beneficiaries included students, academics and administrators at TEIs that implemented at least one Performance Agreement. Number of students completing remediation programs in TEIs with Indicator 4 Performance Agreements Set at September 2014 Restructuring: Original: N/A 50,000 Value Set at (at 10/31/2016) 69,497 (quantitative or September 2014 N/A Revised at qualitative) Restructuring: July 2015 33,925 Restructuring: 65,000 (at 10/31/2016) Date achieved 12/31/2011 N/A 10/31/2016 10/31/2016 Comments September 2014 target surpassed; 221% of target achieved. 7 (inc. % July 2015 revised target surpassed; 114% of target achieved. achievement) Due to incomplete data at Project appraisal in early 2012, baseline and target values were not set. The baseline and target values were set at the September 2014 Restructuring to reflect more detailed information collected subsequent to appraisal, and the end target was revised at the second restructuring in July 2015. Note that 30,503 students completed remediation programs during the same period at TEIs that did not implement Performance Agreements, which is less than half the number of students benefitting from remediation programs at TEIs with Performance Agreements. Indicator 5 Number of graduates from domestic PhD programs supported by the Project Value Original: 433 530 200 Original: 685 (quantitative or Revised: 103 (at 12/31/2015) (at 10/31/2016) Revised: 208 qualitative) 12/31/2011 Date achieved N/A 10/31/2016 10/31/2016 (Revised) Original target surpassed; 260% of target achieved. Revised target surpassed; 108% of target achieved. Comments Indicator’s name and its baseline and target values were revised at the (inc. % September 2014 Restructuring to reflect only graduates of PhD programs at achievement) TEIs supported by the MECESUP 3 Project (not PhD graduates of all accredited TEIs). The Project funded a total of 12 PMs and PMIs that supported domestic PhD programs. Percentage of teaching degrees with redesigned curricula in TEIs with Indicator 6 Performance Agreements Value (quantitative or N/A 60% 71.2% qualitative) Date achieved N/A 8/9/2016 10/31/2016 Target surpassed; 119% of target achieved. This indicator was introduced at the July 2015 Restructuring to reflect progress toward PDO Indicator 3. This indicator monitored the redesign of Comments teacher training curricula in TEIs with Performance Agreements. (inc. % achievement) Teacher training curricula redesign incorporated input from MINEDUC on weaknesses in primary and secondary learning outcomes, as well as from external consultants who advised TEIs on curricula integration of current international best practices for classroom teaching across various subjects and grade levels. Indicator 7 Number of technical and professional degrees with redesigned curricula in TEIs 8 with Performance Agreements Value 150 (quantitative or N/A 202 (at 10/31/2016) qualitative) Date achieved N/A 10/31/2016 10/31/2016 Target surpassed; 135% of target achieved. This indicator was introduced at the July 2015 Restructuring to reflect progress toward PDO Indicator 4. This indicator monitored the redesign of Comments technical and professional training program curricula in TEIs with Performance (inc. % Agreements. achievement) Technical and professional curricula redesign incorporated input from consultation on labor market needs with private sector and civil society representatives, in alignment with international best practices. Indicator 8 Number of policy notes and studies carried out Value 3 3 (quantitative or 0 3 (at 12/31/2015) (at 10/31/2016) qualitative) Date achieved 12/31/2011 10/31/2016 10/31/2016 10/31/2016 Original target not achieved; 33% of target achieved. Revised target achieved; 100% of target achieved. The date for this indicator to be achieved was revised at the July 2015 Comments Restructuring. While one policy note was completed at year-end 2015, the (inc. % timelines for both the Project’s studies were pushed out to Project close in achievement) 2016. This indicator is cumulative, so the total number of policy notes and studies carried out during the Project remains three. This indicator pertains to Component 2 of the Project (Policy Support and Project Management). G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 06/24/2012 Satisfactory Satisfactory 0.00 2 02/04/2013 Satisfactory Satisfactory 0.00 3 12/07/2013 Satisfactory Moderately Satisfactory 0.00 4 06/29/2014 Satisfactory Moderately Satisfactory 13.27 5 01/05/2015 Satisfactory Moderately Satisfactory 13.27 6 08/05/2015 Satisfactory Moderately Satisfactory 13.27 9 7 01/28/2016 Satisfactory Moderately Satisfactory 33.34 8 08/09/2016 Satisfactory Moderately Satisfactory 33.34 9 10/09/2016 Satisfactory Moderately Satisfactory 39.49 H. Restructuring (if any) Board ISR Ratings at Amount Restructuring Restructuring Disbursed at Reason for Restructuring & Approved Date(s) Restructuring Key Changes Made PDO Change DO IP in USD millions The September 2014 Restructuring mainly amended the Results Framework with updated data that was not available at 09/16/2014 No S MS 13.27 Project Appraisal in early 2012. Disbursement estimates were also corrected to reflect delays to Project effectiveness and implementation. The July 2015 Restructuring amended the Results Framework to increase the 07/13/2015 No S MS 13.27 coherence of the Project’s results chain based on a review of Performance Agreements under implementation. I. Disbursement Profile 10 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Country background. At the time of project appraisal for MECESUP 3 during late 2011 and early 2012, Chile was an upper middle income country that had recently acceded to full membership in the OECD. It was a politically stable country with strong institutions and a solid record of economic growth during the previous two decades. The Government that took office in March 2010 was pursuing goals similar to those of previous governments: promoting reforms to boost growth, investing in human capital, and tackling poverty and inequality.5 Within this context, Chile was being challenged to reverse declining relative growth performance 6 and insufficient innovation capacity in order to grow the size and wealth of its middle class. 2. In early 2012, the Government of Chile (GoC) was pursuing a development agenda aimed at achieving high income status by 2018. Areas of strategic emphasis to reach this goal included improving job creation and job quality and investing in human capital. Increased policy attention to education at all levels was recognized as a key driver of economic and social progress. And with approximately one million students in tertiary education,7 the Government simultaneously faced an emboldened student population impatient for improved educational quality and opportunity.8 3. The World Bank Group’s (WBG) Country Partnership Strategy (CPS) with Chile for the period FY2011-2016 identified three areas where Bank activities would be carried out, one of which was job creation and equity improvement.9 The Bank pledged to support Government efforts that focused on improving the quality of education as a means to foster job opportunities, develop labor skills, and enhance productivity. The CPS specifically noted the GoC’s plans to increase quality in the tertiary education system through greater reliance on performance-based funding. 4. Sector background. At Project appraisal, Chile’s effort to expand tertiary education beyond the “elites� to the “masses� while maintaining the quality level within the system was widely recognized. However, key challenges in tertiary education persisted, and a recent study10 highlighted gaps in quality and relevance. Quality could be improved by increasing the percentage of PhD holders in the Chilean professorate, making curricula and teaching practices 5 National Development Plan, 2010-2014. 6 Chile’s growth had slowed from an average of 7.6% during the period 1986 -1997 to 3.9% in the period 2000-2010. 7 At the time of Project appraisal, one-third of all tertiary students attended Chile’s traditional public and private universities (these are the “CRUCH� universities, the Spanish acronym for the National Council of Rectors of Chilean Universities), one-third attended one of the newer private universities, and the remaining third attended technical and professional training centers (called TTCs and IPs). Equity had improved over the decade leading up to MECESUP 3, with the share students from the two lowest income quintiles tripling to reach 25% of total enrollment. 8 Since 2011, student protests have intensified in response to growing costs and unequal access. Students’ main demands are free tuition at all TEIs, that TTCs and IPs be prohibited from operating for profit, and that the quality of tertiary education be improved. 9 The two other areas of intervention outlined in the FY 2011-2016 CPS were public sector modernization and promotion of sustainable investments. 10 “Reviews of National Policies for Education: Tertiary Education in Chile�, OECD and The World Bank, 2009. 11 more rigorous and effective, and raising academic readiness for new students who had received a mediocre secondary education. With respect to relevance, the consensus among higher education stakeholders was that academic programs and course offerings needed to promote the skills and competencies demanded by the labor market. Related challenges included replacing mandatory theoretical content with the skills-based training required by employers, reaching a critical mass of qualified professors to develop and deliver such curricula, and shifting policy to encourage students to seek shorter degree programs with closer connections to employment prospects. In spite of the acknowledged challenges, many TEIs in Chile, especially those that were publicly financed, lacked sufficient discretionary resources to fund initiatives to improve quality and relevance. 5. The Bank enjoyed a substantive partnership with Chile in the tertiary education sector since 1997, and was a valued contributor in the areas of technical assistance, policy analysis, and co-financing. From 1998-2005 the World Bank worked with the GoC on the “Program to Increase Quality and Equity in Tertiary Education,�11 referred to as “MECESUP� for the Spanish acronym.12 MECESUP established a competitive fund, allocated to TEIs based on merit,13 to finance investments to develop curricula more responsive to labor demands and social needs, and expand access to higher education for low-income students. MECESUP also funded the creation of accreditation councils in order to pilot a quality assurance system for TEIs. Lastly, the program invested in domestic and foreign PhD training for university faculty to meet the demands of increasing enrollment at TEIs.14 One of MECESUP 1’s primary achievements was to strengthen government and institutional capacity for strategic and evidence-based decision making. 6. The MECESUP 2 Project followed from 2005-2010, building on the success of MECESUP 1 by introducing results-based funding of TEIs at the institutional level.15 Four large pilot programs were launched in an effort to enhance coherence, responsiveness, equity and quality in four CRUCH universities. Under these programs, total financing of US$10.2M depended on each TEI’s achievement of agreed-upon targets. The pilots focused on bolstering management capacity at the university level, increasing the quality of instruction, particularly in pedagogy programs, implementing remediation programs for poorly prepared first-year students, and promoting scientific research. 11 The Bank’s important role in the early formulation of the competitive funding approach to improve tertiary level education in Chile was cited in a January 2017 UNESCO/Aalborg University report (“Snapshot Review of Engineering Education Reform in Chile�). The Bank is acknowledged as having offered a funding alternative for TEIs based on strategic planning and institutional priorities, as well as transparency and accountability. 12 MECESUP is short for “Programa para el Mejoramiento de la Calidad y Equidad de la Educación Superior .� 13 In 2010, approximately 87% of public resources given to TEIs were allocated according to historical criteria, and not merit-based. 14 From 1990 to 2011, the gross higher education enrollment ratio increased from 14% to over 50%. Total new enrollments increased from about 215,000 in 2005 to 343,000 in 2015. 15 MECESUP 2 also helped fund over 300 small quality-enhancing projects at TEIs, but these grants were not performance-based, as the four larger pilots were. 12 7. MECEUP 1 and 2 were widely credited with having contributed to improved curricula and teaching qualifications, budget and management innovations at TEIs, and better-formulated institutional improvement plans.16 8. The MECESUP 3 Project was appraised in light of the success of the MECESUP 1 and 2 projects, and the continued need to link public funding of TEIs to their ability to meet targets for improved performance. MECESUP 3 scaled up the use of performance-based agreements as a means to enable institutional improvement and innovation at TEIs as they worked to improve the quality and relevance of the education they offered. Bank support for the MECESUP 3 Project was in keeping with its commitment to support improvements in tertiary education that would lead to more highly developed labor skills, thereby increasing employment opportunities and enhancing productivity in Chile and competitiveness in the global economy. 9. At appraisal, Chile had three categories of tertiary education institutions: universities,17 Technical Training Centers (TTCs), and Professional Institutes (IPs). TTCs and IPs were established in 1981 to provide vocational and technical training. Degree programs in Chile ranged from short technical degrees to full doctoral programs. Degree length was intended to be two years at TTCs, three to four years at IPs, and four years at universities (for an undergraduate degree).18 In 2010, 87 of Chile’s approximately 200 tertiary education institutions (TEIs) were accredited.19 1.2 Original Project Development Objectives (PDO) and Key Indicators 10. The objective of the Project was to improve quality and relevance for students in tertiary education by strengthening the link between funding of tertiary education institutions and accountability for performance. The ICR Team decided to treat the PDO as a single objective (rather than one with two dimensions) for two reasons. First, improvements in quality and improvements in relevance “overlap� and are interrelated for students at TEIs. Second, the PDO and Intermediate Outcome Indicators designed to evaluate achievements encompass both quality and relevance at the same time (i.e., increased numbers of faculty with PhDs boost the quality of educational instruction for students; resulting enhanced learning outcomes are relevant to students as they seek employment). 11. As previously established under MECESUP 1 and 2, competitively awarded Performance Agreements were the legal vehicle linking funding of tertiary education institutions to accountability for their performance. Indeed, the funding provided held those institutions 16 “Reviews of National Policies for Education: Tertiary Education in Chile�, OECD and The World Bank, 2009. 17 Universities are divided into two categories: the traditional public and private CRUCH universities, and newer universities, all of which are private and were typically founded after the 1980’s. 18 In reality, study duration at year-end 2010 was approximately three and a half years for TTCs (6.8 semesters), four and a half years for IPs (8.5 semesters), and 6 and half years for universities (13 semesters). Lengthened degree programs often reflected the fact that students took breaks from their studies in order to work to earn money to cover school fees before returning to their TEIs. 19 In 2006, the GoC passed the Quality Assurance Law, which led to the establishment of a quality assurance system to provide a framework for licensing TEIs, institutional accreditation of TEIs, and accreditation of distinct degree programs within TEIs. Only accredited TEIs were eligible for Performance Agreement awards under the MECESUP 3 Project. 13 accountable for accurately monitoring and transparently reporting outcomes related to improvements in quality and relevance. 12. The indicators to measure the Project Development Objective as included in the Project Appraisal Document were:  Number of full-time equivalent faculty members who hold PhDs  Average actual duration of undergraduate studies until the degree is awarded  Retention rate, measured as first-year undergraduate students that remain at the institution in the second year  Employers’ perception of quality and relevance of graduates of the MECESUP 3- supported TEIs, as a proxy for increased employability and wage premiums 1.3 Revised PDO and Key Indicators, and reasons/justification 13. No changes were made to the PDO. Modifications were made to the Results Framework as part of two Project restructurings. The first restructuring, in September 2014, mainly updated the RF to set and/or revise baselines and targets based on data that was not available at Project Appraisal in late 2011/early 2012. 14. The second restructuring, in July 2015, was an effort to increase the coherence of the Project’s results chain. Two of the Project’s original PDO Indicators were dropped from the Results Framework. The first PDO Indicator, measuring average actual duration of undergraduate studies until the degree is awarded at all accredited TEIs, was dropped because the two main contributors to indicator achievement, curricular reform (including implementation of credit transfer systems) and remedial and tutoring support, would primarily demonstrate a positive impact on time to graduate after Project close. The second PDO Indicator, measuring employer perceptions of the quality and relevance of graduates of MECESUP 3-supported TEIs as a proxy for increased employability and wage premiums, was dropped because no baseline had been established, since the survey planned for the beginning of the Project to measure employer perceptions was not conducted on time.20 15. Two new PDO Indicators were added to the RF. The first measured the percentage of students in teaching degrees with redesigned curricula, including a teaching practicum, at TEIs with Performance Agreements. The second measured the percentage of students in technical and professional degree programs with redesigned curricula at TTCs and IPs with Performance Agreements. Both indicators were introduced to better reflect improvements in relevance for students. Two new Intermediate Outcome Indicators were also added to the RF to monitor 20 An “Employer Perception Study� was instead conducted under Component 2 of the Project (Policy Support and Project Management). The study evaluated employers' assessments of the quality and relevance of a small cohort of PhD and pedagogy graduates hired from TEIs that implemented Performance Agreements under MECESUP 3. Preliminary results indicate that employers more positively perceive teachers who were trained at TEIs with MECESUP 3 support than teachers who trained at TEIs without MECESUP 3 support. 14 redesign of teacher training and vocational training curricula in order to reflect progress toward the new PDO Indicators. 1.4 Main Beneficiaries 16. The Project’s main beneficiaries include the following stakeholders at the TEIs that were awarded Performance Agreements: 1) current and future students enrolled at TEIs who benefit from more coherent curricula delivered by better-qualified instructors, shorter degree programs, and expanded remedial support targeting students from less advantaged backgrounds to ensure that they attain knowledge and skills at par with those of their peers; 2) lecturers who pursued advanced study to improve their credentials and teaching methods; 3) administrators who received training to improve their management skills; and 4) employers who profit from more highly qualified job candidates who are prepared to assume work dictated by labor market demands. 1.5 Original Components 17. The Project was comprised of two components: 18. Component 1: Performance Agreements (total US$155 million, of which World Bank US$38 million). Provision of support to TEIs through the award of Performance Agreements that would be used to implement initiatives to increase the quality and relevance of tertiary education offerings. The competitively awarded Performance Agreements were designed to be the vehicle for linking public funding and accountability for results at TEIs. Each TEI committed to specific, measurable impacts via a signed contract with the Government of Chile. Disbursement of project funds was tied to accurate, transparent tracking and reporting of performance against indicators included in Performance Agreement contracts. 19. TEIs were to implement Performance Agreements in areas of Teacher Training Improvement, Technical/ Professional Training Improvement, and Academic Innovation (large scale “PMIs�), and for targeted small projects (“PMs�). The implementation of Performance Agreements—including a wide range of TEI-specific strategies and related activities, from improving skills deficiencies of incoming students, to integrating technology into teaching and learning—was expected to, inter alia: (i) increase the number of full-time faculty members with PhDs at accredited TEIs through activities such as targeted recruitment—thereby improving upon the low percentage of PhD holders in Chile’s professorate, which was expected to boost the quality of educational instruction; (ii) reduce the time to student graduation from tertiary education through activities such as credit transfer systems and curricular redesign—thereby increasing tertiary education relevance for both students (who would spend less time and money in acquiring their education and more time in the labor market) and employers (who would have sooner access to trained employees); and (iii) improve the first-year retention rate of students in TEIs, in part due to learning remediation activities and improved relevance of tertiary programs for students—thereby contributing to increased graduation rates and to larger numbers of educated graduates joining the work force. In the process, TEIs would build institutional capacity as they developed strategic plans, competed for MECESUP 3 funds, and implemented successful Performance Agreements. 15 Component 2: Policy Support and Project Management (total US$5 million, of which World Bank US$2 million). 20. Sub-component 2a: Provision of policy support and studies required for the design and implementation of tertiary education reforms and potential institutional arrangements. This sub- component included the execution of studies and a Policy Note that would support the achievement and the measurement of the PDO by fostering more effective institutional analysis at TEIs participating in MECESUP 3, gauging employer perceptions of the quality of training that employees had received at participating TEIs, and establishing guidelines to develop a National Qualifications Framework for all study programs at universities, TTCs and IPs. 21. Sub-component 2b: Provision of support to DIVESUP within the Ministry of Education in the administration, monitoring, coordination and supervision of the Project, to help carry out Component 1 activities. 1.6 Revised Components 22. No revisions were made to the Project’s components. 1.7 Other significant changes 23. There were no other significant changes to the Project. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 24. Project preparation. The Bank had completed significant knowledge work in the Chilean tertiary education sector as preparations began for MECESUP 3. In 2009, the Bank and the OECD published a review of the tertiary system, and in 2010, a Bank/OECD team carried out a review of Chile’s higher education scholarship program. In 2011, the Bank analyzed the effectiveness of Chile’s student loan program.21 This work informed the Bank’s collaboration with DIVESUP to define and prioritize the changes that were needed at TEIs. 25. Project preparation considered public policy priorities such as training new teachers for basic education and improving technical training in order to increase productivity in Chile, as well as MECESUP 1 and 2’s successful interventions in areas including providing PhD training for university professors and strengthening management capacity at TEIs so that they could better support and sustain improvements resulting from Performance Agreement implementation. Once determined, these priorities could be incorporated into terms of reference for the award of Performance Agreements under the new MECESUP Project. 21 Referenced studies: “Reviews of National Policies for Education: Tertiary Education in Chile� (2009); “Human Capital Formation Abroad: A Review of Chile’s Higher Education Scholarships Programme� (2010); “Chile’s State- Guaranteed Student Loan Program: Analysis and Recommendations� (2011). 16 26. Project design. The two previous MECESUP projects provided valuable learning opportunities that guided the design of MECESUP 3. The main learning was incorporated via the expansion of the Performance Agreement financing approach in order to strengthen accountability of TEIs for their performance. Indeed, MECESUP 2 had demonstrated that such agreements could change institutional culture. The process of developing a proposal and committing to specific, measurable impacts via a contract was a powerful catalyst for reform at TEIs. Within the context of a complex tertiary education system, the flexible nature of Performance Agreements allowed for “tailoring� in order to incentivize changes required at the individual TEI level; individual results in the aggregate would contribute to improvements in quality and relevance throughout the tertiary education system. 27. The Bank’s deep knowledge of the Chilean tertiary education sector informed Project objectives, including two pressing needs: restructuring curricula in order to improve their rigor and practical orientation, shorten time to graduation, and increase on-time graduation rates, and increasing remediation programs in order to better support the bottom 40% of students at TEIs. The MECESUP 3 project would also make technical/vocational schools (TTCs and IPs) eligible for Performance Agreement funding for the first time, as students from the two lowest income quintiles tended to enroll in these institutions. Finally, MECESUP 3 would award bigger grants to TEIs to implement improvements at the institutional level. This “scaling up� of complexity was grounded in the successful implementation of institutional Performance Agreements piloted under MECEUSP 2. 28. Project design for MECESUP 3 enjoyed a “head start� in that an Operations Manual had already been designed for the Performance Agreements piloted under the MECESUP 2 Project. The manual included well-defined prerequisites for submitting a proposal, as well as parameters for the “call for proposal,� evaluation, selection, and negotiation phases of the Performance Agreement award process.22 29. MECESUP 3 Project design incorporated what were considered to be realistic timelines for the implementation of Performance Agreements. Funding would disburse during the life of two-year Improvement Plans (PMs for the Spanish acronym) and three-year Institutional Improvement Plans (PMIs for the Spanish acronym), both of which could be extended for up to one year. This would accommodate the requisite approval processes (by the Comptroller General of the Republic, MINEDUC, and internally at the TEIs) that could result in implementation delays. Project design also permitted a range of Performance Agreement funding (from US$175,000 to $4 million) in order to reflect variance in PMs and PMIs based on their degree of complexity and differences in the institutional capacity of TEIs to implement Performance Agreements. 2.2 Project Implementation 30. Within DIVESUP’s Department of Institutional Financing (DFI for its Spanish acronym),23 a team of 42 people across five units was responsible for the successful 22 During the course of MECESUP 3, this Ops Manual was updated three times to reflect procedural changes. 23 The DFI was created within DIVESUP in 2010 with the mandate of rationalizing and optimizing direct public funding of TEIs. 17 implementation and oversight of MECESUP 3. The Academic Unit was the largest, comprised of 13 analysts responsible for monitoring Performance Agreement implementation and progress toward milestones. Initially, each analyst in the team’s Academic Unit was assigned to manage Performance Agreements in one of four areas: Academic Innovation, Teacher Training Improvement, Technical/Professional Training Improvement, or the smaller PMs. In late 2014, the Academic Unit transitioned to a new supervision model based on an institutional perspective. The analysts began working directly with the TEIs implementing Performance Agreements, establishing communication between different initiatives underway and facilitating institutional support for the changes taking effect as a result of Performance Agreements. 31. The Academic Unit relied on support from an Analysis Unit that tracked metrics for Performance Agreement indicators, a Procurement Unit that oversaw procurement activity, an Administrative Unit that managed Project-related budgetary and accounting information, and a Contracts Unit that processed all contractual requirements related to the Performance Agreements. 32. While the MECESUP 3 Project was approved by the Bank in March 2012, it did not go into effect until June 2013. Delayed effectiveness was the result of several factors. First, the Bank signed the Loan Agreement in August 2012, establishing a 120-day period for effectiveness. However, the loan was not countersigned by the Treasurer and Controller General of Chile until March 2013. It therefore had to be reinstated, which took place in June 2013. Next, the Project code for the loan was incorrectly recorded in Chile’s Budget Law and could only be corrected via a supplemental budget request from the GoC, which further delayed implementation following loan reinstatement. Finally, a new Minister of Education entered office in April 2013, which led to an overall slowdown in work at MINEDUC as staffing changes took effect. While the delay impacted the disbursement of Bank funding, Project implementation nonetheless began in late August 2012 using GoC funds to execute the selection, negotiation and award of Performance Agreements. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 33. The Project’s Results Framework was designed to capture the main outcomes resulting from the implementation of Performance Agreements by TEIs participating in the Project (faculty members holding PhDs, duration of undergraduate studies, retention rates for first-year students). This design reflected a time constraint inherent in most higher education projects; while the Project would be implemented over four years, the process of implementing reforms at tertiary education institutions and fully realizing outcomes such as shortened times to graduation take longer. 34. The Project was restructured twice. At the first restructuring in September 2014, the Results Framework was amended with baselines and targets that were not available at the appraisal stage. A Mid Term Review conducted in March 2015 focused on a review of the Performance Agreements under implementation. This led to the Project’s second restructuring, in July 2015, which was an effort to increase the coherence of the Project’s results chain. Two of the original PDO Indicators (one measuring average duration of undergraduate studies until degree award and a second measuring employer perceptions of quality and relevance of 18 graduates of MECESUP 3-supported TEIs) were dropped from the Results Framework, as described in Section 1.3. 35. New PDO Indicators measuring the percentages of students in teaching and in technical/professional degree programs with redesigned curricula were added in an effort to measure improved educational quality and relevance for students. 36. Two sources provided the data required to report on the PDO and Intermediate Outcome Indicators for MECESUP 3: the Tertiary Education Information System (SIES for its Spanish acronym),24 and the electronic registry maintained by DIVESUP. The statistical compendium tracked by each system allowed for straightforward analysis of the impact of changes at TEIs on high-level metrics. This made monitoring and evaluation of the Results Framework simple and reliable. 37. Project teams within TEIs monitored progress toward indicators included in their Performance Agreements, and submitted biannual progress reports to DFI. The analysts in the Academic Unit evaluated the progress and achievements of each Performance Agreement using a method that applied quantitative analysis to qualitative criteria in three categories: evaluation of overall management of the Performance Agreement, achievement of objectives, and sustainability of advances. 38. The information resulting from progress report evaluations was used in two ways. First, it allowed the DFI to identify Performance Agreements where progress had stalled and institutional support was inadequate, and consequently regulate the disbursement of Performance Agreement funding as required. Second, the learning culled from analysis of progress reports enabled analysts to organize workshops and seminars for TEIs on topics pertinent to improved implementation of Performance Agreements (procurement process reviews, information exchanges related to challenging technical aspects of Performance Agreements, etc.). 2.4 Safeguard and Fiduciary Compliance 39. The MECESUP 3 Project triggered Safeguard Policy OP/BP 4.10, given that indigenous peoples could potentially benefit from Project activities. An Indigenous Peoples Planning Framework (IPPF) was designed as part of Project preparation in early 2012 to ensure that MECESUP 3 Performance Agreements promoted improved quality and relevance at TEIs where indigenous students represented more than 14% of total enrollment. 40. In compliance with the IPPF, TEIs were invited to submit proposals for Performance Agreements that would fund pro-indigenous efforts, such as remediation initiatives to strengthen basic competencies (math, science, and reading comprehension) to increase indigenous students’ success in tertiary education. Twelve TEIs submitted proposals, and three (two universities and one IP) were awarded Performance Agreements to implement PMs focused on pro-indigenous efforts, which mainly encompassed providing the students with supplemental tutoring support in 24 The MECESUP 2 Project financed the establishment of SIES. Accredited TEIs are legally required to report data directly to SIES, which analyzes, tracks and disseminates a variety of statistics related to tertiary education and labor market outcomes for graduates. 19 their studies (including training a tutoring corps comprised of indigenous upperclassmen), creating “student centers� where indigenous students can gather, and developing a Mapuche student profile (Chile’s largest indigenous population) that will be useful in devising future support activities for these students. 41. The seven-person Finance Team within the DFI’s Administrative Unit was staffed with experienced professionals who were diligent in overseeing compliance with financial management requirements during the life of the Project. Quarterly interim financial reports and annual financial statements were produced in a timely fashion in accordance with International Accounting Standards. The team submitted prompt audited financial reports that were issued unqualified (clean) opinions. 42. The TEIs implementing Performance Agreements were responsible for financial management and for conducting procurement in accordance with the Operations Manual for MECESUP 3. The financial accounting system that was developed for the MECESUP 2 pilot Performance Agreements was successfully utilized again for MECESUP 3.25 There were no significant delays to the implementation of Performance Agreements due to procurement issues. 2.5 Post-completion Operation/Next Phase 43. The MECESUP 3 team in DFI continues to monitor implementation of the 23 PMI Performance Agreements that were awarded in 2015, all of which will be in effect through 2018. The GoC is using counterpart funds to sustain this work. 44. The Bank is currently in initial planning stages for a US$50M MECESUP 4 Project in 2018, and the MECESUP 3 team at DFI is a valued participant in this work. The GoC is interested in technical and financial support from the Bank in order to strengthen institutional capacity and further enhance quality at public universities. Building on the achievements of MECESUP 3, MECESUP 4 Performance Agreements will focus on improving university research programs and addressing the needs of local labor markets by increasing scientific innovation and producing higher numbers of technically skilled graduates. This next phase of Bank support will help reduce quality gaps in the tertiary education system and benefit less advantaged segments of the student population that enroll in public universities. 45. Under MECESUP 3, a Policy Note was developed that proposed a set of quality standards for all study programs at TEIs and recommendations for an agency to monitor and enforce the quality standards. This note is helping to guide DIVESUP as it establishes a National Qualifications Framework for higher education study programs, which will help to sustain the curricular improvements that TEIs implemented under MECESUP 3. An institutional analysis study was also conducted under the Project. The study provided TEIs with a blueprint that they are currently using to develop their own plans to better manage information and conduct analysis. Improved capacity in these areas will promote even more effective design, 25 Beginning in mid-2017, TEIs still implementing Performance Agreements awarded in 2015 will adopt new accounting software currently being developed by MINEDUC’s Technology Department. T his new software will also be used by TEIs taking part in MECESUP 4. 20 implementation and monitoring of Performance Agreements (both those still being executed under MECESUP 3 and new initiatives under MECESUP 4) going forward. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Objectives: Pre-restructuring rating: High / Post-Restructuring #1 rating: High / Post- Restructuring #2 rating: High Design: Pre-restructuring rating: Substantial / Post-Restructuring #1 rating: Substantial / Post- Restructuring #2 rating: Substantial Implementation: Pre-restructuring rating: Substantial / Post-Restructuring #1 rating: Substantial / Post-Restructuring #2 rating: Substantial 46. Objective. The Project objective of improving quality and relevance for students in tertiary education by strengthening the link between funding of TEIs and accountability for performance remains highly relevant at the time of completion. Diversifying the economy and raising productivity continue to be policy priorities for the GoC.26 Both these challenges require that the quality and relevance of tertiary education be improved in order to provide students with the knowledge, skills and training required in the workplace if Chile is to succeed in a competitive global environment. 47. The Project objective also remains consistent with the Bank’s CPS for Chile (still in place at Project close), which noted that the development priorities supported by the WBG were congruent with efforts to improve the tertiary education sector through a greater reliance on Performance Agreement funding. A 2015 “Performance and Learning Review� of the CPS noted that Chile had advanced significantly toward achievement of CPS outcomes in higher education due to implementation of results-based financing. 48. Design. By linking funding disbursement to pre-defined targets, MINEDUC was able to incentivize the implementation of plans likely to improve the quality and relevance of tertiary education while increasing accountability for performance. TEIs were empowered to define and pursue institutional goals and targets through PMIs, and to implement targeted activities that would improve teaching and learning outcomes through PMs. These institutions were pushed to strengthen management and administrative capacity by coordinating and integrating efforts to reform curricula, programs, and departments, and by working to effectively oversee the implementation of Performance Agreements. 49. Implementation. The MECESUP 3 team in DFI capably oversaw timely and effective implementation of Performance Agreements that improved quality and relevance at TEIs by, inter alia, redesigning curricula, raising faculty qualifications, providing remedial support to students, and supporting credit transfer systems. These collective improvements are expected to enhance employment prospects and wages for graduates of TEIs that participated in MECESUP 3. 3.2 Achievement of Project Development Objectives 26 National Development Plan, 2010-2014. 21 Pre-restructuring rating: Substantial / Post-Restructuring #1 rating: Substantial / Post- Restructuring #2 rating: Substantial 50. Overall achievement of the PDO is Substantial. This assessment relies on the achievement of the Project’s PDO Indicators, the contributions of the Performance Agreements implemented by TEIs, and supplemental data that supports expectations of positive student outcomes. 51. Sector improvements. During the period 2011-2015, QS ranking data for universities in the LAC region shows that the average ranking for the 14 Chilean universities included in the “Top 100 in LAC� moved from the position of 51st to 48th. The improved average ranking was most impacted by the following three criteria:27 1) the “Employer Perception� score rose from 46 (out of 100) in 2011 to 72 in 2015 (a 56% jump); 2) the “Faculty per Student� score rose from 27 in 2011 to 34 in 2015 (a 26% jump); and 3) the “Papers per Faculty� score rose from 56 in 2011 to 68 in 2015 (a 21% jump). The upward movement in scores for the three criteria can surely be attributed in part to improvements realized at universities under MECESUP 1 and 2, as well as early effects of MECESUP 3 Performance Agreements. 52. Achievement of the PDO Indicators. The table below summarizes achievement for the Project’s PDO Indicators before and after two restructurings. The indicators as designed and restructured measure improvements in tertiary education quality and relevance made possible by the implementation of Performance Agreements. Table 1: Summary of PDO Indicator Achievement Pre-September Post-September Post-July 2015 At Project Close 2014 2014 Restructuring (10/31/2016) Restructuring Restructuring PDO #1: No. of full- time faculty holding Target: 4,900 Target: 7,500 Surpassed N/A PhD at all accredited Actual: 6,734 Actual: 8,332 (135% achieved) TEIs PDO Indicator #2: Partially Target: 72% Retention rate for N/A N/A Achieved Actual: 71.3% first-year students (81% achieved) PDO Indicator #3: % of students in teaching degrees with Target: 25% Surpassed redesigned curricula N/A N/A Actual: 71% (282% achieved) at TEIs with Performance Agreements PDO Indicator #4: % of students in Target: 50% Surpassed technical/professional N/A N/A Actual: 89.4% (179% achieved) degrees with redesigned curricula 27 The average ranking is calculated as a weighted average of five criteria: 1) academic reputation; 2) employer reputation; 3) faculty per student; 4) citations per paper; and 5) papers per faculty. The average scores for the “academic reputation� and “citations per paper� criteria also rose during the period 2011 -2015 for the 14 Chilean universities included in the LAC “Top 100.� 22 at TEIs with Performance Agreements 53. At Project close, targets were surpassed for three of the four PDO Indicators included in the Results Framework. One PDO Indicator was partially achieved. Following are the related improvements in the quality and relevance of tertiary education at the TEIs that implemented Performance Agreements under the Project: 54. Highly trained teachers boost the quality of educational instruction for students at TEIs:  Full-time equivalent faculty members who hold PhDs at all accredited TEIs increased from 5,109 at Project appraisal to 8,332 at Project close.  The number of full-time equivalent faculty members with PhDs at TEIs supported by the Project account for 95% of the 8,332 total, or 7,915 faculty members.  The number of graduates from domestic PhD programs supported by the MECESUP 3 Project increased from 103 to 208 during the Project’s life. 55. Increased retention rates contribute to larger numbers of educated graduates joining the work force each year, raising their standard of living:  First-year student retention rates at all accredited TEIs increased from 68.4% at Project appraisal to 71.3% at Project close.  The first-year retention rate increased to 73.8% when taking into account only TEIs that received support from the MECESUP 3 Project. At TEIs that did not participate in the MECESUP 3 Project, the first-year retention rate was only 67.4%.  The retention rate increases to 79.5% when taking into account first-year students at MECESUP 3-supported TEIs who transfer to a different TEI to pursue their second year of study. 56. Study programs that are higher quality and more relevant as a result of curricula redesign enable more students to complete their degrees on time, and to graduate with skills sets better aligned with labor market needs:  The percentage of students pursuing teaching degrees in programs with redesigned curricula at TEIs with Performance Agreements reached 70.5% at Project close. The percentage of students in vocational training programs with redesigned curricula at TEIs with Performance Agreements reached 89.4% at Project close.  These rates are very high compared to all students at TEIs implementing Performance Agreements. Only 34.3% of those students were studying redesigned curricula at Project close.  Preliminary results from an Employer Perception Study conducted under the Project show that employers more positively value the education and training received by teachers who studied at TEIs that received MECESUP 3 support than that of teachers who studied at TEIs that did not participate in MECESUP 3. 23 57. Performance Agreement achievements and expected outcomes for students. Sixty Performance Agreements were awarded to 30 universities, seven TTCs, and six IPs 28 to implement large scale Institutional Improvement Plans (PMIs) in three categories: Teacher Training Improvement, Technical/Professional Training Improvement, and Academic Innovation. Following are some of the innovations and improvements resulting from the PMIs. 29 58. Teacher Training Improvement PMIs focused on redesigning pedagogy curricula, including incorporating a compulsory teaching practicum component and remediation classes for students who require them. TEIs also hired and provided additional training to faculty members to raise the qualifications and instructional level of their teaching staffs, designed and installed Learning Support Centers to monitor and assist students in their academic progress, and developed and implemented “teaching vocation� campaigns as a strategy to attract more highly qualified students to the teaching profession. 59. Technical/Professional Training Improvement PMIs focused on the redesign of vocational training curricula centered on skills aligned with labor market needs. Enhanced curricula integrate more technology into training, incorporate remediation classes for students who require them, and employ credit transfer systems to convert students’ previous coursework and employment into credit units. TTCs and IPs also collaborated with the productive sector to create internship opportunities for their students, and developed networks with foreign technical schools to share knowledge and best practices related to technical training models. 60. Academic Innovation PMIs focused on the redesign of curricula for a wide variety of study programs, as TEIs sought to make their degree programs more skills-based, innovative, and entrepreneurial. Extensive teacher training in the delivery of new curricula was also a component of the PMIs. Many TEIs also broadly expanded the role of Student Support Centers, empowering them to provide remedial and tutoring support, monitor and evaluate student progress, and establish internship programs with local employers. TEIs also signed joint agreements with foreign teaching and research institutions for faculty and student exchanges, compatible undergraduate and post-graduate degree programs, and to pursue research and publishing agendas. 61. One hundred and nineteen Performance Agreements were awarded to 35 universities, five CFTs, and three IPs to implement Improvement Plans (PMs),30 which were targeted activities focused on improvement and innovation in the quality of teaching, learning, and management of TEIs. Each PM was categorized into one of eleven “lines of action�. The majority of the PMs were implemented in four categories: integrating technology into teaching and learning, addressing the skill deficiencies of incoming students, harmonizing curricula, and designing and implementing collaborative projects to impact local communities. 28 Under MECESUP 3, Performance Agreements were awarded to 50% of all accredited universities, 35% of all accredited TTCs, and 32% of all accredited IPs. 29 Of the 43 TEIs that implemented a PMI, 30 TEIs implemented a single PMI, while 13 TEIs implemented more than one PMI. Under MECESUP 3, 85% of all Performance Agreements were awarded to universities. 30 Of the 43 TEIs that implemented a PM with MECESUP 3 funding, 20 TEIs implemented a single PM, while 23 TEIs implemented more than one PM. 24 62. Following is supplemental Performance Agreement indicator data that underpins the expectation that students will experience positive outcomes due to improvements in educational quality and relevance at TEIs that were part of the MECESUP 3 Project. 63. Students who complete study programs with redesigned curricula will graduate more quickly and with skills more aligned to labor market needs, allowing them to work and earn sooner. The following indicators from Performance Agreements point to this outcome:  Credit transfer systems are a component in all curricula redesigned under MECESUP 3. In 2016, the TEIs that received MECESUP 3 support to refine their curricula reported that 245,788 students had earned credit units based on previous coursework and employment, thus shortening their time to graduation.  Universidad Mayor’s redesigned pedagogy curricula ensure that students graduate with a teaching degree in five years or less.  Pontificia Universidad Católica de Chile redesigned its engineering curricula to eliminate duplicative course material and unnecessary testing, and reported that the percentage of students graduating on time rose from 17.5% to 30% during the three-year PMI implementation period. Since redesigning its pedagogy curricula, the percentage of students graduating on time rose from 75% to 88%.  Universidad de Talca redesigned curricula for all of its study programs, and overall on- time degree completion increased from 20% when PMI implementation began to 29% at completion. 64. Students who complete internship programs with local businesses gain practical work experience and improve their employment prospects. The following indicators from Performance Agreements point to this outcome:  There is very limited student employment data from the TEIs that implemented Performance Agreements under MECESUP 3, but what is available is promising. The Universidad de Talca, which implemented an Academic Innovation PMI from 2012- 2015, reports an increase in the graduate employment rate from 81% to 89% during the PMI implementation period. Two TEIs that implemented PMs from 2014-2016 also report significant improvement in graduate employment rates during the implementation period: at the Universidad Católica de la Santísima Concepción, graduate employment increased from 70% to 88%, while at the Universidad San Sebastián, graduate employment increased from 91% to 97%.31 All three TEIs included internship components in their curricular redesign.  Universidad Católica de la Santísima Concepción reported a 79% student internship rate, while DUOC at the Pontificia Universidad Católica de Chile reported that students participating in their internship programs increased from 13,000 in 2014 to 20,000 in 2015. 65. Students who receive remediation support at TEIs are more likely to earn a degree, find meaningful jobs and improve their standard of living. The following indicators from Performance Agreements point to this outcome: 31 U. de Talca and U. Católica de la Santísima Concepcion measure employment at six months following graduation. U. San Sebastian measures employment at 6-12 months following graduation. The DFI has no related salary information for employed graduates. 25  Canete Technical Institute reported a 35% increase in Mapuche student enrollment as a result of efforts to publicize their remediation program in local communities, a 94% retention rate for indigenous students following their first semester (vs. a 90% non- indigenous student retention rate), and that 90% of indigenous students are graduating on time (vs. a 74% on-time graduation rate overall).  Centro de Formación Técnica San Augustín de Talca redesigned all of its vocational training curricula to include remediation programs, and its graduation rate has since improved by 27%. 66. Working students able to access web-based course material to study as their schedules permit are more likely to earn a degree, find meaningful jobs and improve their standard of living. The following indicator from a Performance Agreement points to this outcome:  Centro de Formación Santo Tomas reported that 40% of their student body holds a job while studying, and that all of these students take advantage of 24/7 access to online academic resources. 67. Annex 10 provides an assessment of Project indicator outcomes based on a split evaluation for pre-restructuring and post-restructuring phases. A “Causal Chain� diagram, included in Annex 11, depicts Project indicators that support a causal link between the Project and its outcomes. 3.3 Efficiency Pre-restructuring rating: Modest / Post-Restructuring #1 rating: Modest / Post-Restructuring #2 rating: Modest 68. The Performance Agreements supported by the MECESUP 3 Project promoted an efficient use of resources based on a positive return on investments, results-based allocation of resources, and effective implementation. 69. Positive return on investments. At Project appraisal in late 2011/early 2012, the economic value of the MECESUP 3 Project was analyzed through a net present value (NPV) calculation that measured increased earnings for graduates resulting from higher first-year student retention rates. During the implementation period for MECESUP 3 (2012-2016), the retention rate for first-year students at all accredited TEIs increased from 68.4% to 71.3%. Updating the NPV analysis accordingly to reflect increased first-year retention rates and corresponding upticks in graduation rates yields a positive NPV of Project investments, justifying the economic rationale of the MECESUP 3 Project. (See details in Annex 3.) 70. Allocation of resources based on results. Prior to the MECESUP program, all public resources were allocated to TEIs based on historical criteria rather than merit. Under this schema, TEIs had no incentive to evaluate their performance against pre-determined targets. The MECESUP Performance Agreement model created an incentive to use resources more effectively and efficiently, as funding is results-based. 71. Efficiency of implementation. Under MECESUP 3, 179 Performance Agreements were proposed, negotiated, awarded, and largely implemented in just four years, surpassing the 26 original and revised targets for total signed Performance Agreements. The fulfillment rate for result targets included in the Performance Agreements was 71.3%, suggesting that PAs were indeed being successfully implemented and that TEIs were making progress toward intended improvements in quality and relevance at their institutions. In addition to overseeing and coordinating all aspects of the Project, the resourceful MECESUP 3 team at DFI coordinated the front-loading of GoC counterpart funds during the Project’s first year so that implementation could begin, and transitioned to a new supervision model for Performance Agreements in order to better facilitate institutional support within TEIs for the changes taking effect as a result of PAs. 3.4 Justification of Overall Outcome Rating Pre-restructuring rating: Moderately Satisfactory / Post-Restructuring #1 rating: Moderately Satisfactory / Post-Restructuring #2 rating: Moderately Satisfactory 72. The overall outcome rating for the Project is Moderately Satisfactory, reflecting an overall rating of Substantial for relevance of objectives, design, and implementation (Section 3.1), a Substantial rating for achievement of the Project Development Objective (Section 3.2), and a Modest rating for efficiency (Section 3.3). Post- Pre-September September Post-July 2015 2014 2014 Restructuring Overall Restructuring Restructuring Outcome Ratings: Objective High High High Design Substantial Substantial Substantial Relevance Substantial Implementation Substantial Substantial Substantial Efficacy Substantial Substantial Substantial Substantial Efficiency Modest Modest Modest Modest Moderately Moderately Moderately Moderately Rating Satisfactory Satisfactory Satisfactory Satisfactory Overall Outcome Rating: Rating Value 4 4 4 Disbursement 33.4% 33.4% 33.2% Weight Weighted 1.34 1.34 1.33 4.01 Value Moderately Final Rating Satisfactory 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 73. Performance Agreements implemented under MECESUP 3 have improved the instructional quality of faculty and strengthened curricula at numerous TEIs, increasing the labor market relevance of their study programs. As a result, students are expected to have increased opportunities to engage in fulfilling economic activity, with long-term impacts on both poverty reduction and social development in Chile. 27 (a) Institutional Change/Strengthening 74. TEIs gained valuable experience managing the implementation of Performance Agreements under MECESUP 3. They were able to strengthen capacity in areas including institutional analysis, alignment of resources and activities with priorities, and strategy formulation and execution. TEIs have also made headway in operating with transparency and accountability as they measured and conveyed progress toward defined PA targets. Students, and ultimately the Chilean labor market, stand to benefit greatly from the boost in managerial effectiveness that many TEIs have experienced under MECESUP 3. (c) Other Unintended Outcomes and Impacts 75. While MECESUP 3 did not specifically target improved equity in tertiary education, strides were made in this area. Historically, students from less advantaged backgrounds who embarked on degree programs at TEIs often lacked basic skills, and therefore faced high drop- out rates and slow educational progress. MECESUP 3 funding was used to establish and expand remedial classes; students completing remediation programs in TEIs with Performance Agreements increased from approximately 34,000 in 2014 to 63,000 in 2016. Increased academic readiness will likely account for a reduction in dropout rates for students from lesser socioeconomic backgrounds. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 76. Not applicable. 4. Assessment of Risk to Development Outcome Rating: Low 77. MECESUP 3 Performance Agreements funded improvement plans that individual TEIs conceived of and designed specifically to address issues within their unique institutions. The resulting improvements in quality and relevance are remedies for identified shortcomings, and as such, there seems little risk that TEIs will “backtrack� on progress made via the implementation of Performance Agreements. Rather, TEIs will endeavor to sustain their higher quality, more relevant educational offerings, and to improve on them over time. 78. TEIs are more likely to succeed in efforts to sustain and build upon achievements in quality and relevance due to strengthened institutional capacity gained under MECESUP 3. The work of redesigning entire curricula, training faculty to deliver new curricula, and developing a framework to evaluate and refine curricula going forward required TEIs to push themselves to effectively and efficiently coordinate efforts across their institutions. 79. The MECESUP 3 team at DFI remains committed to overseeing the ongoing implementation of the Performance Agreements that are still in the execution stage, and ensuring that TEIs fulfill the commitments they have assumed under PAs. 28 80. The MECESUP 4 Project will reinforce and build on achievements realized under MECESUP 3, focusing on improving university research programs and addressing the needs of local labor markets by increasing scientific innovation and producing higher numbers of technically skilled graduates. By concentrating on public universities, this next phase of Bank support benefits the less advantaged segments of the student population that tend to enroll in public TEIs. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 81. The Bank prepared a project that 1) was based on sound analysis of sector challenges; 2) reflected lessons learned from MECESUP 1 and 2; and 3) strategically addressed the need to improve aspects of the tertiary education system while holding TEIs accountable for the results of their efforts to do so. (b) Quality of Supervision Rating: Moderately Satisfactory 82. The Bank conducted regular implementation support missions, vigilantly supervised the Project’s safeguard aspect, and enjoyed a cooperative and productive relationship with the MECESUP 3 team at DFI. Through two Project restructurings, indicator baselines and targets were set and revised based on data that was not available at the appraisal stage, and the Results Framework was amended to increase the coherence of the Project’s results chain. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 83. Given the Bank’s Moderately Satisfactory performance both in ensuring quality at entry and carrying out Project supervision, the overall performance rating is Moderately Satisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 84. The Project benefitted from the commitment and continuous support of DIVESUP, whose staff were involved in Project preparation and design, providing direction and guidance on the changes and reforms required at TEIs to improve the quality and relevance of tertiary education for Chilean students. However, there was a significant delay in Project effectiveness, and it is unclear to what extent the GoC could have avoided or shortened the delay. Impact to the Project was minimal only because GoC counterpart funds could be accessed to begin Project implementation. 29 (b) Implementing Agency or Agencies Performance Rating: Satisfactory 85. The MECESUP 3 team at DFI effectively managed the duties required to monitor and evaluate 179 Performance Agreements during the life of the Project. These ranged from tracking metrics for Performance Agreement indicators to convening networks of TEIs that could share implementation strategies and lessons learned as they executed PAs. The team also successfully “improvised� as required. For example, it coordinated the front-loading of GoC counterpart funds during the Project’s first year when project effectiveness was delayed, and transitioned to a new supervision model in late 2014, necessitated in order to better facilitate institutional support within TEIs for the changes taking effect as a result of Performance Agreements. 86. In the end, the Project fully disbursed on time, and the MECESUP 3 team at DFI continues to diligently monitor the implementation of 23 Performance Agreements for PMIs through 2018. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 87. Given the Moderately Satisfactory performance of the Government and the Satisfactory performance of the MECESUP 3 team at DFI, the Borrower’s overall performance rating is Moderately Satisfactory. 6. Lessons Learned 88. The following lessons from MECESUP 3 relate to effective Project implementation:  Performance Agreements are effectively monitored across individual TEIs. The MECESUP 3 team at DFI initially pursued a project-based focus in carrying out Performance Agreement oversight duties. Small teams tracked the progress of individual PMs and PMIs. As the number of TEIs implementing multiple Performance Agreements simultaneously grew, it became clear that a more effective approach was to monitor Performance Agreements in an integrated way across individual TEIs. This strengthened TEIs’ capacity to leverage the work being done via Performance Agreements, as well as to sustain positive outcomes going forward.  Longer implementation timelines for Performance Agreements should be considered. An adjustment to initial implementation timelines for PMs (two years) and PMIs (three years) should be considered. Nearly 40% of all Performance Agreements awarded under MECESUP 3 required an extension, with 63% of those needing an additional year to complete execution.32 The implementation of PMIs in particular can put strain on a TEI, especially in the initial stages. Longer implementation timelines from the start could optimize the execution of larger scale Performance Agreements and potentially lead to increased achievement of goals. 32 Extensions have not yet been requested for the Performance Agreements for PMIs awarded in 2015. 30  Studies intended to inform sector policy benefit from high-level oversight. Under MECESUP 3’s second component, two studies and a Policy Note were formulated. These were primarily intended to inform the design and implementation of tertiary education reform policy. Given the scope of this effort, more direct involvement from DIVESUP would have been beneficial to improve the timeliness of the studies. A dedicated team from the Sub Secretariat could have overseen and guided the development and execution of the studies and note, ensuring that the resulting learnings were fully utilized in a timely way to influence ongoing efforts to refine and adjust those goals as well as strategic initiatives underway. 89. The following lessons from MECESUP 3 provide guidance to the Bank as the MECESUP 4 Project is prepared:  Bank support of the MECESUP Program remains vital. The GoC could use domestic financing to continue a Performance Agreement schema on its own. However, the Bank’s continued engagement in the MECESUP Program is both relevant and warranted based on its deep knowledge of the Chilean tertiary education sector and its productive involvement in the MECESUP 1, 2 and 3 Projects. The Bank will add value to MECESUP 4 by lending its expertise in several strategic areas, including advising on policies that will continue to increase quality, relevance and equity in tertiary education, sharing international best practices in tertiary level teaching and learning, and providing support in the areas of M&E design and implementation to ensure that Project objectives, activities, indicators, and results are fully aligned.  Identify and assess potential administrative delays that could impact Project implementation. In 2012, Project effectiveness was delayed by approximately 15 months. In 2014, the laws governing the Ministry of Education’s Institutional Development Fund (drawn from to complement World Bank financing of Performance Agreements) were changed, and a lengthy process of updating the rules for the competitive award of Performance Agreements for PMIs ensued. In each case, the resourceful MECESUP 3 team averted delays, first by coordinating the use of GoC counterpart funds up front so that the selection and negotiation of Performance Agreements could begin, and then by shifting the schedule for the award of PMIs out by a year until 2015. A future MECESUP project will benefit from up-front work to identify and avoid administrative obstacles that could hinder implementation progress. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies: The DFI’s Borrower Report is included in Annex 7. (b) Co-financiers: Not applicable. (c) Other partners and stakeholders: Not applicable. 31 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions) Component 1: Performance 155.00 155.00 100.00 Agreements Component 2: Policy Support and 5.00 5.00 100.00 Project Management Total Baseline Cost 160.00 160.00 100.00 Physical Contingencies 0.00 0.00 0.00 Price Contingencies 0.00 0.00 0.00 Total Project Costs 160.00 160.00 100.00 Front-end fee IBRD 0.00 0.10 .00 Total Financing Required 40.00 40.10 100.00 (b) Financing Appraisal Actual/Latest Type of Percentage of Source of Funds Estimate Estimate Cofinancing Appraisal (USD millions) (USD millions) Borrower 120.00 120.00 100.00 International Bank for Reconstruction 40.00 40.00 100.00 and Development 32 Annex 2. Outputs by Component 1. MECESUP 3’s PDO and Intermediate Outcome Indicators cannot fully capture the many improvements that have resulted from Performance Agreements. The following information describes the range and varied achievements of three PMIs implemented under MECESUP 3 Performance Agreements. Due to their comprehensive nature, these PMIs contributed to both improved quality and relevance in tertiary education. Performance Agreement for ~US$2,712,400 awarded to the Catholic University of Valparaiso (CUV) for a Teacher Training Improvement PMI Title Increase the quality of instructional performance of teachers trained at CUV Implementation 12/31/2012 – 6/30/2016 (included a 6 month extension33) Dates Objectives 1. Redesign the curriculum to raise the professional competency of graduates of teacher training programs to improve student learning outcomes 2. Increase the quality and retention rate of incoming students 3. Achieve timely degree completion for new teachers 4. Strengthen the link between CUV and the primary/secondary school system as a means to improve initial teacher training 5. Motivate the corps of pedagogy instructors at CUV by involving them in curriculum redesign and providing training to them in “best practice� delivery methods 6. Implement a monitoring system for pedagogy programs, new instructors, and students Best Practices 1. Critical diagnosis: PMI concept was based on an institutional diagnosis of problems and shortcomings in pedagogy programs at CUV 2. Change management model: PMI team was led by Vice Rector of Academics, who ensured consistency in methodology in order to effectively deploy new curriculum across academic units 3. Participation strategy: representatives of academic units were involved in curriculum planning committee, heads of units traveled to observe international experiences similar to the proposed curriculum reforms at CUV 4. Information sharing: PMI team disseminated learnings from Project implementation within a network of TEIs carrying out related projects Results (per 1. New curricula designed for 14 teacher training programs performance 2. Stronger linkages established with the primary/secondary school system via indicators teaching practicums that allowed pedagogy students to test instructional included in PA) methodologies introduced as part of new curricula 3. Nine new PhD holders added to the pedagogy instructor corps 33 Thirty-eight percent of all Performance Agreements required an extension beyond the standard implementation period (two years for PMs, three years for PMIs). Of those, 63% of PAs were extended for 12 months and an additional 24% were extended for six months. 33 Performance Agreement for ~US$1,461,400 awarded to DUOC34 at the Catholic University of Chile for a Technical/Professional Training Improvement PMI Title Improved management of three priority projects: Recognition of Previous Learning (RAP), Training of Trainers, and Implementation of Projects Financed via GoC Institutional Development Fund Implementation 12/31/2012-12/31/2015 Dates Objectives 1. Design an institutional mechanism for RAP so that students’ previous study and work experiences can be applied toward DUOC curriculum requirements 2. Develop and implement a strategy to create a Team of Trainers to lead and coach DUOC faculty in classroom implementation of DUOC’s educational model 3. Strengthen capacity to manage projects funded via MINEDUC’s Institutional Development Fund through implementation of “best practices� in areas of government-based project financing and oversight Best Practices 1. A project management team comprised of sub-teams in charge of each PMI objective worked collaboratively to utilize newly acquired knowledge to solve problems during the course of implementation, enhancing the pace of overall progress 2. DUOC team leading this PMI collaborated with a separate team leading PMI1304 (“Strategies for Curricular Flexibility and Harmony within a Car eer- centric Educational Model�), taking advantage of synergies for effective implementation of both PMIs with the institution Results (per 1. Forty-two degree programs within DUOC now awarding credit based on RAP performance 2. Eighty-seven Trainers certified and working to assist instructors in 42 degree indicators programs across DUOC included in PA) 3. Knowledge management platform designed and implemented to track ongoing academic innovation initiatives in order to assess them for continued internal/external funding Performance Agreement for ~US$1,670,100 awarded to University of La Frontera (ULF) for an Academic Innovation PMI Title Improve international visibility and productivity of doctoral programs at ULF’s Nucleus of Scientific and Technological Development (“BIOREN�) Implementation Dates January 22, 2013 – January 22, 2017 (included a 1 year extension) Objectives 1. Increase visibility of PhD programs by strengthening collaborative networks internationally 2. Improve student mobility and academic exchange with prestigious universities and research centers abroad 3. Increase capacity for scientific and technological research supported by doctoral programs and transfer resulting knowledge to productive sectors of the economy regionally and nationally Best Practices 1. PMI resulted from successful pilot Performance Agreement under MECESUP 2 in the area of scientific productivity 2. Collaboration with foreign universities, specifically in the area of bioresources, fostered joint research experiences between students, scientists and academics, strengthening academic quality and improving R&D processes at ULF Results (per performance 1. Twenty-eight agreements signed to continue collaboration with 34 DUOC (the Spanish acronym for “University Department of Workers and Peasants�) is a college within the Catholic University of Chile that was founded in 1968 to provide technical study programs and degrees to the children of laborers. It now has 17 campuses and 90,000+ students. 34 indicators included in PA) researchers and universities abroad 2. Five agreements signed to strengthen collaborative networks via student exchanges, research internships and award of “double degrees� from ULF and other universities 3. Expansion of physical spaces and research equipment based on design of R&D laboratories at prestigious universities 2. The PMs that were awarded Performance Agreements under MECESUP 3 focused on discrete activities that improved the quality and relevance of tertiary education for students, and also advanced equity within the tertiary education system. Given the large number of PMs that were implemented (119), each was mapped to one of eleven defined “lines of action� that corresponded to Project objectives: Improve Improve Improve Lines of Action Improve both equity of quality for relevance for for Improvement Plans (PMs) quality and tertiary tertiary ed tertiary ed implemented by TEIs relevance education students students system 1. Improve skill deficiencies of incoming students X X 2. Use ICT to manage X information and knowledge 3. Integrate innovation and technology into X teaching/learning 4. Harmonize curricula X 5. Improve experience of X X disabled students 6. Enhance learning of indigenous (and other minority) X X students 7. Knowledge management via doctoral program required by X Law 20.129 8. Develop capacities to design and implement X X programs/agreements based on performance criteria 9. Enhance teaching/learning in STEM (Science, Technology, X Engineering & Math) 10. Design/implement innovative collaborative models X to improve local communities 11. Fortify management capacity at TEIs to better X implement strategy, change and quality improvements 3. Following are descriptions of several PMs that were successfully implemented during the MECESUP 3 Project. 35 TEI / Dates / Line of Action Title of PM PM Objective(s) Accomplishments Amount 1. Improve skill U. Action plan for Improve retention rate 1. Diagnostic deficiencies of Concepcion / success as a for students with evaluation tool incoming students 2014-16 / student deficient academic developed to assess ~US$238,900 backgrounds by reading/writing/ establishing a math skills monitoring system to 2. Mentors assigned to support “high risk� accompany/ monitor students students 3. Delivery times improved for university services available to first- year students (tutoring, financial, and emotional support services) 2. Use ICT to Professional Enhance student Improve teaching and 1. Assessed digital manage information Institute study skills and learning processes by resources available and knowledge “Virginio learning via upgrading digital for all career Gomez� / establishment of resources available for programs in order to 2014-2016 / a digital library all career tracts, and design most pertinent ~US$222,400 facilitate access to digital library these resources 2. Improved the study environment for students (particularly the 40% that maintain jobs while studying) via increased access to wide-ranging digital resources 3. Trained librarians, faculty and students to effectively explore and access the wealth of digital info available via the new platform 3. Integrate U. Chile / Implement a Design “flipped 1. Established training innovation and 2013-2015 / “flipped classroom� team to guide technology into ~US$334,800 classroom�35 in methodology and teachers in use of teaching/learning the School of install related “flipped classroom� Economy and technology instructional methods Business 2. Implemented a “best practice� used by leading universities for innovative teaching/learning 35 The flipped classroom is a pedagogical model in which the typical lecture and homework elements of a course are reversed. Short video lectures are viewed by students at home before the class session, while in-class time is devoted to interactive exercises, projects, or discussions. 36 3. Designed and utilized an evaluation tool to assess effectiveness of courses taught via “flipped classroom� method 4. Shared experiences with other business schools and universities 4. Harmonize U. de Develop a model Analyze graduate 1. Converted previous curricula Santiago / for training programs coursework and 2013-2015 / implementation offered at CRUCH employment into ~US$358,000 of a system of TEIs across disciplines credit units that transferable in order to implement a could be applied academic credits single credit transfer toward graduate- in graduate-level system that level course training incorporates the requirements within programs previous study and all 25 CRUCH TEIs employment experiences of incoming students 5. Improve Catholic U. Developing a Design a blueprint for 1. Developed a experience of Chile / model to reduce fostering improved handbook of disabled students 2014-2016 / barriers to higher inclusion of disabled replicable curricula ~US$264,700 ed for disabled students at TEIs and course students adaptations and technology access enhancements for TEIs 2. Disseminated handbook to TEIs at all levels 6. Enhance learning U. La Academic Integrate and include 1. Reviewed and of indigenous (and Frontera / support program Mapuche students in strengthened the other minority) 2013-2015 / for Mapuche university life via peer tutorial program so students ~US$376,100 students tutoring and that it better serves development of Mapuche students activities to reinforce (22% of student Mapuche socio- population) cultural values and 2. Formed a interests knowledge-sharing network with other TEIs that seek to integrate their indigenous student bodies 3. Established ongoing monitoring of academic performance indicators for Mapuche students (retention rates, grades, degree 37 award) in order to proactively address problems 8. Develop Catholic U. Strengthen Train a Project 1. Developed software capacities to design of Temuco / evaluation, Management team to to manage and and implement 2013-2015 / measurement monitor, measure and monitor data related programs/ ~US$235,800 and monitoring evaluate institution- to institutional agreements based on of institutional level improvement improvements; performance criteria improvement plans, with emphasis disseminated plans on skills transfer, so software platform that other project across university for managers/teams within use by the university can administrators, implement professors, and improvement plans students to ensure more effectively sustainability of improvements 2. Shared improved project management methods (including software) with other TEIs, especially as they related to tracking essential indicators (student retention and graduation rates, etc.) 4. Annex 2 continues on the following pages, with a list of all of the Project outputs (PMIs and PMs) implemented through MECESUP 3 Performance Agreements. 38 Component 1: Performance Agreements Awarded for Implementation of PMIs (Institutional Improvement Plans) Teacher Training Improvement PMIs A total of 19 Performance Agreements awarded to TEIs in this category Name of Tertiary Year of Performance Education Institution Title of PMI Top Achievements of PMI Agreement Award (TEI) 2012 UNIVERSIDAD DE LOS Improve Initial Teacher 1. Implemented Support Center for Reading and Writing ANDES Training at U. de los to assist student academic progression Andes 2. Defined income profiles, and conducted intermediate institutional evaluation tests to promote timely degree award 3. Restructured curricular "nets" and exit profiles 2012 UNIVERSIDAD DIEGO Teachers for the 21st 1. Created and accredited courses "Differential PORTALES Century Pedagogy" and "Pedagogy in English" 2. Implemented a Doctorate in Education (jointly with U. Alberto Hurtado) 3. Increased indexed publications in the area of Education 2012 PONTIFICIA Improve the Quality of 1. Revised curricula for 14 pedagogy study programs UNIVERSIDAD Classroom Teaching 2. Strengthened linkage with the regional school system CATOLICA DE Performance through implementation of a teacher training triad VALPARAISO 3. Identified and hired 9 new faculty members with PhDs 2012 UNIVERSIDAD DE Phase 1 of PMI: Forming 1. Developed 3 courses of pedagogy (kindergarten, CHILE Teachers for Chile secondary ed biology and chemistry) and PhD program in Education 2. Formed Network of Educational Establishments and formalized strategic alliances 3. Created an Institutional Coordination Unit for School of Ed 2012 UNIVERSIDAD DE Teachers as Agents of Redesigned 19 pedagogy curricula, with emphasis on CONCEPCION Change in a Knowledge specific didactics, English language teaching, and Society visualization of pedagogical practices 39 2012 UNIVERSIDAD DE Substantive Changes in 1. Redesigned 17 pedagogy courses PLAYA ANCHA DE Teacher Training: 2. Redesigned teaching practicum in collaboration with CIENCIAS DE LA Processes to Impact School the regional school system EDUCACION Performance and 3. Developed a diagnostic test for teaching performance Strengthen Educational Communities 2012 PONTIFICIA Innovative Teacher 1. Redesigned all 75 courses in the "Disciplinary" and UNIVERSIDAD Training: Integrating "Didactic" lines and all 12 professional practices for CATOLICA DE CHILE Disciplined Teaching and faculty training programs based on the concept of Effective Classroom Skills generative practices 2. Created pedagogy programs in mathematics and science 3. Developed and implemented a pilot continuing ed program in local schools for non-teaching teachers interested in returning to the classroom 2013 UNIVERSIDAD DE Quality Education for All: 1. Created a network of regional primary and secondary TARAPACA A Commitment to Initial schools linked to the School of Ed at U. Tarapacá Teacher Training 2. Developed a model to link student teachers' instructional practices to learning achievements in schools 2013 UNIVERSIDAD Strengthen the Quality of 1. Developed and implemented a "teaching vocation" CATOLICA DEL Teacher Training for campaign, resulting in 12 new students entering teacher MAULE Improved Educational training program in 2016 Performance 2. Developed strategic plan to support undergraduates to ensure academic success, implemented via the Center for Learning Support 3. Redesigned the curricula of 10 pedagogy programs in collaboration with local schools 2013 UNIVERSIDAD Strengthen and Innovate 1. Redesigned all pedagogy programs offered by the CATOLICA DE Initial Teacher Training to School of Ed. TEMUCO Overcome Educational 2. Created the Inter-Faculty Committee to better interpret Shortfalls Affecting and integrate all dimensions of the redesigned programs Students in Araucania 40 2013 UNIVERSIDAD Teacher Education: An 1. Developed and implemented strategies to attract more MAYOR Integrated Model highly qualified new students and to increase student retention 2. Redesigned curricula to ensure that all pedagogy students can be certified within 5 years 3. Developed a network to ensure strong ties between the School of Ed and local and regional schools 2015 UNIVERSIDAD Improvements in Initial PMI in implementation. Main achievement to date: CATOLICA DE LA Teacher Training with Formed a management team for the Project with strong SANTISIMA Emphasis on Collaborative institutional experience. CONCEPCION Work with Regional School Administrators 2015 UNIVERSIDAD Using Teaching Practice to 1. Approved educational model for all pedagogy METROPOLITANA DE Strengthen Teacher programs in the School of Ed CIENCIAS DE LA Training 2. Developed Procedures Manual and Gantt Charter for EDUCACION the redesign of teacher training curricula 3. Signed an international agreement to collaborate on education initiatives with UCLA 2015 UNIVERSIDAD DE Plan to Strengthen Initial 1. Designed evaluation procedure for students admitted SANTIAGO DE CHILE and Graduate Teacher through special exceptions Training to Address 2. Developed a comparative matrix of information for Quality and Equity in each curricula Chilean Education 3. Conducted a research contest to diagnose the needs of local and regional schools 2015 UNIVERSIDAD DE Initial Teacher Training: PMI in implementation. Main achievement to date: MAGALLANES From Attracting Top created a Strategic Inter-Faculty Council. Talent to Leading a Classroom 2015 UNIVERSIDAD DE LOS Strengthening the Teacher PMI in implementation. Main achievements to date: LAGOS Training System: 1. Established an Employers Network. Curriculum 2. Selected 4 students from municipal schools in the Harmonization, Faculty provinces of Osorno and Llanquihue to attend a Talent Development, and Timely and Pedagogical Vocation Program Degree Completion 41 2015 UNIVERSIDAD A Vocation to Teach 1. Organized a Project Team to ensure integration with CATOLICA DEL Innovatively and other concurrent Performance Agreement efforts NORTE Excellently: Strengthening (Propedéutico, PACE, Delta). Initial Teacher Training 2. Professionals hired for Antofagasta and Coquimbo offices 3. Established institutional management methods (Councils of Career Heads, a Practices Plan, Steering Committee guidelines) 2015 UNIVERSIDAD DE LA Improving Initial Teacher PMI in implementation. Main achievement to date: SERENA Training to Enhance created Integrated Institutional Management Unit of Learning in the Coquimbo Pedagogies Region 2015 UNIVERSIDAD Teacher Training Focused 1. Formed Network of Collaborating Colleges for Initial ARTURO PRAT on Integrated Curriculum Teacher Training and Classroom Practice 2. Engaged faculty members in continuing education study 3. Formed Project Team comprised of faculty Technical/Professional Training Improvement PMIs A total of 17 Performance Agreements awarded to TEIs in this category Name of Tertiary Year of Performance Education Institution Title of PMI Top Achievements of PMI Agreement Award (TEI) 2012 IP DUOC UC Improving Management of 1. RAP credit transfer system adopted by 42 programs in Three Priority Projects for 9 of DUOC's professional schools Technical Ed Students: 2. Five cycles of "Train the Trainer" conducted, certifying Recognition of Previous 87 faculty members from all DUOC professional schools Learning (RAP), Train the 3. "Practices System" approved (internships), increasing Trainer, and Practices student participants from 13,000 (2014) to 20,000 (2015) System 42 2012 CFT SAN AGUSTIN DE Consolidation of the 1. Redesigned curricula of 15 degree programs and TALCA "Curriculum by Labor designed 3 new degree programs Skills" Model: 2. 100% of students completed their "early practices" Convergence of Supply 3. 85% of faculty trained in the "Proficiency Training" and Demand for Skills program 2012 IP VIRGINIO GOMEZ Organizational 1. Hired 15 new faculty members Restructuring to Ensure 2. Developed and implemented "Characterization, Curricula Relevant to Monitoring and Student Support" IT system to provide Chile's Productive Sector timely student performance data for institution-wide use 3. Redesigned curricula of 25 degree programs and accredited 25 more 2012 IP SANTO TOMAS Improve Quality, 1. Implemented methodology of teaching and learning Employability and based on simulation techniques as an input to develop Linkages with the modules for machinery equipment simulators (excavator, Productive Sector by motor grader, etc.) Designing and Executing a 2. Trained 62.4% of teachers (1,505 teachers) Curricular Development 3. Refined curricula for 97% of careers Pilot and Implementing a Plan to Advance the Operation of Mobile Equipment 2013 UNIVERSIDAD Redesign Curricula Based 1. Redesigned 18 career programs CATOLICA DE LA on Competencies Linked 2. Trained 81% of teachers (increase from 7%) in SANTISIMA to the Productive Sector Classroom Curriculum Planning, Complex CONCEPCION Methodologies, and Learning Results Evaluation 3. Achieved 79% student internship rate for students at technical college 2013 CFT INSTITUTO Implementation of a 1. Redesigned 18 (100%) career programs TECNOLÓGICO DE Competency-Centered 2. Trained 82% of teachers CHILE S.A. Educational Model 3. Incorporated compulsory "leveling" (remediation) courses in all curricula 2013 IP DE CHILE Strengthening Formation: 1. Teaching model applied to 100% of career programs Quality Teaching and Linkages with the Employment Environment 43 2013 UNIVERSIDAD Towards Pertinent 1. Renovated curricula in consultation with a range of TECNICA FEDERICO Technical Training academics and employers SANTA MARIA 2. Developed linkages between professional technical education (ETP) and university-level technical training (i.e., engineering) 2013 IP ARCOS Modernizing the Arcos 1. Renovated all career programs Educational Model: 2. Created a formal connection with national and local Academic Innovation in media to create internship opportunities for students Artistic, Technological and 3. RAP credit transfer system adopted by all career Digital Curricula in Tune programs with Today's Media 2015 IP LA ARAUCANA Skills-Based Curricula 1. Renovated the curricula of all career programs via Redesign Supported by a consultative process with team composed of curriculum Plan to Strengthen experts, career managers and academics Teaching 2015 CFT CARITAS CHILE Curriculum Based on 1. Created and Implemented a Directorate of Curriculum (ENAC) Labor Skills: Innovation, and Teacher Development Process, and Linkages to 2. Created 5 advisory councils to support curricular the Technological Sector renewal in keeping with technology advances 3. Developed networks to promote student internships and jobs for graduates 2015 CFT PROANDES Consolidation and 1. Formed a project team with specialized roles based on Strengthening of the PMI objectives Educational Model Based 2. Reviewed duel training curricular models on Linkages to Industry 3. Developed and delivered a time load survey to students and analyzed results 2015 UNIVERSIDAD DE LOS A Proposal to Improve 1. Developed "profiles" for leaders in different industries LAGOS Access and Partner with 2. Implemented standards for career manager and career Regional Industry counselor functions within a technical training school 3. Designed and implemented a training program for academic efficiency and quality assurance indicators 2015 UNIVERSIDAD DE Action Plan for 1. Defined organizational structure for a technological MAGALLANES Professional Technical school within the U. of Magallanes and redesigned 6 Training degree programs 44 2015 CFT Universidad Strengthening and 1. Formed a Project Team Valparaiso Consolidating the 2. Developed a network with foreign technical training Educational Model Based institutions to share internationally recognized technical on Skills and Employment training models Possibilities 2015 ARMADA DE CHILE Improving Training PMI in implementation. Main achievement to date: Programs in Twenty formed a team of experts to guide processes of curricular Technical Specialties renewal and Transferable Credit Systems design. 2015 CFT IPROSEC Renovating Training 1. Designed student internship program at TTC San Programs to Foster Student Agustín de Talca to take advantage of their Learning Success and Developing Support Center Strategic Relationships 2. Defined an institution-wide model for providing with Local Industry remediation and monitoring for students 3. Designed a unit dedicated to the teaching-learning process to ensure efficient implementation of educational models Academic Innovation PMIs A total of 24 Performance Agreements awarded to TEIs in this category Name of Tertiary Year of Performance Education Institution Title of PMI Top Achievements of PMI Agreement Award (TEI) 2012 UNIVERSIDAD Action Plan for Curricular 1. Redesigned the exit profile for all 55 career programs CATOLICA DE Harmonization in the offered TEMUCO Context of Cultural 2. Implemented a “Territory and Inter-Cultural Chair� to Diversity support the needs of Mapuche student population (22%) 3. Improved physical infrastructure within classrooms 2012 UNIVERSIDAD DEL Curricular Harmonization 1. Redesigned curricula to incorporate previous study and BIO BIO of Undergraduate work experience per the Credit Transfer System Offerings in Alignment 2. Redesigned curricula for 15 graduate study programs with Labor Market in alignment with the Credit Transfer System Demands 3. Established a Language Learning Center 45 2012 PONTIFICIA Pilot Program for 1. Established new degree designations: Entrepreneur (4 UNIVERSIDAD Interdisciplinary, years), Civil Engineering (5 years), Civil Engineering + CATOLICA DE CHILE Innovative Civil Other Professional Degree, PhD Engineering 2. Diversified civil and industrial engineering training via establishment of 22 majors and 51 minors 3. Improved percentage of students graduating on time from 17.5% to 33% during three-year PMI implementation period 2012 UNIVERSIDAD DE Curricular Harmonization: 1. Redesigned curricula incorporating the SCT model, CONCEPCION Effective Management of with emphasis on "leveling", academic support, and the Educational Model teacher training, with follow-up revision to curricula as required 2. Developed a Manual of Curricular Redesign, and a model for developing and evaluating student skill competencies 3. Installed a Teaching Direction and Teaching Development Unit and expanded the role of the Student Support Center 2012 UNIVERSIDAD DE Skill-Based Training: 1. Updated curricular plans for all career programs with TALCA From Curricular emphasis on skills, innovation and entrepreneurship Transformation to 2. Established integrated support plan for students: Consolidation of the monitoring academic progress, implementing a tutorial Educational Model system, and establishing contracts with the employment sector to employ graduates 3. Implemented "change and improvement" processes to further reinforce a culture of effective student monitoring and evaluation 2012 UNIVERSIDAD Reinventing Engineering 1. Redesigned and implemented 12 engineering programs CATOLICA DEL Programs at UCN 2. Established the Engineering Innovation Unit, which NORTE works in coordination with all related career programs 3. Developed a management model for the "Basic Sciences" category of degree programs 46 2012 PONTIFICIA Internationalization of 1. Consolidated the internationalization strategy (English UNIVERSIDAD Scientific and program and transversal skills development) CATOLICA DE CHILE Technological Doctoral 2. Carried out academic dissemination missions to Programs to Lead the increase foreign student enrollment in UC doctoral Region in the Formation of programs and to promote participation in Winter School Researchers programs 3. Signed collaborative study/degree award agreements with University of the Republic of Uruguay, Katholieke Universiteit Leuven (Belgium), University of Notre Dame (United States), Universita Cattolica del Sacro Cuore (Italy) 2012 UNIVERSIDAD DE Consolidation of Doctoral 1. Formed partnerships with foreign research centers to CONCEPCION Programs in Science, foster student and faculty exchanges Technology and the 2. Created the Doctoral Network to consolidate and Environment coordinate PhD programs in science, technology and environment, and to ensure achievement of commitments and results 3. Rehabilitated the labs and equipment to facilitate doctoral program academic work 2012 UNIVERSIDAD Internationalization of 1. Increased student academic mobility (study and TECNICA FEDERICO Doctoral Programs in research abroad) SANTA MARIA Physical Sciences, 2. Increased number of research projects and joint Biotechnology and publications with leading international science programs Electronics 3. Increased the number of R&D projects carried out in partnership with the private sector 2012 UNIVERSIDAD DE LA Reinforce International 1. Increased scientific productivity via signing of FRONTERA Visibility and Productivity collaborative agreements with international researchers in Bio-Development R&D 2. Arranged dual degree programs between ULF and international universities 3. Improved infrastructure and equipment available to doctoral students to enhance their research work 47 2012 UNIVERSIDAD Internationalization of 1. Increased foreign PhD students (to 30% of first-year AUSTRAL DE CHILE Doctoral Programs to enrollment) Increase Competitiveness 2. Formalized joint research and study agreements with and Productivity in Bio- foreign centers of excellence Sciences, Biotechnology 3. Increased the number of joint publications and faculty and Veterinary Sciences and student exchanges with foreign universities 2012 UNIVERSIDAD DEL Internationalization of 1. Implemented an Associative Management Model to BIO BIO Engineering and consolidate the internationalization process for the UNIVERSIDAD Agricultural Sciences university consortium AUSTRAL DE CHILE Training for a Consortium 2. Supported the institutionalization process in each UNIVERSIDAD of Chilean Universities in consortium member university TECNICA FEDERICO Collaboration with French 3. Produced 87 student exchanges and 9 scientific SANTA MARIA Institutions publications with French counterparts UNIVERSIDAD DE TALCA UNIVERSIDAD DE VALPARAISO UNIVERSIDAD CATOLICA DEL NORTE UNIVERSIDAD CATOLICA DE TEMUCO PONTIFICIA UNIVERSIDAD CATOLICA DE VALPARAISO UNIVERSIDAD DE SANTIAGO DE CHILE 2013 UNIVERSIDAD Design and 1. Created an international career tract within the School ADOLFO IBAÑEZ Implementation of an of Engineering and Sciences, and signed cooperative Engineering Teaching agreements for study and research with overseas Model for a Globalized universities World 2. Accredited the ABET program 48 2013 UNIVERSIDAD DE Students First: Toward 1. Redesigned 40 careers with SCT (36 of these VALPARAISO Curricular Efficacy and implemented by PMI end), ensuring socio-cultural labor Efficiency for relevance and continuous teacher training. Implemented Undergraduate Study an academic management model that includes monitoring Programs of institutional indicators. 2. Implemented university-wide program to support new students through the first three years of university life 3. Gained international accreditation for several dual degree programs (allowing for a student exchange program) 2013 UNIVERSIDAD DE Curricular Harmonization 1. Redesigned 19 study programs with new exit profile ANTOFAGASTA to Ensure Quality Career criteria based on learning results Programs 2. Implemented a program to improve learning in Basic Sciences tracts 3. Implemented a training program on curricular innovation 2013 UNIVERSIDAD Academic and Curricular 1. Redesigned curricula for 15 study programs CATOLICA Innovation to Strengthen 2. Bolstered the teaching corps, hiring instructors with CARDENAL RAUL Undergraduate Learning advanced degrees SILVA HENRIQUEZ 3. Signed agreements with international universities to promote student exchanges 2013 IP DUOC UC Strategies to Harmonize 1. Redesigned and implemented 13 study programs to Curricula in a Skills-Based ensure coherence between DUOC tracts and high school Educational Model courses in Mechanics, Electricity, Accounting, Collective Food Service, Programming, and Connectivity & Networks 2. Signed agreements with 16 universities (in Chile and abroad) for compatible undergraduate and post-graduate degree programs 3. Implemented a pilot program for a Logistics Management study program 49 2015 UNIVERSIDAD DE Generating Value Through PMI in implementation. Main achievement to date: TARAPAC� Curricular Innovation to curricular redesign process underway for all study Enhance the Quality of programs. Degrees Awarded from U. Tarapacá 2015 UNIVERSIDAD DE Transforming PMI in implementation. Main achievements to date: CHILE Undergraduate Training by 1. Formed team and supporting commissions responsible Strengthening the Culture for curriculum renewal process of Excellence and 2. Organized a proposal call for projects to strengthen Diversity teaching, learning and undergraduate evaluation processes 2015 UNIVERSIDAD DE Curricular Harmonization 1. Approved theoretical model for development of the SANTIAGO DE CHILE Plan to Strengthen the Curricular Evaluation Model Implementation, 2. Evaluated attributes of a USC degree through Monitoring and Evaluation implementation of courses offered by the Department of of Innovations in U. Culture and Sport Santiago Degree Programs 3. Piloted a teaching support programs in 4 areas of study 2015 UNIVERSIDAD DIEGO Integrated Leadership and 1. Formalized 4 new management positions within the PORTALES Academic Management Office of Undergraduate Vice Rector: Director of System for Improvements Curricular and Career Development, Director of in Curricula, Teaching, and Teaching, Director of Learning Support and Inclusion, Learning Support and Director of General Formation and English Resources 2015 UNIVERSIDAD DE LOS Successful Training 1. Implemented a cultural and recreational activities LAGOS Trajectories: Challenges of agenda Quality and Equity for 2. Evaluated the role, coverage and mechanisms of self- Vulnerable Students regulation within Student Support Units 2015 UNIVERSIDAD Student Training for 1. Installed a management team of academics from many ALBERTO HURTADO Academic Excellence and units, schools and career tracts to direct PMI Inclusiveness 2. Designed a web-based platform for monitoring all aspects of PMI 3. Strengthened Curricular Innovation Team by hiring three new professionals 50 2015 CFT INSTITUTO Adaptable Learning: An PMI in implementation. Main achievements to date: TECNOLÓGICO DE Innovative, Flexible Plan 1. Developed strategy for implementation of Inverted CHILE S.A. for Skills-Based Learning Classroom Model 2. Designed Operations Manual for a "Digital Design and Production" career tract Component 1: Performance Agreements Awarded for Implementation of 119 PMs (Improvement Plans, also referred to as Small Projects) (Further information about the following PMs is available here: http://www.mecesup.cl) PMs for Line of Action #1: Improve skill deficiencies of incoming students: A total of 24 Performance Agreements awarded to TEIs in this category PMs for Line of Action #2: Use ICT to manage information and knowledge A total of 7 Performance Agreements awarded to TEIs in this category PMs for Line of Action #3: Integrate innovation and technology into teaching and learning A total of 34 Performance Agreements awarded to TEIs in this category PMs for Line of Action #4: Harmonize curricula A total of 14 Performance Agreements awarded to TEIs in this category PMs for Line of Action #5: Improve experience for disabled students A total of 3 Performance Agreements awarded to TEIs in this category PMs for Line of Action #6: Enhance learning of indigenous (and other minority) students A total of 3 Performance Agreements awarded to TEIs in this category 51 PMs for Line of Action #7: Knowledge management via doctoral program required by Law 20.129 A total of 7 Performance Agreements awarded to TEIs in this category PMs for Line of Action #8: Develop capacities to design and implement programs/agreements based on performance criteria A total of 2 Performance Agreements awarded to TEIs in this category PMs for Line of Action #9: Enhance teaching and learning in STEM A total of 4 Performance Agreements awarded to TEIs in this category PMs for Line of Action #10: Design and implement innovative collaborative models to improve local communities A total of 13 Performance Agreements awarded to TEIs in this category PMs for Line of Action #11: Fortify management capacity at TEIs to better implement strategy, change and quality improvements A total of 8 Performance Agreements awarded to TEIs in this category Component 2: Policy Support and Project Management The MECESUP 3 Project financed the execution of two studies: 1. The OAI (Offices of Institutional Analysis) Study was completed in November 2016. The study: 1) evaluated the state of management and analysis information systems at accredited TEIs; 2) determined the level of development of each information system at all accredited TEIs; 3) identified principal factors that influence the level of development of the information systems at accredited TEIs; 4) developed recommendations to improve the management and analysis of information at accredited TEIs. 2. A draft of the Employer Perception Study was sent to the MECESUP 3 team at DFI for review in March 2017. The study evaluates employers' assessments of the quality and relevance of a limited number of PhD and pedagogy graduates hired from TEIs that implemented Performance Agreements under MECESUP 3. The MECESUP 3 Project financed the execution of one Policy Note: 52 1. The National Qualifications Framework for Higher Education Policy Note was completed in August 2016. The note developed a set of quality standards for all degree programs offered by universities, Technical Training Centers, and Professional Institutes, as well as recommendations for the establishment of an agency responsible for monitoring and enforcing the quality standards. 53 Annex 3. Economic and Financial Analysis 1. At Project appraisal in late 2011/early 2012, the economic and financial rationale behind the MECESUP 3 Project was assessed as being sound. The expected economic value of the Project was assessed through three distinct net present value (NPV) calculations, each focusing on an expected outcome associated with MECESUP 3: a shortening of time to graduation, an increase in the quality of tertiary education, and in increase in first-year retention. The appraisal- stage economic and financial analysis showed that there was a high return on the GoC’s investment and that the Project posed no risk to Chile’s macroeconomic and fiscal stability.36 One constraint to the assessment, however, was that the indicators chosen for the results framework at the time did not easily allow for quantitative measurement of tertiary education quality improvements. 2. The expected economic value of the MECESUP 3 Project was newly assessed in early 2017 for the ICR. 3. The expected economic value of the Project is mainly linked to improved first-year student retention and the resulting increase in graduation rates. 4. Additional potential benefits are related to the effect that the Project had on time to graduation and the increase in the wage premium driven by the better quality of the tertiary education system. While the magnitude of the first mechanism is likely to be small since the overall reduction of the time to graduation is small (on average 10 days), the second is likely to be significant but hard to quantify. 5. The economic value driven by the improved first-year retention has to be considered as the lower bound estimate of the economic return of the Project. 6. As part of the new assessment carried out in 2017, the NPV models developed for appraisal were updated to include current data from DIVESUP, and the following information was used to complete the analysis:  Based on the observed patterns, a 4 percentage point increase in first-year retention is assumed. This assumption is based on the net difference between MECESUP and non- MECESUP universities observed at the end of the Project.  The analysis incorporates the 25 accredited universities in the CRUCH consortium.37 Each of these universities was awarded at least one Performance Agreement to implement a PM and/or PMI. Further, of the 179 total PAs awarded, 130 (72.6%) were 36 NPV analysis conducted for Project appraisal was based on US$160M in total funding for MECESUP 3 (US$120M from the GoC, and an additional US$40M from the Bank). 37 CRUCH universities were disproportionate recipients of MECESUP 3 funding because they are the most well established TEIs in Chile, and the most experienced in institutional planning and implementation. They were therefore the TEIs most prepared to respond to proposal calls under MECESUP 3, and most successful in their pursuit of Performance Agreement funding. 54 implemented by a CRUCH university, resulting in 72.3% of all MECESUP 3 funds going to CRUCH universities.  Earnings for 10 years of graduates are projected, beginning in 2013 (when the lending program formally went into effect) through 2022. Gains are expected to continue as reforms at TEIs are sustained.  NPV totals are discounted back to appraisal period 2011 dollars, and reflect 30 years of combined earnings for graduates of CRUCH universities.  A six percent discount rate is used, as it was for the initial NPV analysis conducted in late 2011/early 2012.  Based on 2015 data from DIVESUP, average time to graduate for CRUCH university students is 6.5 years (12.8 semesters). 7. Table 1 below shows a NPV of increased retention under three alternative scenarios for the wage premium increase. As discussed, the wage premium is likely to increase as result of the improved quality, but the size of the increase would depend on possible counteracting effects, such as an increased supply of university graduates, which could reduce the return to higher education. Under all three scenarios, the Net Present Value of the Project is positive and varies in the range between US$57 million and US$170 million. Table 1. Earnings from Increase in First Year Retention Rate New First Graduates Entry Year Graduation NPV of Lifecycle Earnings Increase for First Year from Year Student Year Retention Increase of: Increased s Retention Wage Premium Increase 1.5% 3.0% 4.5% 2012 66,065 2676 2019 13,698,516 27,397,032 41,095,549 2013 69,368 2809 2020 14,021,142 28,042,284 42,063,426 2014 72,837 2950 2021 14,351,366 28,702,733 43,054,099 2015 76,479 3097 2022 14,689,368 29,378,736 44,068,105 Total: 56,760,393 113,520,786 170,281,179 55 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Lending Michael F. Crawford Lead Education Specialist GED02 Maria Paulina Mogollon Senior Private Sector Specialist GTC01 Maria Elena Paz Gutzalenko Program Assistant GED04 Supervision Diego Ambasz Senior Education Specialist GED04 Javier Botero Lead Education Specialist GED04 Venkatesh Sundararaman Program Leader EACNF Patricia de la Fuente Hoyes Senior Financial Management Specialist GGO22 Selene del Rocio La Vera Procurement Specialist GGO04 Nelly Ikeda Financial Management Specialist GGO22 María Virginia Hormazabal Finance Officer WFALA Renata Pantoja Financial Analyst WFALA Lelia Sampaio Werner Senior Finance Assistant WFALA Guillermo Toral Junior Professional Associate Daniela Duran Junior Professional Associate German Nicolas Freire Social Development Specialist GSU04 Janet K. Entwistle Representative AFMLS Marcelo Daniel Barg E T Consultant Alexandra Gonzalez Rubio E T Consultant Rory Narvaez E T Consultant GSU04 Francisco Marmolejo Lead Economist GED06 Chingboon Lee Consultant Hiroshi Saeki Senior Economist GED01 Yves Jantzem Senior Operations Officer GED04 Sara Burga Program Assistant LC66C (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including Number of staff weeks travel and consultant costs) Lending FY 2011 13.48 53,338.71 FY 2012 28.93 179,408.03 FY 2013 1.22 6,938.14 Total: 43.63 239,684.88 Supervision/ICR FY 2013 23.73 115,150.72 FY 2014 32.03 158,096.28 FY 2015 6.10 53,996.34 FY 2016 10.00 49,726.35 56 FY 2017 9.82 43,834.12 Total: 81.68 421,087.20 Supervision/Safeguards FY 2016 2.93 8,021.12 FY 2017 0.95 12,519.76 Total: 3.88 20,540.88 57 Annex 5. Beneficiary Survey Results Not applicable. 58 Annex 6. Stakeholder Workshop Report and Results Not applicable. 59 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR I. Assessment of Project Objectives, Design, and Implementation 1.1. Assessment of Objectives38 The Higher Education Equity and Quality Improvement Program (Tertiary Education Finance for Results Project III, Loan 8126-CL), known as the MECESUP3 Program, was executed over the period 2012-2016, carrying forward and scaling up the work done under previous loans with the International Bank for Reconstruction and Development (IBRD). 39 This opportunity made it possible to continue strengthening the higher education system through a competitive funding format in which institutions undertook initiatives aimed at improving the quality and effectiveness of tertiary education through Performance Agreements (CDs, Convenios de Desempeño) geared toward the fulfillment of objectives in priority areas, strategic positioning of tertiary education institutions (TEIs), and accountability for performance. This approach led to progress in terms of quality, relevance, and equity, thanks to the creation of incentives to encourage the TEIs to make needed improvements in the system, including changes to the curriculum, strengthening of human capital, and efforts in the areas of resource management and efficiency, within a context of increased institutional capacity and regulation by the Ministry of Education (MINEDUC). 1.2. Project Design and Implementation The MECESUP3 Program, a scaled-up version of the pilot MECESUP2 program, is based on the same conceptual model and methodological design, while also including agreements on improvement measures and institutional changes in Chilean tertiary education. In terms of management structure, the Project retained the initial definition of the Division of Higher Education (DIVESUP), which had been responsible for execution of the previous loans. DIVESUP continued to fall under MECESUP General Coordination, institutionalized by MINEDUC under the Department of Institutional Financing (DFI). The DFI had responsibility for monitoring and evaluating execution of the CDs, measuring progress toward the fulfillment of targets, managing finance and procurement, with support from units responsible for direct intermediation with the executing TEIs and internal management (academic, financial, procurement, acts and contracts, and analysis units), among others. 1.2.1. Component 1: Performance Agreements Progress under Component 1, which is based on continued application of a competitive financing mechanism, included establishment of the key role of the performance agreement, understood to be a public agreement between the tertiary education institutions (TEIs) and the State that establishes commitments and institutional development targets in pre-identified priority areas and recognizes outstanding financial performance. This instrument takes the form of a plan aimed at promoting outstanding improvements and changes in alignment with institutional development policies, the agenda for education, and the overall strategic institutional plan. The goal is to strengthen the quality and relevance of tertiary 38 The objective of the Project is to improve the quality and relevance of tertiary education for students by strengthening the link between the funding of tertiary education institutions and accountability for performance. 39 MECESUP1: Higher Education Equity and Quality Improvement Program (1998-2005); MECESUP2: Tertiary Education Finance for Results Project (2005-2010). 60 education through such approaches as remediation, managerial capacity building, improvement of institutional analysis, and development of programs in specific priority areas, including faculty education, academic innovation, technical and professional education, and the development of medium-scale programs, which require targeted actions. Management of the subprojects was entrusted to teams consisting of government officials, academics, professionals, and administrators identified by the TEIs, usually working in coordination with direct counterpart institutional coordination units (ICUs). As part of this process, the DFI deployed strategies for reviewing, monitoring, and following up on the commitments made. While awaiting final signature of the loan in 2012, in order to ensure ongoing scale-up of the CD instrument, the Government of Chile arranged to finance initiatives along the agreed-upon lines of the Project using national resources. Once the loan was finalized in 2013, these initiatives came together to carry on with the processes. a. Types of Performance Agreements Project implementation included two lines of subprojects. The first were institutional improvement plans (IIPs) of up to three years’ initial duration with awards ranging from about $493,000 to $1,990,000 Chilean pesos each, which involved cross-cutting institutional changes in the areas of initial teacher training, academic innovation; and professional and technical capacity building. The second line of subprojects were program improvement plans (PM, small projects), defined as specific or limited projects of up to two years’ duration focused on topics related to academic innovation and management (with awards ranging from about $100,000 to $201,000 Chilean pesos each). In both cases it was possible to extend the subproject for up to one additional year in order to finalize the results. As a general rule, to comply with social safeguard policies, in particular OP/BP 4.10 – Indigenous Peoples, TEIs with an indigenous student population of more than 14 percent were requested to include a plan of action to accommodate those students in their proposals, regardless of the topic area. In addition, the PMs had a specific requirement to incorporate measures to assist indigenous students. b. Call for Proposals and Negotiation of Applications The process of calling for proposals and negotiating the applications involved the following steps: formulation, approval, and dissemination of the criteria; submission of proposals by the TEIs in the form of an improvement plan that specifies strategies and capacity-building actions, targets and indicators, and a budget; review of the eligible proposals by the Evaluation Committee and the Preselection Committee; negotiation of the preselected proposals (TEIs and MINEDUC) to reach agreement on the specific terms; and finally, granting of the award. In the case of IIP applications in 2012 and 2013, the negotiations were conducted by external experts from different academic areas contracted by MINEDUC. However, this arrangement was changed in 2014 and 2015; instead, the negotiations were conducted by a team led by an analyst from the academic unit in question, who coordinated both the internal process (joint review with DFI personnel) and the external process (discussion of adjustments between the TEIs and the external expert). 61 Table 1: Calls for Proposals, by IIP Topic Areas and PM Lines of Action Year of Call for Topic Areas and Lines of Action Proposals Priority areas (IIPs) 2012 2013 2014 2015 Professional and technical capacity building X X X Initial teacher training X X X Curriculum standardization X X X Academic Exchanges and international academic mobility— X innovation undergraduate Internationalization of doctoral programs X X Lines of action (PMs) Improving deficient competency of entering students X X Information management and technical and scientific knowledge X X Technology innovation and integration in teaching and learning X X Curriculum standardization X X* X* Facilities for disabled students X * Effective learning for students of indigenous populations and X X other minorities Advanced knowledge management through accredited doctoral X X programs, pursuant to Law 20,129 Building capacity to design, propose, and implement X performance-based programs and agreements Teaching and learning in science, technology, engineering, and X math (STEM) Design and implementation of innovative models for media outreach for the productive or social advancement of surrounding X communities Institutional management for strategic purposes, management of X change, and improvement of quality *Excludes applications for undergraduate programs already included in CDs awarded in the areas of initial teacher training, academic innovation, and professional and technical capacity- building. **Proposals were not accepted for graduate programs included in CDs awarded in the ID area that were already in execution at the time of the competition. c. Follow-up and Monitoring of Commitments The academic follow-up model had two phases. The first began with initiation of the Project and extended through the third quarter of 2014 and the second one ran from the fourth quarter of 2014 through the end of the Project. During both phases, emphasis was placed on reviewing progress and monitoring implementation, making use of the same tools already established for the Project—namely: semester progress reports and a final report (TEIs), progress assessment reports (Academic Unit in the DFI), and visits and/or meetings with the teams executing the subprojects, including ongoing communication regarding procedures. 62 During the first phase, monitoring was the main role of personnel entrusted with this task (MINEDUC external experts), who worked with the TEIs in the specific areas of action. Follow- up teams were identified for the purpose, consisting of the person in charge of the particular area, those responsible for follow-up, and academic analysts. The second phase saw increased emphasis of the role of the academic analyst who had coordinated project performance during the previous iterations of MECESUP, who was also the person appointed by MINEDUC to keep track of the proposals. Accordingly, the Academic Unit was reorganized with emphasis on institutional follow-up. As needed, external expert advisers were hired depending on the institutions’ needs or on specific areas of action under the CDs. d. Review of Procedures With regard to reviewing the procedures followed in contracting and procurement, this process, which was coordinated by the Procurement Unit, included participation by personnel from the DFI in deciding on the acceptability of expenditure relevance and reporting. The Procurement Unit is specifically responsible for providing technical and administrative support to the TEIs that have been awarded projects on the proper execution of procedures in compliance with IBRD regulations and other requirements established for the purpose. It is also responsible for assisting in the review of annual planning and follow-up in the area of expenditures and eligible procedures based on the regulations in place, monitoring the Procurement Plan Execution System (SEPA, Sistema de Ejecución de Planes de Adquisiciones), and training the teams responsible for its use. With regard to SEPA, making it available to the TEIs was a slow process; it was finally installed and operational for use by the subprojects awarded in 2013 and 2014, as part of the Project’s commitments. During this process the expenditure recording and planning tool was also implemented, as well as the list of goods and services, which was made available to all the initiatives currently in execution. Financial Reporting The Finance Unit was in charge of reviewing the use of funds allocated to each subproject, the reporting on which is in accordance with existing legislation,40 based on the manual on financial reporting, which includes instructions, procedures, and backups to use in the proper handling of funds allocated in the MECESUP3 context, as well as funds committed by the TEIs for the management of awarded projects under the signed agreement. Financial management and reporting is subject to review by the Office of the Comptroller General of the Republic (CGR), which has the authority to monitor quality and efficiency in the management and use of the resources agreed upon under the loan. In 2013 the TEIs were given access to the ICU system, financial management software that makes it possible to keep differentiated accounting records and generate reports on the performance of each of the subprojects. Thus, records and accounting documents were available for the subprojects in 2014 and 2015. The Technology Department in the Undersecretariat of Education is currently making final adjustments in the Budget Control System, which is a condition of the loan. 40 Article 85, Law 10,336, on Organizations and Powers of the Office of the Comptroller General of the Republic; Article 55, Decree Law 1263; Resolution 759 of 2003 of the Office of the Comptroller General of the Republic; and Resolution 30 of 2015. 63 1.3.2. Component 2: Policy Support and Project Management Component 2 was devoted to studies on monitoring the execution of MECESUP3 and support for tertiary education policies and reform processes. It included a series of seminars on related subjects and workshops to define pertinent lines of work and the promotion of competitions to encourage joint activities among stakeholders in the tertiary education system. From the beginning, implementation and administration of the Project was designed to build on the model from the previous loan. There was a General Coordination office that worked with support units to carry out execution. Emphasis was on the creation of technical coordination units for each of the CD areas of action defined under the Project. The institutional structure of MECESUP3 reinforced the teamwork approach that had been followed with the previous loans. By assuming the management of other State tertiary education funds, MECESUP3 was able to articulate their work and activities with its own objectives. The management and coordination between the TEIs and the DFI involved the following units: Procurement and Finance, which reviewed the procedures and reported on expenditures; Analysis: which was responsible for managing the indicators; Academic and Infrastructure, which followed up and monitored fulfillment of the targets and the commitments made in the agreements; Internal Administration, responsible for the management of human and material resources; and Acts and Contracts, responsible for preparing the calls for proposals and for overseeing compliance with regulations, conditions, and agreements. II. Assessment of Project Outcomes vis-à-vis Objectives 2.1.General Outcomes MECESUP3 carried out its work according to plan, obtaining satisfactory outcomes in the areas of finance, development of support initiatives, and tertiary education improvement, as well as continuing to meet the objectives for strengthening the tertiary education system. Calls for Proposals and Projects Awarded Under Component 1, five calls for proposals were initiated – two for PMs (2013 and 2014) and three for IIPs (2012, 2013, and 2015) – leading to the award of $83,843 Chilean pesos for a total of 179 subprojects (60 IIPs and 119 PMs). Table 2. Number of CDs by Area of Action, Type of TEI Award, Amount Awarded (in Chilean Pesos), and Year of Award by Area of Action Subtotal Total Type of CDs by CDs by Area of action of 2012 2013 2014 2015 type of area of TEI TEI action Univ 7 4 8 19 IP Teacher training 19 CFT AF Univ 12 4 6 22 Academic IP 1 1 24 innovation CFTs 1 1 AF Univ 2 2 4 Professional and IP 3 2 1 6 technical 17 CFT 1 1 4 6 capacity building AF 1 1 64 Univ 43 64 107 Medium-sized IP 1 3 4 119 Projects CFT 3 5 8 AF Total CDs awarded 23 61 72 23 179 Legend: AF = armed forces; CFTs = technical training centers; IP = professional schools; Univ = universities. The distribution shows a heavy concentration of universities (Univ), with a total of 152 subprojects, or 85 percent of all awards, compared with 15 awards (8.4 percent) to technical training centers; 11 awards (6 percent) to professional schools (IP), and only one award to an armed forces academy (AF). In the area of social safeguards, three PM awards were granted for support to indigenous students of Mapuche origin in 2013 and 2014, implemented by regional universities (Biobío and Araucanía Region), out of a total of 12 applications (eight in 2013 and four in 2014). 2.2.Studies and Public Policy Notes Studies  Title: “Evaluación de los niveles de desarrollo de la gestión de la información en el sistema de educación superior chileno, medidos a través de los niveles de desarrollo de las oficinas de análisis institucional� (Evaluation of Levels of Development of Information Management in the Chilean Tertiary Education System, Measured by Levels of Development of the Offices of Institutional Analysis). The study made it possible to diagnose the level of analysis and information management in accredited Chilean TEIs, as well as to identify the main uses and elements that generate the development of levels of analysis and information management, with special focus on State universities.  Title: “Percepción de los empleadores en cuanto a la calidad y la pertinencia de los graduados/titulados de las instituciones de educación superior que cuentan con el apoyo del MECESUP3� (Perception of Employers Regarding the Quality and Relevance of Graduates/Degree Holders from Institutions of Higher Learning Who Have Received MECESUP3 Support). This study sought to discover perceptions regarding the quality of education received by teachers and doctorate degree-holders from national doctoral programs (completion in March 2017). Public Policy Note  National Qualification Framework for Tertiary Education. With focus on improving the quality of tertiary education, the National Qualification Framework (MAC, Marco Nacional de Calificaciones) defines qualifications for each of the certifications (degrees and diplomas) granted by the TEIs and the duration of each. 2.3.The Recipient TEIs According to the definition of institutions participating in the model,41 it can be seen that the majority of awards went to universities, especially those in the CRUCH system, and to a lesser 41 The TEI participants in the MECESUP3 program include State universities, so-called “traditional� private universities that belong to the Council of Chilean University Rectors (CRUCH, Consejo de Rectores de 65 extent, private non-CRUCH universities, professional schools, and technical training centers, suggesting the relative weight of these institutions in the educational system. The outcomes in terms of institutional strengthening, management, and improvement of the quality of professional education took the following aspects into account:  Optimization of institutional management. For the most part, the introduction of a strategic planning mechanism in the formulation of proposals for improvement brought with it, by necessity, a strengthening of diagnostic capacity in the TEIs, especially the institutional coordination units and analysis units. This process, already initiated in the earlier MECESUP iterations, was particularly evident in improvements during the definition of action plans, budget management, and information processing.  Improved quality and academic management through the implementation of micro- and macro-curricular harmonization and innovation initiatives, and expanded implementation of SCT-Chile.  Improved infrastructure and equipment. An important element in the process was the improvement of work spaces and technological equipment for strengthening academic education and research.  Increased teaching and management capacity through actions focused on teacher training—specifically, strengthening teaching faculties with doctoral degrees and building human resources capacity in institutional management. III. Evaluation of MINEDUC Performance during Preparation and Implementation of the Project 3.1.Department of Institutional Finance The DFI, under the guidance of the head of the department, demonstrated satisfactory management of the loan and implementation of the MECESUP3 Program; performed all the functions and responsibilities assigned to it; and provided collaboration and coordination through its various support units and team of responsible professionals. a. Follow-up and Monitoring In connection with the execution of subprojects by the TEIs, the DFI completely fulfilled its functions in the areas of: monitoring and academic follow-up of subprojects, by requesting half- yearly and annual reports from the executing TEIs, conducting reviews and providing feedback, and making follow-up visits, and review of the procurement and expenditure execution plan, by requesting monthly or half-yearly financial reports, depending on the nature of the TEIs, and maintaining a record of account statements in the ICU system. The various units participated actively in all these areas, working with the counterpart authorities, officers, and professional teams responsible for the initiatives in each TEI and fulfilling their technical assistance function in each of the required areas. Universidades Chilenas), private non-CRUCH universities, technical training centers, training institutes, and accredited armed forces academies. 66 b. Financial Oversight and Auditing With regard to the response to commitments with the IBRD, financial management remained in the hands of the DFI, with the support of other entities at the State level. The information management and budget execution systems continued to be used for record-keeping and oversight of the subprojects in Component 1 and for parallel records on Component 2. Both the processes and the general procedures were guided by the Operations Manual except when overridden by requirements of the IBRD or legal provisions of the Government of Chile, especially those of the Comptroller General (CGR). The CGR, for its part, conducted regular reviews throughout the process in a timely manner. In addition, financial statements and half-yearly consolidated reports on the Project were submitted to the IBRD, as directed, and they were approved without any suggested modifications. c. Tracking Fulfillment of the Loan Indicators Information on progress toward fulfilling the indicators adopted under the loan was compiled and analyzed. In some cases the information had to be requested directly from the TEIs and/or extracted from their half-yearly progress reports, for each initiative. In other cases it was obtained with the help of the Higher Education Information Service (Servicio de Información de Educación Superior, SIES) from its database. IV. Evaluation of IBRD Performance during Preparation, Implementation, and Supervision of the Project IBRD follow-up and supervision included a review of execution of the agreements on financial management, as well as a review of progress in the implementation of subprojects and the outcomes in the institutions that administered them. Implementation of MECESUP3 emphasizes the link with the responsible IBRD team, especially collaborative engagement in working toward successful development of the Project. However, difficulties were experienced in origination of the loan, with delays in its entry into effect, which were eventually resolved. Furthermore, during execution of the Project there were changes in management that required time for adaptation and transition. With regard to the preparation of the fiduciary tasks, especially those related to procedures, thresholds, updating and use of standard documents, procurement planning, and use of the SEPA platform and the public procurement market, support was received from various counterparts in the IBRD Procurement Unit, all of whom graciously made themselves available to assist in meeting the requirements. Thanks to this experience, it was possible to meet the requirements for the procedures and at the same time have the flexibility to incorporate new aspects (such as concrete use of the public market in the case of State universities), which ended up providing better operational options for procurement and contracting for the subprojects that received awards. Tracking fulfillment of the indicators in the Project Appraisal Document (PAD) proved to be a protracted undertaking because the negotiation period prior to signature of the loan agreement took more than two years. During this long waiting period and during execution of the loan itself, working with the IBRD was a collaborative experience that provided valuable guidance for MINEDUC. An example of IBRD’s flexibility and support was the restructuring of the matrix of 67 indicators, a joint effort carried out during the first half of 2015 which culminated in approval of the new matrix on July 9 of that year. V. Lessons Learned a. Implementation Process  The following aspects of the experience are considered positive: (i) The work with the TEIs was approached from an integrated and institutional perspective rather than by areas of action or specific topics. This approach made it possible to avoid the tension created previously in some of the institutions when there were several different MINEDUC teams (teams organized according to the CD areas of action) and they each approached the institution from their particular point of view, which did not always coincide with the others. (ii) The work was consistently framed in terms of strategic areas (defined according to the CD areas of action), which created the opportunity to discuss other instruments and related areas of public policy that the situation may have called for. (iii) The process of evaluating initiatives underway requires both a qualitative and a quantitative perspective—a viewpoint that considers the milestones and indicators within their own context, weighing them against the stage of development. (iv) The TEIs are now inspired to make major changes in their structure, organizational culture, operational approach, curriculum, and so on, no longer confined by strict academic notions. The entire outlook of the TEIs, their central authorities, and the Ministry itself has been broadened. A major cultural change has emerged from this mechanism, though it was necessary to evaluate and make adjustments to it, in terms of its expectations and mechanisms of operation.  While carrying out studies was a strategic objective of the loan, the lack of a unit responsible for orienting these studies made it difficult to fulfill the commitment agreed upon. The scope of the topics proposed bore out the need to not only coordinate with key MINEDUC units at the central level to ensure their execution and sustainability, but also to define study topics relevant to the work of the DRI, development of initiatives, and management of institutions, with a view to strengthening these development areas. Provision should be made for ongoing support and monitoring of the implementation of public policies to allow for adjustments.  With regard to execution of the CDs, it was necessary to revise the time periods initially proposed for execution of the initiatives—i.e., two years for PMs and three years for IIPs, both extendable for up to 12 months. In fact, a significant proportion of the projects were extended—38 percent, to be exact.42 Of these, 63 percent (43 CDs) had to be extended to the maximum period allowed. But beyond these numbers, it should be recognized that taking on initiatives of the scope of the IIPs has the effect of mobilizing and creating tension not just for a single institution but for all the related institutions as well. This process usually occurs during the first year of execution of the CDs, after which the initiative takes on its own momentum. Thus, allowing for longer periods would make it possible to optimize execution of the proposals, which would be especially helpful when they turn into agreements with targets, milestones, and indicators. 42 The projects from the 2015 competition are all in their first year of execution and therefore have not yet requested an extension. 68 b. Outcomes The loan execution process revealed important lessons about its effects on the tertiary education system and the participating institutions, as well as about the indicators being sought. The first lesson was that framing the proposals as competitive mechanisms for financing improvement plans generated space for institutional strengthening with built-in performance evaluation, thus creating a virtuous cycle for the system. It also strengthened the case for using CDs as a financing alternative that can be extended to other national financing funds. However, although the approach was designed to prioritize institutional needs and provide incentives for improvement, in practice it led to gaps in managerial capacity and strategic planning within the TEIs, as well as to a strong concentration of initiatives in universities as opposed to technical training centers and professional schools, primarily at the expense of technical and professional education. This outcome points to the need for measures that will address the system’s shortcomings, including modifications in the proposed financing model that will recognize the heterogeneity of the participating TEIs and at the same time make it possible to continue making relevant improvements in the areas of development, management, and education quality. Also, the original program design had not anticipated some of the complications that arose depending on the management capacity of the respective TEIs when they participated in more than one activity. The large, complex institutions were able to carry out the activities by creating teams responsible for each of the CDs, requiring a greater effort on the part of the TEIs of average complexity. Still, the challenges of coordinating the various proposed objectives was a crosscutting issue—not a flaw in the design, but rather a question of performance. c. MECESUP3 Performance The development areas encountered changes in the demands posed by the system and in the practice itself of managing the subprojects. In some cases there were requests from the TEIs regarding aspects of managing the resources and reviewing the expenditures because of parameters established under various rules and regulations. Events of this kind occurred at several different points in the process. Insofar as possible, relations with the responsible TEI teams were optimized, as were the expected outcomes. VI. Future Arrangements for Sustainability of the Actions The proposed future arrangements focus mainly on building capacity in the DFI in its role as the entity responsible for handling the financial instruments for strengthening tertiary education in the country. This general idea, above all, is based on the commitment assumed by MINEDUC to continue pursuing the subprojects still being executed and to ensure fulfillment of the commitments assumed by the TEIs. The second priority is to assign further responsibilities to the DFI for the management of other initiatives that continue to focus on strengthening the tertiary education system, but through mechanisms other than the CDs developed under MECESUP3. This new assignment of responsibilities provides the bases for continuity of the work that has already been done, while at the same time, and perhaps even more important, configuring and undertaking new lines of work that will address the particular characteristics of the Chilean tertiary system. 69 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders Not applicable. 70 Annex 9. List of Supporting Documents Department of Institutional Financing, Division of Higher Education, Ministry of Education, Biannual Progress Reports on the Tertiary Education Finance for Results Project III (July- December 2013 - January-June 2016) Department of Institutional Financing, Division of Higher Education, Ministry of Education, Regulations and Challenges of Implementing Performance-Based Agreements in Tertiary Education in Chile, 2011. Graham, Ruth, UNESCO and Aalborg University, Snapshot Review of Engineering Education Reform in Chile, 2017. OECD and The World Bank, Reviews of National Policies for Education: Tertiary Education in Chile, 2009. Reich, Ricardo, El Programa MECESUP, Chile 1999-2016, 2016. The World Bank, Implementation Completion Report (ICR) for the Higher Education Improvement Project, Report No. 34617, 2005. The World Bank, Implementation Completion Report (ICR) for the Tertiary Education Finance for Results Project, Report No. ICR00001840, 2011. The World Bank, Country Partnership Strategy for the Republic of Chile for the Period FY2011- FY2016, Report No. 57989-CL, 2011. The World Bank, Performance and Learning Review of the Country Partnership Strategy for the Republic of Chile for the Period FY2011-FY2016, Report No. 94271-CL, 2015. The World Bank, Tertiary Education Finance for Results Project III (MECESUP 3) project files: Project Information Document (PID): Concept Stage (June 2011) Project Information Document (PID): Appraisal Stage (November 2011) Integrated Safeguards Data Sheet: Concept Stage (July 2011) Integrated Safeguards Data Sheet: Appraisal Stage (November 2011) Indigenous Peoples Planning Framework (IPPF) (January 2012) Project Appraisal Document (February 2012) Project Loan Agreement (August 2012) Project Restructuring Papers (September 2014, July 2015) Project Implementation Status Reports (Sequence 1-9, 2013-2016) Woodrow Wilson School of Public and International Affairs, Graduate Policy Workshop Paper on Chilean Higher Education, 2016. 71 Annex 10. Evaluation of Overall Outcomes Using a Split Ratings Approach (Pre- and Post-Restructuring) Table 1 below summarizes outcomes for the Project’s original PDO and Intermediate Indicators, as well as those indicators tha t were added during the September 2014 and July 2015 restructurings. Table 1. Summary of Outcomes Revised/Added at Revised/Added at Actual at Indicators Baseline Original Target September 2014 July 2015 Project Close (Original 2011) (from PAD) Restructuring Restructuring October 2016 PDO #1: Number of full- time faculty holding PhDs at 4,148 4,900 Baseline: 5,109 8,332 all accredited TEIs 344% Achieved Target: 7,500 N/A 135% Achieved Baseline: PDO #2: Average duration Unis: 12.9 Unis: 13 semesters; of undergrad studies until semesters; IPs: 9.3; N/A IPs: 8.5; TTCs: 6.8 Dropped from N/A degree award at all TTCs: 7.2 Target: Results Framework accredited TEIs Unis: 12.0 semesters; IPs: 8.0; TTCs: 6.5 Targets not met PDO #3: Retention rate (% first-year undergrads 71.3% remaining at TEI for second 68.4% 72% N/A N/A 81% Achieved year) PDO #4: Percentage of students in teaching degrees N/A N/A N/A Target: 25% 70.5% with redesigned curricula at 282% Achieved TEIs with PAs PDO #5: Employers’ perceptions of grads of Downgraded to MECE 3-supported TEIs as Intermediate proxy for increased N/A N/A Indicator N/A N/A employment/earnings 72 PDO #6: Implementation of curricula reform in pedagogy schools at TEIs N/A N/A Baseline: 0% Dropped from N/A with PAs Target: 70% Results Framework PDO #7: Percentage of students in technical and professional degrees with N/A N/A N/A 50% 89.4% redesigned curricula at TEIs 179% Achieved with PAs Intermediate #1: Number of N/A 58 Target: 29 N/A 179 signed PAs 309% Achieved 617% Achieved Intermediate #2: Fulfillment rate for indicators included N/A N/A Target: 70% N/A 71.3% in PAs 102% Achieved Intermediate #3: Project Target: beneficiaries (number), of 270,000 Target: 279,883 which women (%) N/A N/A beneficiaries / 280,000 beneficiaries / 51% female beneficiaries / 51% female 102% Achieved 51% female 99% Achieved Intermediate #4: Number of Baseline: 33,925 students completing N/A N/A Target: Target: 69,497 remediation programs at 50,000 65,000 114% Achieved TEIs with PAs 221% Achieved Intermediate #5: Number of full-time faculty holding N/A N/A Dropped from N/A N/A PhDs in pedagogy schools Results Framework Intermediate #6: Employers’ perceptions of grads of MECE 3-supported TEIs as N/A N/A N/A Dropped from N/A proxy for increased Results Framework employment/earnings Intermediate #7: Number of grads from domestic PhD 530 Baseline: 200 208 programs supported by 433 260% Achieved Target: 103 N/A 108% Achieved Project 73 Intermediate #8: Percentage of teaching degrees with N/A N/A N/A Target: 60% 71.2% redesigned curricula in TEIs 119% Achieved with PAs Intermediate #9: Number of new technical programs introduced/existing N/A N/A Target: 250 Dropped from N/A programs improved Results Framework supported by Project Intermediate #10: Number of technical and professional degrees with redesigned N/A N/A N/A Target: 150 202 curricula at TEIs with PAs 135% Achieved Intermediate #11: Number of policy notes and studies 0 3 (at 12/31/2015) N/A 3 (at 10/31/2016) 3 carried out 33% Achieved 100% Achieved 74 Table 2 below rates Project indicators during the Project’s three phases: Table 2. Indicator Achievement by Project Phase Indicator achievement rating: Phase 1: Pre-September 2014 Phase 2: September 2014 – July Phase 3: Post-July 2015 Restructuring 2015 Restructuring Surpassed (HS = 100+%) PDO #1, PDO #1, PDO #4, #7, Intermediate #1, #7 Intermediate #1, #2, #3, #4, #7 Intermediate #4 Substantially Achieved (S = 85- 99%) Intermediate #3, #8, #10, #11 Partially Achieved (MS = 65- 84%) PDO #3 Not Achieved (MU = 41-64%; U/HU = 0-40%) Intermediate #11 PDO #2 Total: 5 7 7 Percent Surpassed and Achieved: 60% (MU = 3 points) 86% (S = 5 points) 100% (HS = 6 points) Percent of Loan Disbursed: 33.4% 33.4% 33.2% *Rating percentages above from slide 18 of “ICR – Preparation & Evaluation� PPT (January 28, 2016) Table 3 below yields a “Moderately Satisfactory� Overall Outcome Rating for the MECESUP 3 Project. The “Efficacy� rating below takes into account only the achievement of Project indicators. Note that in Section 3.2 of the ICR, the “Efficacy� rating for achievement of the PDO is “Substantial� both before and after the two Project restructurings. This rating takes into account not only the achievement of Project indicators, but also the contributions of the Performance Agreements implemented under the Project and anticipated positive outcomes. The Overall Outcome Rating for the Project in Section 3.2 remains “Moderately Satisfactory.� Table 3. Weighted Project Rating Phase 1: Pre- Phase 2: Phase 3: Post-July Overall September 2014 September 2014 - 2015 Restructuring Restructuring July 2015 Pre- and Post-Restructuring Outcome Rating: Objectives High High High Relevance Design Substantial Substantial Substantial Implementation Substantial Substantial Substantial Efficacy Improve quality and 75 relevance for students in tertiary ed by strengthening Modest Substantial High link between TEI funding and accountability for performance Efficiency Modest Modest Modest Rating: Moderately Moderately Satisfactory Unsatisfactory Satisfactory Overall Outcome Rating: Rating Value: 3 4 5 Disbursement 33.4% 33.4% 33.2% Weight: Weighted Value: 1.00 1.34 1.66 4.0 FINAL RATING: Moderately Satisfactory Table 4. Relationship between Outcome Ratings and Overall Outcome Ratings Outcome ratings used for Relevance, Efficacy, and Relation to ratings used for Overall Outcome: Efficiency: High Highly Satisfactory (6 points) Substantial Satisfactory (5 points) Modest Both Moderately Satisfactory (4 points) and Moderately Unsatisfactory (3 points) Negligible Both Unsatisfactory (2 points) and Highly Unsatisfactory (1 point) 76 Annex 11. The Causal Chain for the MECESUP 3 Project MECESUP 3 strove to improve quality and relevance for Under MECESUP 3, 179 PDO Indicators showed students in tertiary education by Performance Agreements 1. TEIs built institutional improvements in quality and strengthening the link were implemented by 54 capacity as they developed relevance at TEIs made between funding of TEIs and TEIs in areas of Teacher strategic plans, competed for possible by the accountability for Training Improvement, MECESUP 3 funds, and implementation of 179 performance. Technical/ Professional implemented successful MECESUP 3-funded Training Improvement, and Performance Agreements (i.e., a Performance Agreements. Competitively awarded 71.3% fulfillment rate for Performance Agreements Academic Innovation (large scale “PMIs�), and for Performance Agreement Improvements realized by were the vehicle for linking indicators, and 86% of all public funding and targeted small projects TEIs under the Project will (“PMs�). MECESUP 3 funds utilized by result in positive outcomes accountability for results at TEIs to implement Performance TEIs. Each TEI committed to for students of tertiary Implementation of Agreements at Project close) education, including the specific, measurable impacts 2. ~69,000 students from via a signed contract with GoC. Performance Agreements led following: to the following disadvantaged backgrounds Ongoing funding disbursement completed remediation was tied to accurate, transparent improvements in quality 1. Students who complete and relevance at TEIs: programs during the Project* study programs with tracking and reporting of 3. 208 students graduated from performance against indicators redesigned curricula will 1. 3,000+ more full-time domestic PhD programs that graduate both more quickly included in Performance were supported by the Agreement contracts. faculty members with PhDs and on time, allowing them to at accredited TEIs since MECESUP 3 Project, and are work and earn sooner. Improvements in quality and Project start boosting the quality of 2. Students who complete relevance for students at TEIs 2. Retention rate increase of educational instruction for internship programs gain were measured via 4 PDO 2.9% for first-year students at students at TEIs. practical work experience Indicators: accredited TEIs since Project 4. Curricula were redesigned and improve their 1. Number of full-time faculty start for 71.2% of teaching degree employment prospects. members with PhDs at 3. 70.5% of students enrolled programs* 3. Students who take accredited TEIs in teaching degree programs 5. Curricula were redesigned remediation courses at TEIs 2. Retention rate of first-year with redesigned curricula* for 202 technical and are more likely to earn a students 4. 89.4% of students enrolled professional degree programs* degree, find meaningful 3. Percentage of students in in technical and professional employment, and improve teaching programs with degree programs with their *At TEIs that implemented a standard of Performance living. Agreement redesigned curricula redesigned curricula* 4. Percentage of students in technical and professional programs with redesigned curricula 77 MAP 78