Document of The World Bank Report No: ICR00003198 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-74920) ON A LOAN IN THE AMOUNT OF US$20 MILLION TO THE DEPARTMENT OF ANTIOQUIA WITH THE GUARANTEE OF THE REPUBLIC OF COLOMBIA FOR AN UPPER SECONDARY EDUCATION PROJECT March 25, 2015 Education Global Practice Mexico and Colombia Country Management Unit Latin America and the Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective October 24, 2007) Currency Unit = Pesos US$1.00 = US$2,004.58 US$ = COP1 FISCAL YEAR January 1 - December 31 ABBREVIATIONS AND ACRONYMS APL Adaptable Program Lending CAS Country Assistance Strategy CGA Contraloría General de Antioquia DANE National Administrative Department of Statistics (Departamento Administrativo Nacional de Estadísticas) EDATEL Colombian Telecommunications Company EAFIT School for Administration, Finance, and Technology (Escuela de Administración, Finanzas y Tecnología) EPM Public Utilities of Medellín (Empresas Públicas de Medellín) GER Gross Enrollment Rate HD Human Development HEI Higher Education Institution ICFES Colombian Institute for the Promotion of Higher Education (Instituto Colombiano para el Fomento de la Educación Superior) ICR Implementation Completion Report ICT Information and Communication Technologies IDEA Decentralized departmental public entity with administrative autonomy and its own financial resources (Instituto para el Desarrollo de Antioquia) IPP Indigenous Peoples Plan IPPF Indigenous Peoples Planning Framework IRR Internal Rate of Return ISR Implementation Status Report MEN National Ministry of Education (Ministerio de Educación Nacional) M&E Monitoring and Evaluation NBI Basic Unsatisfied Needs (Necesidades Básicas Insatisfechas) NER Net Enrollment Rate NGO Non-Governmental Organization OECD Organization for Economic Cooperation and Development PAD Project Appraisal Document PCPV Portage Council for the Prevention of Violence PDO Project Development Objective SAT Tutorial Learning System (Sistema de Aprendizaje Tutorial) SD Social Development SEDUCA Secretariat of Education of Antioquia (Secretaría de Educación de Antioquia) SENA National Learning Service (Servicio Nacional de Aprendizaje) SEPA Procurement Plan Execution System SHA Secretariat of Finance of Antioquia (Secretaría de Hacienda de Antioquia) SIMAT Student Enrollment System (Sistema de Matrículas Estudiantil) ii SIP School Improvement Project SISBEN System for Selecting Beneficiaries of Social Programs (Sistema de Identificación de Potenciales Beneficiarios de Programas Sociales) Vice President: Jorge Familiar Country Director: Gerardo M. Corrochano Sector Manager: Reema Nayar Project Team Leader: Martha Laverde ICR Team Leader: Marcelo Becerra and Janet Entwistle iii COLOMBIA ANTIOQUIA UPPER SECONDARY EDUCATION (P052608) CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ................................................................ 1 2. Key Factors Affecting Implementation and Outcomes ................................................................ 4 3. Assessment of Outcomes ........................................................................................................... 10 4. Assessment of Risk to Development Outcome .......................................................................... 20 5. Assessment of Bank and Borrower Performance....................................................................... 20 6. Lessons Learned......................................................................................................................... 22 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ............................ 23 Annex 1. Project Costs and Financing ........................................................................................... 25 Annex 3. Economic and Financial Analysis .................................................................................. 39 Annex 4. Bank Lending and Implementation Support/Supervision Processes .............................. 47 Annex 5. Beneficiary Survey Results ............................................................................................ 49 Annex 6. Stakeholder Workshop Report and Results .................................................................... 50 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ...................................... 51 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ........................................ 64 Annex 9. List of Supporting Documents ....................................................................................... 65 MAP ................................................................................................................................................. 1 iv A. Basic Information Antioquia Upper Country: Colombia Project Name: Secondary Education Project ID: P052608 L/C/TF Number(s): IBRD-74920,TF-93829 ICR Date: 02/09/2014 ICR Type: Core ICR Lending Instrument: SIL Borrower: Department of Antioquia Original Total USD 20.00M Disbursed Amount: USD 18.88M Commitment: Revised Amount: USD 20.00M Environmental Category: C Implementing Agencies: Secretariat of Education of Antioquia Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 11/02/2006 Effectiveness: 05/07/2008 05/07/2008 Appraisal: 04/10/2007 Restructuring(s): 12/18/2012 Approval: 11/27/2007 Mid-term Review: 07/19/2010 11/11/2011 Closing: 12/31/2012 06/30/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Unsatisfactory Risk to Development Outcome: Modest Bank Performance: Moderately Unsatisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Quality at Entry: Government: Moderately Satisfactory Unsatisfactory Moderately Implementing Quality of Supervision: Moderately Satisfactory Unsatisfactory Agency/Agencies: Overall Bank Moderately Overall Borrower Moderately Satisfactory Performance: Unsatisfactory Performance: v C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments (if Indicators Rating Performance any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Supervision No None time (Yes/No): (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Secondary education 81 81 Sub-national government administration 11 11 Tertiary education 3 3 Vocational training 5 5 Theme Code (as % of total Bank financing) Education for all 40 40 Education for the knowledge economy 40 40 Social safety nets 20 20 E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Pamela Cox Country Director: Gerardo M. Corrochano Axel van Trotsenburg Practice Manager/Manager: Reema Nayar Eduardo Velez Bustillo Project Team Leader: Martha Laverde Martha Laverde ICR Team Leader: Marcelo Becerra and Janet Entwistle ICR Primary Author: William Experton and Janet Entwistle F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The objective of the Project is to support the Borrower’s long-term vision to improve the employability of young people and increase the competitiveness of the labor force by: (a) increasing equitable access to and retention in upper secondary education; (b) improving equitable education achievement in upper-secondary education; and (c) increasing the proportion of secondary education graduates who continue on to tertiary education. vi Revised Project Development Objectives (as approved by original approving authority) The Project’s PDO remained unchanged throughout the life of the Project. (a) PDO Indicator(s) Original Target Actual Value Values (from Formally Revised Achieved at Indicator Baseline Value approval Target Values Completion or documents) Target Years Increase in the poorest areas upper secondary gross enrollment from 46% to 66%, and Indicator 1 : net enrollment rate from 22% to 36% Value GER 46% GER 66% GER 52.3% quantitative or NER 22% NER 36% NER 26.5% Qualitative) Date achieved 12/31/2006 12/31/2012 06/30/2013 Not achieved (36.5% achieved for gross enrollment and 16.2% achieved for net Comments enrollment). This corresponds to an increase in gross enrollment in the 25 poorest (incl. % municipalities from 12,529 students in 2006 to 14,676 in 2013 and net enrollment from achievement) 5,837 students to 7,426 students. Original: Increase average Mathematics and language test scores in the poorest areas Indicator 2 : Revised: Reduce the gap in achievement test scores (national ICFES test) between public and private school students. Original: Math Public 44 Math Public 47 Math Public 41.4 Language Public 46 Language Public 49 Lang. Public 44.2 Revised (2007): Value Math Pri. 41 Math Pri. 42.1 Math Private 44.8 quantitative or Math Pub 42 Math Pub. 41.4 Math Public 43.6 Qualitative) Gap: -1 Gap: 0.7 Gap: 1.2 Language Pri. 45.0 Lang. Pri. 43.5 Lang. Pri. 44.2 Language Pub. 44.4 Lang. Pub. 43 Lang. Pub. 44.5 Gap:0.6 Gap: 0.5 Gap: -0.3 Date achieved 12/31/2006 12/31/2012 O6/30/2014 06/30/2014 Original: Data correspond to Antioquia’s 25 poorest municipalities. The scores correspond to average results of the SABER 11 exam (designed by ICFES). Since SABER 11 scores are not comparable across time because the content of the tests changes annually, the 2012 restructuring revised this indicator to monitor the gap between public Comments and private schools students to try to measure progress in achievement. However, the gap (incl. % in scores between private and public schools also has weaknesses, since it tends to achievement) oscillate from year to year and the Project targeted both public and private schools. Revised: Surpassed for Language, and partially achieved for Math. The revised indicator did not clearly specify that the gap was being measured only in the poorest areas. However, the values that are reported in the restructuring paper and here are those of the 25 poorest municipalities. vii Indicator 3 : Increase upper secondary completion rate in the poorest areas Value Original: 81.2% quantitative or 90% 85% 72.2% (2012) Revised: 78.5% Qualitative) Date achieved 12/31/2006 12/31/2012 06/30/2014 06/30/2014 Original and Revised: Not achieved. The methodology that was used during the life of the Project to calculate the baseline, progress data and target was the retention rate: the percentage of students who complete grade 11 multiplied by the percentage of students who complete grade 10 (that same year) multiplied by 100. However, the definition of Comments the indicator in the PAD is the proportion of 9th grade students who finish 11th grade. At (incl. % restructuring, the baseline and the target were adjusted according to the new information achievement) provided by DANE (National Statistics Department) and the Ministry of Education (MEN). While the retention rates decreased, the number of students in the 25 poorest municipalities who successfully completed grade 11 increased from 3,453 students in 2006 to 4,317 in 2013 (an increase of 25%, above the 21% average increase for all uncertified municipalities). Increased transition rate from secondary to tertiary education in the beneficiary Indicator 4 : municipalities Value Original: 13% (2006) Original: 19% (2013) quantitative or 18% 31% Revised: 23% (2006) Revised: 29% (2010) Qualitative) Date achieved 12/31/2006 12/31/2012 06/30/2014 06/30/2014 Original: Surpassed. The baseline and target values for this indicator were not available at Project Appraisal. They were established in a 2009 ISR. However, the methodology for measurement was unclear. A reconstruction indicates that the baseline and target were calculated as the first-year enrollment in higher education institutions in the department Comments of Antioquia divided by the enrollment in grade 11 in the uncertified municipalities. (incl. % Revised: At restructuring, the Government changed its methodology to the following: achievement) the number of first-year students from uncertified municipalities enrolled in any higher education institution in Colombia who were enrolled in grade 11 the previous year (thus removing upper secondary graduates who took longer than a year to enroll in a higher education institutions) divided over the number of students enrolled in grade 11 in uncertified municipalities the previous year. (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values documents) Target Years Indicator 1 : Reduced upper secondary dropout rate in the poorest areas Value Original: 7.5% (2006) (quantitative 4.3% 3.5% 5.4% Revised: 5.8% (2006) or Qualitative) Date achieved 12/31/2006 12/31/2012 06/30/2014 06/30/2014 Comments 65% achieved against original target and 17% against revised target. At (incl. % restructuring, baseline and target were adjusted to align with a revised methodology achievement) adopted by the government. Indicator 2 : Proportion of secondary school students from the targeted population who viii receive direct subsidies. Value (quantitative 0% 94% 71.08% or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 Surpassed. This indicator was meant to capture the cumulative percentage of the targeted population that received subsidies over the life of the Project. Instead, it monitors the number of subsidies as a fraction of the yearly target population (estimated at 4,080 at Appraisal and raised to 5,000 in 2009; this is the student population under SISBEN 1 that is not already receiving Familias en Acción transfers, and the student population under SISBEN 1 and 2 in municipalities where Familias en Acción is not present). In 2014, 3,554 students received direct subsidies representing 71.08% of the 5,000 yearly target Comments population. However, in the PAD, the total target population was estimated to be 22,640, (incl. % or around 4,080 times 5.5 years of Project implementation. 23,380 students received achievement) direct subsidies during the life of the Project, representing 103% of the targeted population. The beneficiaries were selected according to the following criteria: students enrolled in grades 10 and 11 belonging to low income families (SISBEN 1 and 2), with a priority given to indigenous students and victims of conflict, who do not benefit from Familias en Acción in schools, and located in the 117 uncertified municipalities with a priority given to schools with the highest dropout rates. School principals agreed to monitor and encourage good performance for students that received direct subsidies. Indicator 3 : Rate of targeting leakage among beneficiary population is less than 5% Value (quantitative NA Less than 5% Less than 5% or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 Comments Achieved. Annual audits were carried out, using a sample of beneficiaries, to verify (incl. % that students receiving the subsidies met the eligibility criteria. Values were achievement) consistently below 5%. Original: Proportion of the 9th grade students in target population who benefited from the conditional scholarship and end up entering into tertiary education institutions. Indicator 4 : Revised: Proportion of the students who benefit from Project interventions receive the Conditional Scholarship. Value (quantitative 0% 15% 52% Unknown or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 06/30/2014 Comments Unknown. In 2014, 3,000 students in Antioquia’s certified and uncertified municipalities (incl. % received a scholarship and 2,728 in 2013. In 2014, the scholarships represented 15.5% of achievement) the 19,347 students who completed grade 11 and 26.9% of the students who applied for a conditional scholarship. It is not possible to calculate an actual value that could be compared to the target since it is not clear how the numerator and denominator were measured. The beneficiaries must meet a number of conditions, including graduating from 11th grade, reaching a minimum ICFES score and enrolling in a higher education institution. The distribution of the scholarships that were granted in 2013 and 2014 by socioeconomic origin was the following: 21% of the students belong to stratus 3, 47% to stratus 2 and 32% to stratum 1. ix 80% of students in grades 10 and 11 who received vocational and professional orientation from a school counselor trained under the Project said they believed they Indicator 5 : received the information they needed to identify suitable education and job opportunities. Value (quantitative 0% 80% Unknown or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 Unknown. 71% of the 451 schools in the 117 municipalities provided vocational and Comments professional orientation to their students, but student opinions on this support is (incl. % unknown. The data comes from a survey done by SEDUCA during the second semester achievement) of 2013. Every year, 80% of schools are able to determine the status of 50% of their graduates 6 Indicator 6 : months after those graduates left school Value (quantitative 0% 80% 70% or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 Comments 88% achieved. 70% of the 451 schools in the 117 municipalities tracked their (incl. % graduates. The data comes from a survey done by SEDUCA during the second semester achievement) of 2013. Indicator 7 : 80% of schools implement a competency-based curriculum. Value (quantitative 28% 80% 86.4% or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 Comments Surpassed. In 2013, 86.4% of the 451 schools in the 117 municipalities implemented a (incl. % curriculum based on competencies and corresponding lessons plans. achievement) 80% of upper secondary rural schools with predominantly indigenous and Afro- Indicator 8 : Colombian student populations implement flexible curricula models adjusted to the cultural context and expectations. Value (quantitative 5% 80% 65% or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 Comments 81% achieved. 65% of the 170 schools with indigenous and afro Colombian students in (incl. % the uncertified municipalities have a curriculum adapted to the cultural context and achievement) expectations of this population. Secondary school principals and coordinators are reported by their staff to use modern Indicator 9 : management skills most of the time. Value (quantitative 0% 80% 90% Unknown or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 06/30/2014 Comments Unknown. 100% of school principals of the 451 schools and 4 teachers per school were (incl. % trained in management. Staff opinions on the use of modern management skills are achievement) unknown. x Teachers who have participated in Project activities have improved their pedagogical Indicator 10 : practices. Value (quantitative 0% 60% 70% Unknown or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 06/30/2014 Comments Unknown. 3,014 teachers and school principals representing 65% of the total number in (incl. % the 451 schools in the 117 municipalities were trained. It is not clear if this training achievement) improved their pedagogical practices. 80% of schools have achieved at least 75% of the objectives of projects that were Indicator 11 : designed to strengthen peace and social cohesion among the student population. Value (quantitative 0% 80% NA or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 Unknown. 298 schools in 69 municipalities out of 451 schools implemented the Comments programs designed to strengthen peace and cohesion among the student population. The (incl. % percentage of the schools who have achieved 75% of the objectives of the program is achievement) unknown. A reform proposal to improve technical training in upper secondary education in the Indicator 12 : Department has been adopted by the government following public consultation with key stakeholders. A reform proposal has been prepared but Value Proposal not formally adopted (quantitative None adopted by Proposal by SEDUCA. Policy or Qualitative) SEDUCA actions were implemented. Date achieved 10/24/2007 12/31/2012 06/30/2014 06/30/2014 Comments (incl. % Achieved against the original target. Partially achieved against the revised target. achievement) The reform (to improve technical training) is implemented in accordance with the Indicator 13 : annual action plans (as amended during the course of implementation). Value Reformed Some policy actions (quantitative None 15% Implemented were implemented or Qualitative) Date achieved 10/24/2007 12/31/2012 06/30/2014 06/30/2014 Partially achieved. Some policy actions in the reform proposal were implemented: (i) 9 schools with technical programs were renovated; (ii) in 2012, an agreement was signed Comments with SENA to ensure eight hours of technical training in addition to the 30 hours of (incl. % academic education, this agreement is in place in 82 municipalities; and (iii) technical achievement) training for adults or out of school youth in 4 Educational Parks (Parques Educativos) (out of 80 planned). School principals, teachers and municipal staff report that they are satisfied with the Indicator 14 : support they are receiving from the Local Advisor Groups. Value (quantitative 0% 75% 90% 96.7% or Qualitative) xi Date achieved 10/24/2007 12/31/2012 06/30/2014 06/30/2014 128% Surpassed against original target and 107% surpassed against revised target. The survey that was done in 2014 showed that 96.7 % of the school principals and Comments teachers of the 451 schools of the 117 municipalities were satisfied of the assistance (incl. % provided by the local advisers. The schools received technical assistance (at least two achievement) visits a year) of the 12 local advisors to: (i) support strategies, instruments and knowledge for pedagogical and school management models; (ii) implement School Improvement Plans; and (iii) use student achievement test results effectively. 80% of upper secondary schools with monitoring and evaluation software installed and Indicator 15 : producing standard reports, and fulfilling their obligations to publish information about their school performance. Value (quantitative 0 80% 100% or Qualitative) Date achieved 10/24/2007 06/30/2014 06/30/2014 Surpassed. The monitoring system is in place and being used in the 451 schools in the Comments 117 municipalities. Report cards are produced annually. Data for monitoring indicators (incl. % is being transmitted on line to SEDUCA. Selected information with regard to achievement) enrollment, drop out, completion rates and SABER results are published. The Department has implemented the impact evaluation, published the results, Indicator 16 : and released a set of proposed policy actions based on the results of the evaluation. Impact Value Evaluation Ongoing impact (quantitative None Reports evaluation or Qualitative) completed and delivered Date achieved 10/24/2007 06/30/2014 06/30/2014 Partially achieved. The final impact evaluation is being prepared by the University of Comments Antioquia. It has been designed, and is expected to be finalized in 2015. In addition, a (incl. % direct subsidies analysis, measuring the impact on SABER scores was done in 2013, achievement) showing a positive impact on language, English and physics, but less so on chemistry and mathematics. G. Ratings of Project Performance in ISRs Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 04/15/2008 Satisfactory Satisfactory 0.00 2 12/20/2008 Satisfactory Satisfactory 0.50 3 06/21/2009 Satisfactory Satisfactory 0.50 4 12/20/2009 Satisfactory Satisfactory 2.00 5 06/29/2010 Satisfactory Satisfactory 3.00 6 02/26/2011 Satisfactory Satisfactory 4.90 7 08/14/2011 Satisfactory Satisfactory 7.89 8 04/15/2012 Satisfactory Moderately Satisfactory 12.19 9 11/15/2012 Satisfactory Moderately Satisfactory 16.04 10 06/24/2013 Satisfactory Satisfactory 16.94 xii 11 01/07/2014 Satisfactory Satisfactory 16.97 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Approved Reason for Restructuring & Key Restructuring Date(s) PDO Changes Made in USD Change DO IP millions 12/18/2012 S MS 16.04  Modification of Component 1 to include: (i) a design of a monitoring system of the Scholarships for Tertiary Education program and (ii) to specify that the design for an impact evaluation will be developed. Originally, the plan was to undertake annual impact evaluations of the program. Since the first scholarships will be distributed in 2013, it is only possible to design the impact evaluation before the Project closes.  Modifications to the Project’s implementation agreements to assign all Project implementation responsibilities to SEDUCA and to eliminate all obligations and references to the Supporting Agency (IDEA).  Revision of three of the Project’s four PDO indicators, and eight of the sixteen intermediate indicators, in part since the government has changed its data sources to measure some outcomes and has created new methods for measuring other outcomes, and to reflect further expected progress during the extended Project period.  Change in Project costs across Components to better align with the Departmental Education Plan and reallocation among categories to account for the peso revaluation and the increase in the cost of some activities.  Modification of the Disbursement Table to: (i) reallocate funds among categories of expenditure, and (ii) clarify that under category (2) goods procured by either the Eligible Schools or SEDUCA are eligible for financing under said category. xiii ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Approved Reason for Restructuring & Key Restructuring Date(s) PDO Changes Made DO IP in USD Change millions  Extension of the Project’s closing date by eighteen months, from the originally scheduled closing date of December 31, 2012 to June 30, 2014 in order to complete the activities and achieve the Project Development Objectives. I. Disbursement Profile xiv 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. The Department of Antioquia had made significant progress in expanding education coverage at the primary and lower secondary level, closing its gap with the rest of the nation. However, access to upper secondary education and tertiary education was low and inequitable. In 2006, the gross enrollment rate in upper secondary education in Antioquia’s uncertified 1 municipalities was 56.4% and the net enrollment rate 30.1% compared to 69.8% and 36% respectively at the national level. Uncertified municipalities were heterogeneous (gross enrollment rates in 2006 ranged from over 100% in the wealthy municipality of San Pedro to under 30% in Nariño), but on average poorer than the capital city of Medellín and the other four certified municipalities. The Basic Needs Index (NBI or Necesidades Básicas Insatisfechas) of the eight certified municipalities was 34.5 (calculated in 2011 with data from the 2005 census), whereas for the uncertified municipalities it was 38.8. The quality of education was low, as evidenced by Antioquia’s performance on national assessments which showed scores below the national average. For many poor families, upper secondary education was unaffordable. As a result, the completion in upper secondary education was low and only 13% could access tertiary education in Antioquia. 2. Antioquia was also plagued by internal conflict and violence, which made the delivery of public services to rural areas difficult. In 2006 Antioquia was the department with the highest number of cases of violence reported, accumulating 53,455 cases, or almost 12% of the national. It was also one of the top recipients of displaced victims, receiving 39,901 people in 2006 (9.7% of the national total), according to Colombia’s Department of Victims (Unidad de Víctimas). 3. Antioquia’s 2004-07 Development Plan aimed at increasing educational coverage focusing on the needs of the rural sectors in uncertified municipalities. Objectives included: (i) increasing the enrollment in lower- and upper-secondary education; (ii) reducing the number of students performing poorly; (iii) better articulating the education system with the productive sector by providing students with general labor competencies oriented toward market demand; and (iv) improving educational opportunities for secondary school graduates by fostering access to higher education. 4. The rationale for Bank involvement included the strong link of the Project to the Country Assistance Strategy (CAS) for 2003-06, in particular the pillars sharing the fruits of growth. The 1 Since the 1990s, the national government began to decentralize public education to the departments and municipalities. Municipalities that have been granted the ability to manage their education systems are said to be certified. There are 125 municipalities in the department of Antioquia. Five municipalities which met capacity requirements had been certified to deliver public education services in their jurisdiction (Law 715 2001) by Project approval, resulting in 120 uncertified municipalities. During Project implementation, three additional municipalities (Apartado and Rionegro in 2009, and Sabaneta in 2010) were certified, bringing the total of uncertified municipalities down to 117. The department of Antioquia is responsible for the administration and provision of educational services in the uncertified municipalities, as well as the provision of technical support to certified municipalities. 1 CAS explicitly mentioned the need to tackle the challenges of “enhancing equity in coverage, quality and (…) efficiency” in basic education especially in areas with internal conflict and violence, very much in line with the Project Development Objective. It was likewise aligned with Colombia’s national development plan for 2006-10 as well as the Country Partnership Strategy for 2008-11, that was under preparation at the time, which emphasized support to the High and Sustainable Growth and Peace pillars. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 5. The PDO is to support the Department of Antioquia's long-term vision to improve the employability of young people and increase the competitiveness of the labor force by: (i) increasing equitable access to and retention in upper-secondary education, (ii) improving equitable education achievement in upper-secondary education, and (iii) increasing the proportion of secondary-education graduates who continue on to tertiary education. The PDO in the PAD differs from the PDO in the Loan Agreement. The third objective in the PAD was: (iii) increasing the proportion of public school secondary education graduates who continue on to tertiary education. 6. Key indicators for rating Project outcomes were: 1. Increase public upper-secondary gross enrollment from 46% to 66% and net enrollment rate from 22% to 36% in the poorest areas. 2. Increase average mathematics and language test scores (national ICFES test) in public schools in the poorest areas. (Mathematics test scores from 44 to 47, Language test scores from 46 to 49). 3. Increase upper secondary completion rate in the poorest areas from 81.2% to 90%. 4. Increase transition rates from secondary public students to tertiary education from 13% to 18%. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 7. The PDO remained unchanged throughout the life of the Project. Three of four PDO Indicators were changed in the 2012 restructuring: 1. The second PDO Indicator was changed to reduce the gap in achievement test scores (national ICFES test) between public and private school students. The change was made to allow comparison of learning outcomes across time since student assessment tests change and are therefore not comparable from year to year. 2. The baseline and target for the completion rate monitored by the third PDO Indicator was changed to 78.5% and 85% due to a change in methodology. 3. The baseline and target for the fourth PDO Indicator that measured transition rates was changed to 23% and 31% due to a change in methodology. 1.4 Main Beneficiaries 8. The Project was intended to primarily benefit the upper secondary students (grades 10 and 11) of the uncertified municipalities of the department of Antioquia. This comprises the approximately 260,000 students who were enrolled in upper secondary education at some time 2 during Project implementation. Beneficiaries by component were: (i) for subcomponent 1.1, the students belonging to the Project specific private schools or NGOs selected to implement the flexible pedagogical model in the 117 uncertified municipalities: approximately 8,937; (ii) for subcomponent 1.2, students in grade 10 and 11 who belong to families in SISBEN 1 who received direct subsidies (but do not already receive subsidies from the program Familias en Acción) and students from families in SISBEN 22 in schools which are in 117 uncertified municipalities and in municipalities with the highest rate of drop outs in grade 9: the number of subsidy beneficiaries was 23,380; (iii) for subcomponent 1.3, the students who received conditional scholarships for tertiary education belonged to the same targeted group and in addition meet a number of conditions including graduating from 11th grade, reaching a minimum ICFES score and enrolling in a higher education institution. These include at least 11,075 students. The school improvement sub-projects in component 2.1 benefitted academic counselors, principals, coordinators and teachers who received training. 3,014 teachers, principals and other school professionals were trained. Component 3 benefitted administrators of the Secretariat of Education of the department of Antioquia, which we assume to be around 100. This brings the total number of beneficiaries to around 306,506. 1.5 Original Components (as approved) 9. The Project consisted of three components: Component 1: Equitable access and retention to upper secondary education and opportunities for transitioning to tertiary education. This component financed: (i) on a competitive basis the implementation of flexible pedagogical models in private schools or NGOs; (ii) direct subsidies to upper secondary students from poor families in the uncertified municipalities to help with expenditures such as tuition fees, transportation and foods; (iii) conditional scholarships to students from poor families in the uncertified municipalities to increase access to tertiary education, and (iv) a program for guiding and counseling students at the upper secondary level with respect to vocational training and for providing employment prospects and access to tertiary education. Component 2: Quality and relevance of upper-secondary education. This component financed: (i) technical assistance to provide tools and instruments to the Department to develop competency based curricula as well as training in management and collective leadership strategies and school grants to strengthen upper secondary school management, implement competency based curricula and improve the pedagogical practices of teachers; and (ii) technical assistance to carry out a feasibility study for improving the effectiveness of secondary technical education and provide financial transfers to eligible schools to implement selected subprojects. Component 3: Institutional strengthening, monitoring and evaluation and Project management. This component aimed at: (i) strengthening the capacity of the Secretariat of Education of Antioquia (SEDUCA - Secretaría de Educación de Antioquia) to provide technical assistance to the municipalities and schools; (ii) implementing an impact and evaluation strategy; and (iii) strengthening SEDUCA to comply with its responsibilities for managing the Project. 2SISBEN is a proxy means based test tool used to target social programs to the poor. It has six levels, with 1 being the poorest and 6 being the richest. 3 1.6 Revised Components 10. The 2012 restructuring modified component 1: (i) to include the design of a monitoring system for conditional scholarships to increase access to tertiary education; and (ii) to specify that the design for an impact evaluation of the conditional scholarship program would be developed, instead of annual evaluations of the program as had been originally envisaged. 1.7 Other significant changes 11. The 2012 restructuring: (i) assigned Project implementation responsibilities for component 1.3 to SEDUCA instead of Instituto para el Desarrollo de Antioquia (IDEA); (ii) revised eight of the sixteen intermediate indicators to reflect changes of data sources and/or new methodologies and expected progress during the extended Project period; (iii) changed Project costs to better align with the Departmental Education Plan; (iv) modified the Disbursement Table to reallocate Bank funds among categories of expenditure, and to clarify that under category (2) goods procured by either the Eligible Schools or SEDUCA are eligible for financing under the category; and (v) extended the Project closing date for 18 months from December 31, 2012 to June 30, 2014 in order to complete all activities and achieve the Project Development Objective. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 12. The Project was prepared by SEDUCA (during the period July 2006 to December 2007) with a series of six studies financed by a US$500,000 Japanese grant.3 It built on the experience of the previous Antioquia Basic Education Project (P006891), which strengthened schools at the preschool, primary, and lower-secondary levels, and increased local management capacity. It incorporated lessons from previous decentralized projects in Antioquia, Pasto, and Cundinamarca, along with the ongoing Rural Education Project and the cash transfer project Familias en Acción in Colombia as well as on a series of consultations with stakeholders and lessons from international experience. The main lessons learned included: (i) inclusion of school grants, (ii) application of a participatory approach involving stakeholders at all levels of Project implementation to reduce resistance, (iii) mobilization of additional resources for the sector, such as funds from municipalities, (iv) use of cash transfers to stimulate demand (an approach that has proven successful in a number of cases, such as Oportunidades in Mexico), (v) public-private partnerships, (vi) empowering schools as the centers and agents of change, and (vii) reform technical secondary education curricula. The preparation studies could have included an institutional assessment for instance: (i) of the Gilberto Echeverri Fund to manage the conditional scholarship program; or ii) to identify the Universad Católica del Norte and the SAT (Sistema de Aprendizaje Tutorial) as major actors of a virtual platform for distance education. 3The studies were the following: i) Economic Analysis of the Project (Econometría Consultores), ii) Design of the Student Cash Transfer Program (Econometría Consultores), iii) Profile of Upper Secondary Schools in Antioquia (Corpoeducación), iv) Design of a Monitoring and Evaluation System (Luis Piñeros), v) Proposal for the Use of ICT (Álvaro Galvis), and vi) Design of Competence-Based Curricula (Econometría Consultores). 4 13. The Project was well designed technically with a good balance between measures to improve the quality of the supply of upper secondary education through sub-projects presented by the municipalities and the mobilization of additional resources for the sector and measures to stimulate demand and improve equity and participation through cash transfers and scholarships to students from low-income families. It used a decentralized and participatory approach to prepare sub-projects within the School Improvement Plan framework. This approach empowered local school communities, reduced the resistance to change, facilitated the dissemination of successful education alternatives. The subprojects aimed at integrating the different categories of inputs in order to generate a transformation process and to liberate creativity of all the stakeholders. 14. Project management was fully integrated in the regular administrative structure of SEDUCA. This facilitated implementation, ensured better compliance with Bank rules and increased sustainability. Project components were well targeted on upper secondary schools in the uncertified municipalities and on students from low income families. 2.2 Implementation 15. Mining activities, drug micro-trafficking and violence with the support of some guerilla groups impacted the education system and the Project in different ways: (i) lower student participation and increase in the dropout rate; (ii) direct violence to students; (iii) forced displacement of population; and (iv) money laundering in some NGOs related to education. In response, Project activities which favored a school culture of peaceful coexistence and prevention of violence became an important element of School Improvement Plans and were mainstreamed in the Department’s education policy. 16. In 2009, the Government of the Department of Antioquia established free public education from grades 0 to 11. 4 In 2010, free public education was extended to the 17 teacher training institutions. In the same year, stricter and more homogeneous rules for evaluating and promoting students were implemented. This particularly affected the pass rate in grade 10.5 17. The election and the first change of Administration in January 2008 did not affect the implementation of the Project. The new authorities were supportive of the Project, the selection of the Project coordinator took place at the Project launching and the Project implementation team was quickly established and operational.6 18. An election and second change of administration took place in January 2012 and subsequent changes in the Project slowed down implementation. The Ley de Garantía avoided the use of public budget during election time. As a result, all procurement was suspended during eight months from January 2011 to the end of September 2011 and implementation slowed down. The Project Coordinator and all the technical staff (3 financial specialists, 1 procurement specialist, 1 evaluation and monitoring specialist, 4 school advisors and the 3 advisors for peace and coexistence) were changed after the election. As a result, there was no Project implementation 4 This decision was formally adopted by Ordinance 10 in August 2010 and at the national level in 2012. 5 The rules were approved through national level Decree 1290. 6 During the first semester of 2008, 23 workshops were conducted in the 9 regions of Antioquia with 440 school principals, 400 academic coordinators, 35 representatives of the Municipal education Board and the 22 Municipal Secretaries of Education. 5 team from January to July 2012 when the new nominations took place. The new team was trained in the Bank’s financial and procurement procedures. The impact on Project disbursement was minimal (disbursements picked up in 2012) because major contracts had been prepared in 2011 and launched during the last semester of 2011. The new Administration confirmed that education was its top priority and its willingness to use the Project as the main channel to promote the transformation of upper secondary education. 19. The sub-component to provide conditional scholarships to increase access to tertiary education was implemented only for two years starting in 2012. The Project mid-term review identified the need to establish a new fund to replace the Gilberto Echeverri fund, which had been initially considered to manage the conditional scholarships supported under the Project. In 2012, this new scholarship fund was established, which was managed by IDEA and financed by the Department of Antioquia (63%), with financing also provided by the City of Medellín (18.5%) and the EPM Foundation (18.5%). 20. While the Project’s direct subsidies to students attending upper secondary education began in 2009, additional funding from the private sector was never raised as envisaged. EAFIT (Escuela de Administración, Finanzas y Tecnología), a higher education institution focusing on management, finance and technology, developed the training model for the school principals and also provided support for it in 2013 and 2014. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization Design 21. The Results Matrix included four Project Development Indicators to monitor progress toward the achievement of the PDO, and 16 Intermediate Indicators to monitor implementation progress. It did not include any PDO Indicators to measure the employability of young people and increase the competitiveness of the labor force as these were long-term goals beyond the scope of the Project as indicated in the PDO itself. 22. To monitor the objective of increasing equitable access to upper secondary education, the Results Matrix included a PDO Indicator that measured upper secondary gross and net enrollment rates in the 25 poorest municipalities. Monitoring of the objectives to increase equitable retention and improve equitable educational achievement was also only in the poorest of Antioquia’s uncertified municipalities. These indicators, however, were not good measures of Project impact since most Project activities targeted all of Antioquia’s 117 uncertified municipalities and not just the poorest twenty five. The Project did target students from low-income families, but there was no evidence that these families were concentrated in the 25 poorest municipalities. The PAD, instead, identified the need to close the gap between the lower gross enrollment rates in Antioquia’s uncertified municipalities and its certified municipalities and the gap with national rates.7 Reducing these gaps would have been more appropriate measures of Project impact on equitable access. An alternative option would have been to measure the percentage of poor students accessing upper secondary education (in all 117 uncertified municipalities) since the 7 See PAD, Annex 1, Section D “Coverage and Equity.” 6 direct subsidies that were expected to increase access and retention were focused on students from low income families (but data is not available for this option). 23. A fourth PDO Indicator monitored the objective to increase the proportion of secondary education graduates who continue on to tertiary education. A baseline was to be introduced based on 2007 data of the transition rate from secondary to tertiary education. No plans were made to introduce a baseline for the number of secondary education graduates even though the PDO in the legal agreement included an objective to increase the number of secondary education graduates. The PAD did not include a methodology to calculate the transition rate. 24. The intermediate outcome indicators tracked implementation progress of most activities with the exception of the implementation of flexible pedagogical models, and the activities related to school improvement sub-projects. Only, the number of subprojects signed, the amount of financing per subproject and their percentage of implementation and achievement were to be monitored during Project implementation but not their achievements. Most intermediate indicators were designed to be monitored through percentages, with the methodology for calculating them often unclear. Implementation 25. SEDUCA closely monitored progress toward targets in the Results Matrix. Substantial efforts were made to involve the municipalities and the participating schools. Indicators in the Results Matrix were also tracked at the school level. Each school participating in the Project had a score card that included key indicators. 26. There were some practical issues with collecting monitoring data for some indicators: (i) it was difficult to measure on a yearly basis the transition rate from secondary to tertiary education because individual students could not be tracked through their identification numbers (the number of the identity card used at tertiary education level is different from the number used for the identification of students in secondary) and universities do not publish data on their students; and (ii) calculating gross and net enrollment rates relied on projections from a population census done in 2005 that was not adjusted for high internal migration or other effects of violence that plagued Antioquia during the Project period, resulting in inaccurate data. In addition, although the PAD indicated that the completion rate was to be calculated based on the proportion of 9th grade students who finished 11th grade, throughout Project implementation it was calculated based on the proportion of 10th grade students that finished 11th grade. Data was not available to classify and track students by level of poverty in all the uncertified municipalities. Finally, data on the numbers behind the percentages used to track intermediate indicators was never recorded, making it difficult to monitor Project outputs. 27. To measure improvements in equitable educational achievement, the Project relied on the national student assessment test, SABER 11. SABER 11 is administered by Instituto Colombiano para el Fomento de la Educación Superior (ICFES) in grade 11. The test is obligatory (but not necessary to complete grade 11) and a prerequisite for accessing tertiary education. Since the content of the test evolved with the curriculum, SABER 11 scores are not fully comparable from one year to another. The Results Matrix was revised during the 2012 restructuring to provide an 7 alternative measure of equitable educational achievement through a reduction in the gap in SABER 11 scores between public and private school students in Antioquia’s uncertified municipalities. Since the Project supported activities in both private (i.e. the flexible pedagogical models) and public (i.e. curricula reform, SIPs, language and math programs) upper secondary schools in Antioquia’s 117 uncertified municipalities, a more appropriate measure of Project impact on equitable educational achievement would have been to measure the gap between SABER 11 test scores nationally and in Antioquia’s certified municipalities compared to Antioquia’s uncertified municipalities. The 2012 restructuring also adjusted the baseline and targets for the PDO indicators that measured completion rates and the transition rate and revised the methodology for both. The PDO Indicators measuring equitable enrollment and retention were not revised to measure Project impact in all 117 municipalities instead of just 25. 28. SEDUCA complemented its monitoring of the Results Matrix with specific evaluations: (i) a diagnostic test to evaluate student readiness the SABER 11 exam (pre-SABER 11) was applied in 2010 to grade 10 students (results were used for the Bank 2013 flagship study Building Better Teachers); (ii) a classroom observation study (Stallings method) was done in 2011 in grade 11 (showing that pedagogical time in grade 11 was 63% compared to OECD average of 85%; the data was later used in the book Great Teachers, published by the Bank in 2014); (iv) a review of tertiary education led by OECD was done in 2011 with Bank participation; (v) a study was carried out in 2013 by Econometría to measure the impact of subsidies on SABER 11 results; and (vi) a study was done in 2014 on policy options to improve education quality in Antioquia (University of Antioquia). Utilization 29. School score cards were widely used not only by school principals and teachers but by other stakeholders, including parents, the municipality and the policy makers in the Department of Antioquia. They were used on a daily basis by a team of assessors. Two issues were identified at the Project’s midterm review: (i) the devolution of information from SEDUCA to the schools was considered to be insufficient to inform technical decisions at the classroom level; and (ii) the information system needed to be improved to provide more reliable data. These two issues were addressed in 2013 and some of the data that was previously inflated was corrected. The baseline and targets for the transition rate from secondary to tertiary education (13% and 18%) were added through the ISR in 2009. 2.4 Safeguard and Fiduciary Compliance Environmental safeguards 30. This was a category C project because it included only minor school rehabilitation and maintenance, mainly to improve sanitary facilities and adaptation of classrooms for computer installation. Nevertheless, guidelines (including provisions such as basic health and safety precautions for workers, management of dust and noise levels, and disposal of construction waste) were incorporated into all bidding documents for contractors. 8 Social Safeguards 31. The Project triggered Operational Policy 4.10 (Indigenous Peoples) and the Borrower prepared an Indigenous Peoples Plan (IPP) to address the needs of the Indigenous population (0.48 percent of the population of Antioquia). It also prepared an Afro-Colombian Peoples Planning Framework to address the demands of the Afro-Colombian population (23 percent of the population of Antioquia). The Project financed the design and implementation of a competency based curriculum that respected cultural identity and ethnic differences. Of the 140 upper secondary schools participating in the Project that had Indigenous and Afro- Colombian populations, about 65% implemented a curriculum adapted to these populations. Financial Management 32. At the preparation stage, SEDUCA’s and SHA’s (Secretaría de Hacienda de Antioquia) financial management arrangements were appropriate. Several fiduciary measures were included to mitigate financial management risk, particularly risk associated with the management of funds at the school level. Still in the early years, there was inadequate monitoring of grants to the schools and direct subsidies to the students. Audit reports done by the Contraloría Departamental de Antioquia from 2008 to 2009 were unacceptable, and a private firm was hired by the Project to carry out audits. During the life of the Project, the external auditors qualified their opinion on the Project financial statements, but based on the Bank’s review of audit reports, the issues were satisfactorily resolved, except for one aspect that did not have a material impact on Project financial information. From June 2013 to the Project closing date, acceptable financial reports were submitted to the Bank. Procurement 33. Project procurement arrangements and action plans were adequate, reflecting the recommendations of a Procurement Capacity Assessment. Antioquia has strong capacity in procurement since it had experience with a previous Bank financed education project. However, in 2010 the number of qualified staff assigned to procurement was insufficient. This issue was addressed with the hiring of two procurement staff in 2011. In addition, there were issues with the distribution of duties between the Procurement Specialist and the Project coordinator, and with the procurement files system (SEPA system). The results of an ex-post procurement review that took place in May 2013 were positive in general and the Project archives were fully updated. Compliance with covenants 34. All legal covenants stipulated under the Loan Agreement related to institutional arrangements, Project monitoring reporting and evaluation and procurement were met with no delay. 2.5 Post-completion Operation/Next Phase 35. While there is no follow up Bank-financed operation envisioned, several initiatives supported by the Project continue to be supported by the Department of Antioquia: (i) the 9 conditional scholarship to increase access to tertiary education; (ii) the vocational and professional counseling and guidance; (iii) the use of the School Improvement Plan to provide resources to make secondary school more effective; (iv) decentralization at the municipal and school level through structured and systematic technical assistance; and (v) school based management, monitoring of indicators and the management by results. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Rating: Substantial 36. The objectives of the Project are fully in line with the priorities of the Country Partnership Strategy for Colombia for FY12-16, approved by the Bank’s Board of Executive Directors on June 12, 2011, to expand opportunities for social prosperity in particular by increasing enrollment in education in upper secondary education including vocational and technical education. The CPS explicitly mentions “pronounced disparities in access and quality of secondary” education as an obstacle for the country, and sets an increase in secondary enrollment of students from poor and rural households as an outcome. The Department of Antioquia Education Development Plan (approved in May 2012) included a specific orientation to improve the quality of secondary education. The objectives in this Plan which are similar to the ones of the Project are: (i) to increase equitable access and retention in upper secondary education; (ii) to improve the quality of learning; and (iii) to better coordinate upper secondary education with technical and vocational training and higher education in order to improve the transition to the labor market. The Project is also in line with the Colombian National Decennial Education Plan 2006-16 (Plan Nacional Decenal de Educación 2006-16), which set targets for, among others: (i) increasing net coverage in upper secondary education, (ii) improving the performance of students and schools in SABER 11 scores, and (iv) increasing higher education coverage. Pre-restructuring rating: Substantial Post-restructuring rating: Substantial 37. The direct subsidies and scholarships that targeted public school students from low income families were an appropriate intervention to promote access of students from low income families and thus increase enrollment and improve retention in upper secondary education.8 Introducing flexible pedagogical models, such as online learning, is a good low cost option to improve access to education in remote rural areas with few students such as in Antioquia’s Bajo Cauca and Urabá subregions. There is emerging literature that school based management can improve learning outcomes, and thus equitable educational achievement particularly when it is introduced in tandem with training of school principals in schools with underperforming students as was done under the Project.9 Introducing a competency based curriculum that promotes healthy behaviors, 8 Attanasio, Meghir, ansd Santiago (2012) find, for instance, that the Mexican conditional cash transfer Progresa had a strong impact on school enrollment, especially after primary school. Soares, Ribas and Osório (2010) also find increases in school enrollment as a result of the Bolsa Família conditional cash transfer program in Brazil. 9 Evidence of links between SBM and improved learning outcomes is mixed. Evidence from Kenya shows that parent-teacher associations lead to better learning outcomes (Duflo et al 2014) when parents are empowered through specific capacity building. In The Gambia, SBM was shown to reduce student and teacher absenteeism; SBM only had a positive effect on learning 10 such as team work and responsibility, and critical thinking can also boost academic achievement. Scholarships to higher education, as well as offering workshops and support services to students, are key to increasing access to tertiary education for students that finish secondary school and receive a sufficient score on the SABER 11 exam. There were some minor shortcomings in design since the Project did not factor in the short supply of accredited higher education institutions in Antioquia. Over the long-term, the introduction of the competency-based curriculum in secondary schools, more secondary school graduates with improved test results, and more students enrolled in higher education could be expected to lead to improved employability of young people and increased competiveness in the labor force. The Project design remained relevant throughout implementation. 3.2 Achievement of Project Development Objectives Increase equitable access to upper-secondary education Rating: Substantial 38. Access to upper secondary education in Antioquia’s 25 poorest municipalities increased modestly. The gross enrollment rate reached 52.3% in 2013, falling far short of the targeted 66%, and the net enrollment rate reached 26.5% significantly less than the 36% target (Table 1). Since the indicators in the Results Matrix were inadequate to measure increases in equitable access in all of the 117 municipalities supported by the Project, the ICR introduces additional evidence of Project outcomes in these municipalities. Equitable access to upper secondary education in Antioquia’s uncertified municipalities did increase significantly, as evidenced by a reduction in the enrollment gap between the national level and Antioquia’s 117 uncertified municipalities. The gap between Antioquia’s uncertified municipalities and the national level was reduced from 12.5 percentage points in 2007 to 9.9 percentage points for the gross enrollment rate and from 5.9 to 4.0 percentage points for the net enrollment rate. Additionally, the selection mechanism for the 25 poorest municipalities could have been improved. Ranking the uncertified municipalities in Antioquia by their NBI index10 (Necesidades Básicas Insatisfechas, or basic unsatisfied needs), which is a standard poverty indicator in Colombia, shows that the 20 poorest, for instance, grew substantially in coverage and converged with the national average during Project implementation. outcomes in communities with high literacy, otherwise the effect could be potentially negative in communities with low literacy (Blimpo, Evans 2011). In Indonesia, SBM was found to have links to improved learning outcomes only when school based management is complemented by linkages between the school committee (Pradhan et al. 2014). Evidence also shows that SBM has improved social participation, governance, transparency, and accountability, leading to lower dropout and repetition rates (Gertler et al 2012). A recent observational study suggests that school autonomy may lead to improved learning outcomes when strong central institutions are in place, but otherwise possibly detrimental (Hanushek et al. 2012). Finally, another study argues that SBM programs need to be in place for at least 5 years to achieve significant results and at least 8 years to “transform” schools (Borman et al. 2003). 10 The NBI index is designed by DANE using 2005 census data and is based on a series of simple indicators, including dwelling conditions, lack of services, economic insecurity, and school attendance for children. Data is available at the municipal level (as the fraction of households falling under a chosen score). 11 Table 1: Gross and net enrollment rates in upper secondary in Antioquia’s uncertified municipalitiesi and at the national level 2007 2013 Indicator Baseline Actual Antioquia Uncertified Municipalitiesii Gross enrollment rate 56.4% 68.3% Gross enrollment 56,894 67,887 Net enrollment rate 30.1% 37.3% Net enrollment 30,352 37,115 Population of the age group 100,792 99,377 Antioquia 25 Poorest Municipalitiesiii Gross enrollment rate 46.9% 52.3% Gross enrollment 12,529 14,676 Net enrollment rate 21.9% 26.5% Net enrollment 5,837 7,426 Population of the age group 26,691 28,069 Antioquia 20 Poorest Municipalities (according to NBI)iv Gross enrollment rate 38.4% 51.4% (2012) Gross enrollment 6,707 9,856 Population of the age group 17,421 19,167 National levelv Gross enrollment rate 68.9% 78.2% Gap between Antioquia uncertified municipalities and national level 12.5 9.9 Net enrollment rate 36.0% 41.3% Gap between Antioquia uncertified municipalities and national level 5.9 4.0 i Source: Population figures are based on the 2005 census and are likely exaggerated since they have not been adjusted for high migration or other effects of violence. This means that enrollment rates could also be understated due to the denominator being smaller in reality. Enrollment figures are based on SIMAT data, and on MEN statistics. NBI data is from DANE. ii iii PAD indicators. iv v Additional evidence brought in. National level includes Antioquia, unless otherwise noted. 39. Both the gross and net enrollment rates in the uncertified municipalities also improved relative to the certified municipalities in Antioquia. The gap in the gross enrollment rate decreased from 26.9 percentage points in 2007 to 23.3 in 2013, whereas the net enrollment rate gap dropped from 12.5 percentage points to 11.2 in the same time period (Table 2). Table 2: Comparison of gross and net enrollment rates in upper secondary between Antioquia’s uncertified municipalities and its certified municipalities Aggregate Enrollment Ratesi 2007 2013 Baseline Actual Gross enrollment rate -Antioquia Certified 83.3% 91.6% Gross enrollment rate – Antioquia Uncertified 56.4% 68.3% Gap between Antioquia Uncertified municipalities and Certified municip. 26.9 23.3 Net enrollment rate - Antioquia Certified 42.6% 48.5% Net enrollment rate – Antioquia Uncertified 30.1% 37.3% Gap between Antioquia Uncertified municipalities and Certified municip. 12.5 11.2 iSource: SIMAT and Ministry of Education statistics. Data from certified municipalities was brought in to make analysis easier. 12 41. The Project contributed to increased equitable access in public schools by providing direct subsidies to upper secondary students from low income families in all 117 uncertified municipalities. On average 3,897 grade 10 and 11 students received direct subsidies annually over the period 2009-14. The number of beneficiaries varied every year because the subsidies were awarded to students from families in SISBEN level 1 that did not already benefit from the national funded program Familias en Acción and also to students from families in SISBEN level 2. The implementation of School Improvement Plans also contributed to the attractiveness of public upper secondary education in Antioquia through: a school more integrated in the community, an improved school environment and better trained and motivated school principals and teachers. Finally, the flexible pedagogical models provided under the Project by private organizations in the remote subregions of Bajo Cauca and Urabá also contributed to increased access to upper secondary education. Increase equitable retention in upper secondary education Pre-restructuring rating: Modest Post-restructuring rating: Modest 42. Retention in Antioquia’s 25 poorest municipalities decreased. The upper secondary retention rate in these municipalities decreased from 78.5% in 2007 to 72.2% in 2013, significantly missing the pre-restructuring target of an increase to 90% and the post-restructuring target of 85% (Table 3). Since the indicators in the Results Matrix were inadequate to measure increases in equitable retention in all 117 municipalities supported by the Project, the ICR introduces additional evidence of Project outcomes in these municipalities. The upper secondary retention rate in Antioquia’s 117 uncertified municipalities also declined, from 81.4% in 2007 to 72.5% in 2013. Most of the decline occurred in grade 10 retention (the grade 10 retention rate in Antioquia’s 117 uncertified municipalities was almost 15 percentage points lower than the grade 11 rate in 2013), likely due to the 2010 introduction of the stricter evaluation and promotion rules. 43. However, there is some evidence of Project impact since the uncertified municipalities in Antioquia performed better in retaining students than the certified municipalities, which saw an even greater decrease. Without the Project, the retention rate in the uncertified municipalities likely would have declined even more. To further account for the changes in the evaluation and promotion rules, which affected the whole country, the ICR analyzes data on the percentage of students enrolled in grade 11 as a percentage of students enrolled in grade 10 the previous year. Since this data is available for both certified and uncertified municipalities (but the retention rate in certified municipalities is not), it allows us to compare retention trends throughout Antioquia, and therefore attempt to control for external shocks that affected all schools in the area, both targeted and non-targeted. Using this methodology, the uncertified municipalities compare favorably to the certified ones, according to the latest figures. The gap disappeared during Project implementation, down from 2.9 percentage points in 2007. 13 44. Improvements in the graduation rate provide further evidence of Project impact. The graduation rate in Antioquia’s uncertified municipalities (defined as the number of students who graduated from upper secondary education as a percentage of the relevant total school-age population) increased significantly during the implementation of the Project from 25.4% in 2007 to 30.8% in 2013. This is due to the increase in coverage in these municipalities during the same period, which more than offset the decline in retention. Table 3: Retention and graduation in upper secondary education in Antioquiai Indicator Baseline value Actual value 2007 2013 Retention rate of both grades 10 and 11 25 poorest municipalitiesii 78.5% 72.2% Uncertified municipalities iii 81.4% 72.5% iv Retention rate by year Grade 10 - 25 poorest municipalities 84.4% 79.3% Grade 10 - Uncertified municipalities 86.2% 78.1% Grade 11 - 25 poorest municipalities 92.6% 91.1% Grade 11 - Uncertified municipalities 94.5% 92.8% Students in grade 11 as a fraction of students enrolled in grade 10 the previous year v Uncertified municipalities 85.6% 82.3% Certified municipalities 88.5% 82.3% Gap 2.9 0 Upper secondary graduation rate – Uncertified municipalities vi Number of students who completed grade 10 18,417 20,096 Number of students who completed grade 11 15,966 19,347 Population of the age group / 2 13,583 14,035 Upper secondary graduation rate 25.4% 30.8% i Source: Project Monitoring System (based on SIMAT). ii 2012 Restructuring indicator. iii iv v vi Additional evidence. 45. The Project contributed to softening the decline in the retention rate and improving graduation rates in Antioquia’s uncertified municipalities in the same way that it contributed to the impact on enrollment: through direct subsidies to low income students, an improved school environment through the School Improvement Plans and support to the flexible pedagogical models in Bajo Cauca and Urabá. Improve equitable educational achievement in upper-secondary education Rating: Modest 46. There is limited evidence of improvements in SABER 11 test scores in Antioquia’s 25 poorest municipalities. Average SABER 11 scores in public schools in these municipalities increased from 41.28 in 2009 to 41.4 in 2014 in math, but decreased from 44.28 to 44.2 in language though the declines could be due to the change in test methodology rather than to lower achievement (Table 4). However, there is strong evidence of an improvement in the performance 14 ratings of schools in these municipalities based on SABER 11 scores, as evidenced by the large drop in the number of schools rated low or below (9.7 percentage points in the 2009-2013 period) and the mirror increase in the percentage of schools rated medium or above. 47. Since the indicators in the Results Matrix were inadequate to measure increases in equitable educational achievement in all 117 municipalities supported by the Project, the ICR introduces additional evidence of Project outcomes in these municipalities. Equitable achievement improved modestly in language as evidenced by the narrowing of the SABER 11 gap between Antioquia’s uncertified municipalities and the national average. Math scores did not improve in Antioquia’s uncertified municipalities relative to the national average. In the 117 uncertified municipalities, performance ratings of schools based on SABER 11 scores improved. There was an almost 11 percentage point drop in the number of schools rated low or below in 2013 than there was in 2009, and the same increase for those rated medium or above. Certified municipalities improved more during the same time period, with the fraction of schools rated medium or above rising 22.8 percentage points. 48. The improvement in the percent of schools in the 117 municipalities that were rated medium or above is consistent with an impact evaluation carried out in 2013 to measure the impact on learning achievements of the beneficiaries of the direct subsidies program. The study, which used a differences-in-differences approach to analyze the effect of subsidies on SABER 11 scores at the municipality level, concluded that the program had a positive impact on language, English and Physics scores while it had no impact on Math and Chemistry.11 49. The Project also contributed to improved equitable educational achievement by financing School Improvement Plans (SIPs) in all 451 schools of the 117 uncertified municipalities of Antioquia. The SIPs, designed in 2009 and under implementation starting in 2010 with all schools implementing a SIP by 2011, supported: (i) the use of test results at the school level to provide incentives for better performance; (ii) process in which all stakeholders (at central, local and school level) participated and interacted to make the school more effective; (iii) the training of all school principals that improved their leadership and management of the change process; (iii) the improvement of the learning environment with equipment, laboratories, ICT and libraries; (iv) a program that favored peaceful coexistence and democratic participation to increase collaborative thinking among staff and students; (v) in 2013 the training of 2,593 teachers in the competency based curriculum and teaching practices and of all teachers in the core disciplines (language and mathematics) complemented by class room observations and support from the pedagogic advisors; and (vi) school based management and more specifically school accountability. 11Evaluación del Proyecto de Mejoramiento de la Educación Media en Antioquia y Selección de Beneficiarios de Inventivos . Econometría Consultores (December 2013). 15 Table 4: Results of the SABER 11 test in Language and Mathematicsi 2009 2013 2009 2013 Language Math Poorest 25 municipalities – 44.28 44.2 41.28 41.4 Public schoolsii Antioquia’s uncertified 45.83 46.17 45.07 43.10 municipalitiesiii National leveliv 47.01 47.23 45.31 45.00 Gap between national and 2.73 poorest 25 (Public) 3.03 4.03 3.60 2.63 2.73 4.83 2.90 (Private) Gap between national and 1.18 1.06 0.24 1.90 uncertified Performance ratings of schools in SABER 11 testsv 2009 2013 Change Poorest 25 municipalities – % of 74% 64.3% 9.7 schools rated low or below Poorest 25 municipalities – % of 26% 35.7% -9.7 schools rated medium or above Uncertified municipalities – % of 56.7% 45.9% 10.8 schools rated low or below Uncertified municipalities – % of 43.3% 54.1% -10.8 schools rated medium or above Certified municipalities – % of schools 37.2% 14.4% 22.8 rated low or below Certified municipalities – % of schools 62.8% 85.6% -22.8 rated medium or above i Source: Project Monitoring System (based on ICFES data) and ICFES statistics. ii 2012 Restructuring indicator. iii iv v Additional evidence. 50. The development and implementation of a competency based curriculum, which put greater emphasis on skills such as communication, autonomy, teamwork, ICT, responsibility, and citizenship, in all 451 schools in the 117 municipalities. Finally, in 2013 the Project supported training of 16,398 students enrolled in grades 10 and 11 in 20 of the 25 poorest uncertified municipalities through a program to strengthen competencies in Language and Mathematics. Increase the number and proportion of public school secondary-education graduates who continue on to tertiary education Pre-restructuring rating: Substantial Post-restructuring rating: Substantial 51. The transition rate increased from 13% in 2007 to 19% in 2013, which exceeded the original 18% target. However, this indicator had limitations, since it only took into account students entering higher education institutions in the department of Antioquia. The transition rate 16 using the post-restructuring methodology increased from 23% in 2007 to 31% in 2010.12 The revised methodology was the following: the number of first-year students from uncertified municipalities enrolled in any higher education institution in Colombia who were enrolled in grade 11 the previous year (thus removing upper secondary graduates who took longer than a year to enroll in a higher education institution) divided over the number of students enrolled in grade 11 in uncertified municipalities the previous year. No data using this methodology is available for 2013 or 2014. This indicator, however, still understates transition. Table 5: Transition rate to tertiary education of students from Antioquia’s uncertified municipalities. Baseline Original Revised Indicator Actual value 2013 value 2007 Target 2014 Target Original Indicator: Transition rate to tertiary education in the uncertified 13% 18% 19% municipalities of Antioquia Revised Indicator: Transition rate to 29% (number tertiary education in Antioquia 23% 31% provided for 2010) New indicator: First year students as a 48.9% 54.5% proportion of upper secondary graduates NA NA (2008) (2012) in the uncertified municipalities New indicator: Number of first-year 10,552 students from uncertified municipalities 8,700 (2012) NA NA enrolled in tertiary education (2008) Growth rate: 21.3% New indicator: Number of first-year 30,223 students from certified municipalities 27,997 (2012) NA NA enrolled in tertiary education (2008) Growth rate: 8% 52. New evidence tracking individual students and brought into the ICR to ensure time- comparability shows that the proportion of upper secondary graduates enrolling in higher education institutions (mentioned in the PDO of the Project but not adequately monitored) increased by 8 percentage points from 48.9% in 2008 to 54.5% in 2012 for the uncertified municipalities. The number of first year students from uncertified municipalities enrolled in higher education institutions increased from 8,700 in 2008 to 10,552 in 2012, a growth rate (21.3%) well over that experienced by students from the certified municipalities (8%). 53. The Project contributed to the improved transition rate by supporting workshops and support services for upper secondary students in 2010 and 2011, including on financing alternatives, higher education programs and life skills. It also supported the creation of a Fund in 2012 to provide incentives for students to continue to tertiary education. The Fund (75,597,887.906 Colombian Pesos) is financed by the Department of Antioquia (63%), with additional funding provided by the City of Medellín (18.5%) and the EPM Foundation (18.5%). 12These numbers were established through a study done in 2011. The methodology used to calculate the transition rate post- restructuring was the following: Number of first year students enrolled in a higher education institution in year n divided by the number of students who enrolled in grade 11 in year n-1. 17 It grants full scholarships for tuition and conditional support for the subsistence of students (food, transportation and/or housing). It provided 568 scholarships in 2012, 4,326 in 2013, and 6,181 in the first semester of 2014 to low income students (strata 1, 2, and 3) from the uncertified and certified municipalities enrolled in disciplines relevant for the economy. Of the scholarships granted in 2013 and 2014, 21% went to students belonging to stratum 3, 47% to stratum 2 students and 32% to stratum 1 students.13 The distribution by institution shows that the majority (67.5%) of beneficiaries were concentrated in 3 institutions: Universidad de Antioquia, Politécnico Colombiano, and Tecnológico de Antioquia. Finally, increased enrollment and graduation rates, combined with improved SABER 11 test scores, increased the pool of students from Antioquia that met university entrance requirements. 3.3 Efficiency Rating: Substantial 54. The economic analysis carried out at Appraisal left out several items that are now considered key and are routinely included in economic analyses in human development projects: i) public health and education externalities due to increased educational attainment, and ii) higher tax revenue for the Government due to increased incomes. To correct for this, the present analysis uses a revised methodology that takes into account and quantifies these factors. The reference point is, thus, the cost-benefit analysis and internal rate of return calculated using the original targets as expressed in the PAD, but under the new methodology. The actual observed cost-benefit analysis uses observed data collected through Project monitoring systems and Government databases. 55. The cost-benefit analysis performed using projected targets provides a sound basis to estimate the actual economic results of the Project. The Project yielded a Net Present Value (NPV) of US$67.7 million (COP172.4 billion), compared to the forecasted US$76.9 million (COP195.9 billion). The actual internal rate of return (IRR) stands at 11.9%, slightly below the projected IRR, which was 13.3%. The NPV was hampered by the modest improvements in retention, but this was more than offset by higher than expected results in transition to higher education. The actual IRR and NPV also compare favorably to those calculated at appraisal (10.8% and US$11.5 million, respectively). 3.4 Justification of Overall Outcome Rating 56. The overall outcomes use a split rating to comply with ICR guidelines. The rating is Moderately Unsatisfactory (MU) both pre- and post-restructuring. The rating is calculated on a 6-point scale from 1 (Highly Unsatisfactory) up to 6 (Highly Satisfactory). 13 There are six strata (1 being the poorest and 6 the richest). 18 Pre- Post- Overall restructuring restructuring Relevance of objectives Substantial Design Substantial Substantial Overall Relevance Substantial Substantial Increasing equitable access in upper-secondary Substantial education Increasing equitable retention in upper secondary Modest Modest education Improving equitable educational achievement in Modest upper-secondary education Increasing the proportion of public school Substantial Substantial secondary-education who continues on tertiary education Overall Efficacy Modest Modest Efficiency Substantial Overall Outcome rating MU MU Rating value 3.0 3.0 Weight (by percent of disbursements) 80% 20% Weighted value 2.4 0.6 3.0 Final rating MU MU MU 3.5 Overarching Themes, Other Outcomes and Impacts Poverty Impacts 57. More years of schooling, as the result of direct subsidies and conditional scholarships, and improved learning outcomes could have a lasting poverty impact. The direct subsidies supported under the Project benefited upper secondary students from families registered in the lowest levels of SISBEN. The conditional scholarships for higher education benefitted university students from families in lower socioeconomic strata (which were originally created to distinguish which household would get subsidized access to public services and utilities but are also used for eligibility for other government benefits). Gender Aspects 58. No specific activities related to gender were implemented. Social Development 59. The Project response to the issues of violence and criminal activities was substantial. All the activities that promote a school culture of peaceful coexistence and prevention of violence are now mainstreamed in the education policy of the Department of Antioquia. (a) Institutional Change/Strengthening (b) Other Unintended Outcomes and Impacts (positive or negative) 19 60. In preparing and implementing the SIP through a participatory process involving all the local stakeholders, the Project had a significant positive impact in the partnership between the municipalities and the schools. 61. There was strong support from the Universidad Católica del Norte and the SAT (Sistema de Aprendizaje Tutorial) to provide the virtual platform for distance education to train teachers and school principals. This virtual platform can now be used for large scale in-service training of the teaching staff. 62. EDATEL was used for the payment of the direct subsidies to upper secondary students. This was particularly difficult for students located in remote areas and for such a small amount (US$175 a year in monthly installments). EDATEL connected all the 451 schools participating in the Project. 4. Assessment of Risk to Development Outcome Rating: Modest 63. The risks that the outcomes achieved under the Project might not be sustained are modest. The Government of Antioquia continues to be committed through the Antioquia Education Development Plan 2012-15 to improve access and quality of upper secondary education and the transition rate to tertiary education. Gains in enrollment are likely to continue through Antioquia’s continued support to flexible pedagogical models in rural remote areas. Antioquia plans to continue the use of SIPs to provide resources to improve school effectiveness, including learning outcomes. To continue to achieve increases in the transition rate to tertiary education, Antioquia plans to expand the conditional scholarship program to include 9,000 beneficiaries in 2015 and continue counseling and guidance. There is some risk that enrollment could taper off since the provision of the type of direct subsidies to upper secondary students that were supported under the Project are being discontinued. Risks to increased achievement include Antioquia’s plan to integrate into policy, but scale down, teacher and school principal training. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 64. The Project benefited from a sound design that capitalized on the lessons learned from previous Bank assistance to Antioquia. Innovative approaches were adopted to increase local participation, to promote school based management, and to increase students’ competencies. The direct subsidies were well targeted to the needs of low income families in Antioquia’s 117 uncertified municipalities. The Bank engaged in a fruitful dialogue with the national government to ensure that the Project complemented the national development plan and with the government of the Department of Antioquia to integrate the Project into the departmental priorities. A series of consultations were carried out by SEDUCA between November and January 2007 with 20 stakeholders, parents and the communities to improve Project design and ensure ownership. This process was enriched by an international seminar where the experiences of Chile, Brazil, South Korea and Spain were discussed. Bank performance had some significant shortcomings: (i) Project monitoring was based on key indicators impacting only on a small subset of the 117 municipalities resulting in insufficient monitoring data to inform Project implementation; and ii) the methodology for calculating baseline and target values of many indicators was unclear, further exacerbating the usefulness of Project monitoring. (b) Quality of Supervision Rating: Moderately Unsatisfactory 65. The Project launch was successful and followed by an intensive program of workshops (23 during the four months following the launch) in the 9 regions of the department to provide technical support and practical instruments to the schools. The Bank carried out a minimum of two full supervision missions per year, well-staffed with highly qualified education and implementation specialists and provided additional technical assistance in various areas. The Bank encouraged additional studies to supplement monitoring and evaluation through the Results Matrix. The Bank reacted promptly to potential issues that could affect implementation. That was the case: (i) in 2009, when the Bank recommended that all the NGOs and private providers be screened for possible affiliation with illegal groups (risk of money laundering and other); and (ii) in 2012, when the new administration took office, the Bank was successful in developing a dialogue that could ensure the continuity of the policy supported by the Project. Supervision had the following significant shortcomings: (i) monitoring did not reflect the Project impact accurately since it reflected impact only on a subset of municipalities supported by the Project; and (ii) the indicators, targets and the methodology to calculate them should have been revised/clarified during the restructuring. In addition, the Bank could have proposed the restructuring much earlier. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory 66. Since Quality at Entry is moderately unsatisfactory and Quality of Supervision is moderately unsatisfactory, overall Bank performance is considered moderately unsatisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 67. The Project benefited from the constant support of the authorities at the national and departmental levels. Even with two changes in Administration, the Project remained at the center of the strategy to improve secondary education. The Government was open at the national and 21 departmental levels to innovative activities and to participate in international assessments – i.e. Antioquia volunteered to take part in the Better Teachers regional study in LAC, which included classroom observations in 83 schools out of 452 and in the OECD assessment of tertiary education – to learn from experience. However, there were moderate shortcomings; after the 2012 election it took about eight months for the new Government to resume implementation of the Project. (b) Implementing Agency Performance Rating: Moderately Satisfactory 68. SEDUCA exhibited strong ownership of the Project and commitment to the achievement of its objectives. The Project was staffed with qualified professionals. It was committed, proactive and responsive to challenges. The issues, mostly in financial and procurement management, were addressed in a timely manner. However, there were moderate shortcomings: (i) the data for the monitoring and evaluation indicators was not always provided in time for supervision reports; and (ii) the design of the impact evaluation was delayed and only completed after Project closing. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 69. Overall Borrower performance is considered moderately satisfactory given the moderately satisfactory performance of the Government and the moderately satisfactory performance of the implementing agency. 6. Lessons Learned 70. It is essential that Project Results Frameworks include indicators to measure the expected impact of all key Project activities and not just a subset of activities. To ensure consistent measurement, and ensure clarity on what projects are trying to achieve, it is also important to spell out the methodology for calculating baselines and targets as well as the data sources. 71. SIPs and School Improvement Projects prepared through a participatory approach can be powerful instruments of change: (i) promoting school based management and accountability; (ii) stimulating creativity for better teaching and learning practices; (iii) involving the school community and key stakeholders at the municipal level; (iv) combining all the resources allocated to achieve a greater impact on school transformation; and (v) improving the school climate, positively affecting the attitudes of teachers and students particularly with regards to peaceful coexistence and democratic participation. 72. For SIPs and School Improvement Projects to be developed and implemented successfully and effectively, they require: (i) structured and systematic technical assistance with highly qualified local advisors; (ii) training of school principals and teachers to promote strong leadership at the school level and team building; (iii) significant number of activities aimed at exchanging experiences and best practices; and (iv) connectivity and ICT to facilitate distance education and management and monitoring. The peaceful coexistence and prevention of violence 22 projects supported under some SIPs financed by the Project were an appropriate and effective tool for a conflict environment. 73. Improvements in school based management and competency based curriculum do not translate to measurable improvements in academic results within a typical Project timeframe of 4-6 years. Expected outcomes should be tailored to what can be reasonably expected in the timeframe of the Project. 74. A decentralized and participative process to elaborate and finance school subprojects is compatible with a centralized management of procurement. Even when financial transfers are allocated to schools it can be more cost efficient to consolidate the procurement of some of the goods that are common to the schools such as computer equipment and teacher training. 75. School profiles and the use of performance indicators at school level are important tools to devolve responsibility to local stakeholders, to increase accountability and to manage by results. 76. The management of projects by regular staff is not sufficient to mitigate political risk. In countries where regular staff change during each election, a transitional period is recommended, to allow time for the team from the previous administration to overlap with the newcomers in order to ensure continuity of project management. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies The Borrower had the following additional comments based on the draft ICR: In accordance with the suggestions of the draft ICR sent for comments, the impact evaluation of the Project will take into account the following indicators: o Gap in SABER 11 results between uncertified municipalities and the national level. o Gap between the gross enrollment rates of uncertified and certified municipalities, and the national level. o Intermediate indicators that tracked the progress of implementation, such as: i) percentage of students benefitting from scholarships, ii) percentage of year 10 and 11 students that benefitted from professional counseling and were satisfied with the results, iii) principals and coordinators that use modern administrative skills to carry out their jobs, iv) teachers who have benefitted from Project activities have improved their pedagogical practices, and v) percentage of schools that have achieved at least 75% of the objectives of their school projects. Additionally, although the preliminary impact evaluation does not include data from all 117 uncertified municipalities, it does include data from municipalities in all three levels of income (poor, medium, and rich). 23 Regarding the impact of the subprojects, more research is currently being carried out, focusing on the benefits derived by the schools, and whether they are sustainable or not. Lastly, a preliminary impact evaluation was carried out during the fall of 2014 in the eastern region of Antioquia; it provides additional evidence on Project impact by component, according to the beneficiaries’ perception. (b) Cofinanciers N.A. (c) Other partners and stakeholders (e.g. NGOs/private sector/civil society) N.A. 24 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions) Component 1: Equitable Access and Retention to upper-secondary and 13.50 14.14 104% opportunities for transiting to tertiary education Component 2: Quality and Relevance of Upper-Secondary 21.20 20.82 98% Education Component 3: Institutional Strengthening, Monitoring and 7.20 6.82 95% Evaluation, and Project Management Total Baseline Cost 40.83 41.78 102% Physical Contingencies 0.28 0.00 Price Contingencies 0.79 0.00 Total Project Costs 41.89 41.78 99% Front-end fee PPF 0.00 0.00 Front-end fee IBRD 0.00 0.00 Total Financing Required 42.00 41.78 99% (b) Financing Appraisal Actual/Latest Type of Percentage of Source of Funds Estimate Estimate Cofinancing Appraisal (USD millions) (USD millions) Borrower 22.00 22.00 100.00 International Bank for Reconstruction 20.00 19.88 99.38 and Development 25 Annex 2. Outputs by Component Component 1: Equitable Access and Retention to upper-secondary and opportunities for transiting to tertiary education Sub-component 1.1. Implementation of flexible pedagogical models 1. In 2013 the following activities were implemented: (i) school visits to present the flexible models in 20 schools in 12 municipalities in Bajo Cauca and Urabá with the greatest percentage of out-of-school youths and over aged students; (ii) the training of 218 teachers and 10 principals for the duration of 3 days; and (iii) technical assistance to implement the flexible models, including to reconfigure the curriculum. SEDUCA contracted private providers or Jóvenes con Futuro, a technical vocational program, to offer flexible models. This was complemented by a strategy to help municipalities to identify out-of-school youths and put them in contact with schools. 2. Under the Project, the enrollment of students in flexible models in secondary education increased from 1,032 in 2009 to 2,413 students in 2014. Enrollment in indigenous education remained the same (105 students in 2009 to 106 in 2014). Flexible models 2009 2014 Post primary 32 Telesecondary 15 Indigenous education 105 106 SAT 5,234 Rural secondary 1,032 2,413 Total 6,418 2,519 Sub-component 1.2. Direct subsidies to upper secondary-education students 3. On average, 3,897 students received direct subsidies annually over the period 2009-14. The beneficiaries of the subsidies were students in grades 10 and 11 from families in SISBEN 1 and 2, who did not already receive subsidies from Familias en Acción and were less than 19 years old (26 for the indigenous population). A priority was given to students whose families were victims of conflict and to the indigenous population. The subsidy was about $175 per year. 335 upper secondary schools participated in the program. SEDUCA contracted EDATEL to manage the payment of the subsidies to the students. 2009 2010 2011 2012 2013 2014 Number of students 63,841 66,481 66,950 67,484 68,601 67,887 in grades 10 and 11 Number of direct subsidies financed 5,142 4,496 3,733 1,587 4,868 3,554 by the Project 26 Sub-component 1.3. Conditional Scholarship to increase access to tertiary education 4. Until 2011 and part of 2012, scholarships for students to access higher education were provided through the fund Gilberto Echeverri (COP 6,000,000,000) but not financed by the Project. A new scholarship fund was established and managed by IDEA in 2012, to grant full scholarships for tuition and conditional support for the subsistence of students (food, transportation and/or housing) to allow more young people from Antioquia to access higher education in disciplines relevant for the economy. The Fund (COP 75,597,887,906) was financed by the City of Medellín (18.5%), the EPM Foundation (18.5%), and the Department of Antioquia (63%). 5. In total, 6,085 scholarships were granted in 2013 and 2014. The distribution of beneficiaries by socio-economic origin was the following: 21% of the students belonged to stratus 3, 47% to stratus 2 and 32% to stratus 1. 2013 semester 1 2013 semester 2 2014 semester 1 2014 semester 2 Registered students 2,077 7,120 7,860 5,576 Pre-selected 1,406 3,083 2,675 1,917 students Beneficiaries of 568 2,160 1,857 1,500 scholarship 6. The distribution by institution shows that the majority (67.5%) of beneficiaries were concentrated in 3 institutions: Universidad de Antioquia, Politécnico Colombiano, and Tecnológico de Antioquia. Institution of Higher Number of Education beneficiaries of scholarships Universidad de Antioquia 1,837 Politécnico Colombiano 1,236 Técnico de Antioquia 1,029 Universidad Católica de Oriente 488 Escuela Normal Superior 299 Fundación Universitaria Católica 345 del Norte Tecoc 46 Tecnológico Pascal Bravo 219 Universidad Nacional de 113 Colombia COREDI 168 ITM 79 SENA 156 Colegio Mayor de Antioquia 8 27 7. The distribution by sectors was the following: Sectors Percentage of the beneficiaries of scholarships Sciences and 30.0% Engineering Agricultural 22.8% Education 19.1% Accounting, 16.4% Administration, Economy Law, Social Sciences 5.7% and Humanities Health 3.5% Art, Communication, 2.4% Journalism Sub-component 1.4. Vocational and professional counseling and guidance 8. In 2011, two events (Fábricas del Futuro) were organized in La Ceja and Santa Rosa to facilitate vocational and professional counseling and guidance of students. 3,500 students participated in these events. Students had the opportunity to meet staff of higher education and vocational training institutions, employers and organizations that specialize in youth guidance and counseling. 9. In 2012-13, the main activity of this subcomponent was the organization of incubators for entrepreneurship (Semilleros de Emprendimiento). These indicators consisted of giving students of grades 9, 10 and 11 50 hours of training organized in six sessions delivered during two months to help them develop creative ideas for their future professional life. The training provided students with competencies in creative thinking, analysis of the economic potential of their region and opportunities, team work, conflict resolution, decision making and communication. It was organized in 189 schools in 62 municipalities (of which 17% were in rural areas) in which 9,982 students participated in 2012 and 11,103 in 2013. The training included a competition for creative ideas (3,131 were elaborated) with monitoring of student projects and job counseling. 10. In addition, to facilitate the intermediation between students and employers and inform students on job opportunities ten Feria del Bachiller events were organized in which about 78.6% of the students in grades 10 and 11 in the uncertified municipalities under the Project participated. 11. Finally secondary schools tracked their graduates. A methodology was established and distributed to all the schools in which the school principal or the person delegated to this task should call the parents of the student 6 months after his/her graduation to determine his/her status. The information is reported in the Alphasig system and consolidated in SEDUCA. It is estimated that 70% of the schools did the tracking of their graduates (this percentage was 17% in 2012). 28 Component 2: Quality and Relevance of Upper-Secondary Education Sub-component 2.1. Strengthening Upper-Secondary School Management for Education Quality through school grants School Improvement Plans (SIP) 12. The sub-component supported in the uncertified municipalities: (i) the training of teachers in areas such as the design of a competency-based curricula, strategies to incorporate the competency-based curricula into the teacher training colleges, and strategies to design and implement incentives for best practices at the school level. 13. At the school level, the sub-component provided financial transfers for SIPs directed to address activities related to: (i) the competency-based curricula; (ii) pedagogical practices; (iii) effective use of test results to manage quality; (iv) learning environments; and (v) education for peaceful coexistence, among others. The provision of school grants was an instrument to help upper secondary students complete their education and achieve higher test results to allow them to gain access to tertiary education institutions or eventually enter the labor force. 14. The steps to prepare and implement the SIPs were the following: 2008: Diagnostic done by Corpoeducación of each of the 451 schools. 2009: Establishment of school profiles for each of the schools and design/preparation of subprojects. 2010: Implementation of the SIP (in June 2010, 50% of the schools had initiated the implementation of the SIP). 2011: All 451 schools had their SIP under implementation. 15. The School Improvement Plans had seven lines of action and financial transfers Activities of the SIP Financing (in Percentage million of COP) Vocational and 283 1.9% Professional counseling and guidance Design and 307 2.1% implementation of a competency based curriculum Teacher training 355 2.4% Improvement of learning 6, 968 47.8% environments Pedagogic resources 5,288 36.3% including TIC Education for peaceful 1,242 8.5% coexistence School based management 140 1.0% Total 14,581 100.0% 29 16. Of the total financing of the SIPs: (i) 35.1% was allocated to the 25 poorest municipalities with an average cost per school of COP 86.7 million (ii) 30.2% to middle income municipalities with an average cost per school of COP 78.3 million; and (iii) 34.7% to rich municipalities with an average cost per school of COP 85.7 million. The level of execution in all municipalities was over 98.8%. Implementation of competency based curricula: Curriculum assessment 17. First, a study of the current status of the curricula in the schools was done, which included an analysis of the pedagogic model, the internal evaluation system, the use of pedagogic resources and the links of institution with the community and the productive sector. The study showed the need to strengthen the curriculum, to revise teaching and learning practices, and to use a competency-based curriculum. The final product was "A proposal for competency based curriculum for secondary education in the Department of Antioquia," which provided the principals and teachers with a clear structure to organize the key areas, to introduce elective subjects related to the schools context and local needs, to develop initiatives related to citizenship and vocational skills and to include cultural and sports activities. Curriculum design 18. For the design of the competency based curriculum six activities were proposed: (i) organization of quality teams in schools; (ii) socio-cultural and educational diagnosis; (iii) orientations for education; (iv) definition of achievements and competencies; (v) specific cultural context and relation with the curriculum standards issued at the national level; and (vi) curriculum micro design. The development of the curriculum was not a linear process, but the result of ongoing reflection taking into account the progress made in teaching practices in which students and the school community stakeholders were involved. The curriculum was not confined to the academic and work competencies, but went much further to determine how educational goals respect cultural identity, ethnic differences, citizenship, defense and promotion of human rights and the protection of the environment. Teacher training and student training 19. Training to implement the competency-based curriculum included: (i) tools that principals and teachers used to build a curriculum taking as references the vision and mission of the school, the profile of the person to educate, the principles and values, the pedagogical model, the needs and expectations of students, educational standards, curriculum guidelines and educational projects proposed in the General Education Act; (ii) a tool for competency assessment and the sustainability of the training process at the institutional level with principals and teachers; and (iii) tools for adjusting the SIPs. In 2013, and the first semester of 2014: (i) 2,593 teachers participated in training to implement the competency-based curriculum of which 492 were located in the poorest municipalities; (ii) 16,398 students were trained in a program to strengthen their competencies in Language and Mathematics. 30 Implementation 20. For the implementation of the competency-based curriculum, Corpoeducación: (i) established the education profiles of the schools (including the political , social, ethical, emotional, and vocational dimensions); (ii) formulated the curriculum components including the enhancement of knowledge and the skills/abilities and attitudes necessary for the students to interact in different social and work contexts; (iii) included educational philosophy, learning purposes, conceptual domains, competencies, pedagogical and assessment strategies, and logistical processes in the curriculum; (iv) designed the curriculum with regard to knowledge content, indicators of achievement, teaching practices and evaluation criteria. Training for school principals leadership 21. The training of school principals consisted of 4 modules: (i) one dedicated to staff management of human resources; (ii) two dedicated to learning practices and pedagogical management; and (iii) one dedicated to administrative and financial management. The duration of the training was one week of intensive courses per module. During the weeks between each intensive course, support was provided to the schools to help the principal and 4 staff from his/her team to put learning content into the management practices of the schools. A monitoring and evaluation system was implemented to provide feedback to improve the processes. It reviewed the following: classroom teaching and learning practices, coexistence, staff participation, institutional transformation. The Fundación Empresarios por la Educación y la Universidad (EAFIT Foundation for entrepreneurship in education), which developed the training model for the school principals help guide and the support the transformation process in schools for two years. In 2013, EAFIT brought 52% of the resources of the program and 40% in 2014. 22. In 2013, the program was organized in 126 schools and in 2014 in 119 schools. There was strong support from the Universad Católica del Norte and the SAT (Sistema de Aprendizaje Tutorial) to provide the virtual platform for distance education to train teachers and school principals. Provision of computers and interactive whiteboards to the schools 23. In 2011, the following equipment was provided to 450 schools: (i) 6,900 computers; (ii) 400 interactive whiteboards; (iii) 400 laptop computers; and (iv) access points, software and AC adapters. 24. School training was organized for the use and management of the interactive whiteboard in the classroom. A Help Desk for the management and support of the technology was created to implement the entire process and coordinate requests from schools to suppliers. In 2012, the local pedagogic advisers were trained in using the interactive white boards in order to bridge between the schools and the Help Desk for technology management and support. From November to May 2013, preventive maintenance of the equipment was performed. In 2013 EDATEL was in charge of the Help Desk. 31 Improvement of pedagogical practices 25. Between February to April 2011, training sessions on designing the competency-based curricula were organized by Codesarollo in which 3,014 teachers and school principals participated. The activity followed the following steps: (i) designing three training modules for teachers about managing a classroom by competencies, testing and vocational orientation; (ii) designing the field strategy; (iii) providing the training (48 hours of training, of which 18 were done through the internet) and monitoring the delivery of each module; and (iv) systematizing the experience. 26. The training modules were the following: (i) How I see myself in this project; (ii) Learning from my peers; (iii) What I will do and how I will do it. Classroom observations 27. The Project financed a study of classroom observations: “Evidence on the practices of teachers and the dynamics of the classroom in Latin America and the Caribbean.” The methodology also known as Stallings method was used by the Bank in several countries and reviewed the following: (i) use of time by teachers; (ii) use of instructional materials; (iii) key pedagogical practices; and (iv) ability to keep students engaged. The information was collected and analyzed in 89 educational institutions in grades 5, 9 and 11. School profiles 28. School profiles were designed to provide each of the 451 participating secondary schools feedback about the data generated at the school level with the objective to improve schools’ interaction with SEDUCA. The school profiles were a tool to improve capacity of the school community to make decisions, underpinned by statistics and indicators about their educational work. Institutional Profiles delivered to each educational institution, information related to: (i) description of the institutional structure; (ii) characteristics of enrollment; (iii) performance indicators such as dropout, repetition, overage students, and achievement; (iv) quality of Education (Results of SABER - 5, 9, 11); and (v) benefits of the Project (incentives, investment). Effective use of test results to manage quality 29. Since 2009, ICFES has been carrying out skills assessment in 10th grade. In November 2010, SEDUCA organized Pre-SABER 11 tests financed by the Project, during which 21,251 students and 363 schools participated (85% of the 451 schools that participated in the Project). Improving learning environment 30. As part of the subprojects, equipment was provided to 437 schools such as: laboratories of physics, chemistry, biology, mathematics, and libraries. This was accompanied with training for the teachers to promote the intensive use of this equipment. 32 Education for peaceful coexistence 31. With the support of the Project, activities that promoted a school culture of peaceful coexistence and prevention of violence became a key element that was mainstreamed in the education policy of the Department of Antioquia. Year Actions Results 2008 - Organization on October 2008 of an - A methodology to promote peaceful international workshop to promote the coexistence and to prevent violence school culture of peaceful coexistence (PCPV) was defined and tested. and prevention of violence in the municipality of La Ceja with Bank participation. Six schools participated in the workshop. - Implementation of a pilot in La Ceja. - Team of one coordinator and 8 advisors 2009 - Organization of workshops in various - 56 schools prepared a PCPV to regions with the objective of discussing include in their subproject. an action plan and advising schools to insert a PCPV in their subprojects. 2010 - Organization of workshops in June and - A document “Strategy for November 2010 to help the local coexistence” was prepared describing advisors and people responsible for the issues, the factors of risk, the PCPV locally to present various objectives and possible actions to methodologies (such as Perdón y address the issues. This document Reconciliación, Aula en Paz). was used by the Advisers to help the schools. 2011 - SEDUCA established a specific PCPV - 82% of the schools established a team (8 advisors and one coordinator) to team in charge of PCPV visit each municipalities at least once a - 15 collective encounters were year to launch the activity organized in municipalities of the “Methodological path for a peaceful regions of Norte, Valle de Aburrá, coexistence in the schools.” Occidente, Suroeste and Oriente in - The PCPV team managed the action which the school community plan and prepared tools for its (students, parents teachers) and other evaluation. institutions or stakeholders of the municipalities participated in the areas of health, security, social programs, youth organization, unions, etc. - September 15 institutionalized as the day of peaceful coexistence in the school system. 2012 - The PCPV team was reduced to 5 - The PCPV activities were monitored: advisors (i) first visit to 34 municipalities and 114 schools; and (ii) second visit to 74 schools. 2013 - The PCPV team had 1 coordinator and 7 - The National Law 1620 gave a legal advisors. framework to PCPV. - Strengthening of school and municipal committees of coexistence in schools. 33 - Based on a diagnostic, preparing Action Plans for coexistence, articulated with Institutional Improvement Plans. - Coordinating PCPV with other departmental and municipal actors (Family Commissions, the Colombian Institute of Family Welfare, Police Children and Adolescents, Community Police, Family Courts, NGOs, Local Health, Local Hospitals, Regional Health and Social Protection of Antioquia) with interagency partnerships. - Monitoring and evaluation of institutional action plans, assessing the impact on coexistence and school problems. - Integrating legal culture and citizenship to PCPV. - Expanding opportunities for participation of student and staff representatives in PCPV initiatives. - Teacher Training to coordinate SIPs with Government-Health-MANA ICBF according to the needs and context. 2014 - The PCPV team had 1 coordinator and 8 - 69 municipalities and 298 schools are advisors. implementing PCPV. - Strengthening of school and municipal committees of coexistence in schools. - Based on a diagnostic, preparing Action Plans for coexistence, articulated with Institutional Improvement Plans. - Updating manuals for coexistence. - Mainstreaming the citizenship competencies in curricula, classroom strategies, institution and community. - Alternative mechanisms for conflict resolution. - Tracking, monitoring and evaluation of actions undertaken. Subcomponent 2.2. Improve the Effectiveness of technical upper-secondary education 32. A study to improve the quality and relevance of technical upper secondary education was done in 2013 and completed in 2014. It recommended: (i) to close 18 technical programs out of 65 which were not sustainable and not relevant to the labor market needs; (ii) to upgrade programs in the health, commerce and agricultural sectors; (iii) technical training, when relevant, can be useful to students when it complements a good academic education; and (iv) leadership and better linkage with the productive sectors are necessary to transform technical upper secondary education. 33. No reform for the improvement of upper secondary technical education was formally designed and adopted. Instead, some new orientations were implemented: (i) 9 schools with 34 technical programs were renovated; (ii) in 2012, an agreement was signed with SENA to ensure eight hours of technical training in addition to the 30 hours of academic education (this agreement is in place in 82 municipalities); and (iii) technical training for adults or out-of-school youths is being provided in the Educative Park (Parque Educativo) to be created in 80 municipalities (4 have been recently inaugurated). Component 3. Institutional Strengthening, Monitoring and Evaluation, and Project Management Sub-component 3.1. Building the Department’s Capacity to Provide Technical Assistance to Municipalities and Schools Pedagogical support 34. Provide technical assistance to local advisors groups to: (i) support strategies; instruments and knowledge for pedagogical and school management models; (iii) prepare SIPs; and (iii) use of achievement test results. Year Actions Results 2009 Training in the formulation of One visit to each municipality. investment subprojects Identification of the needs of the schools based • Identification of needs in on their improvement strategies. educational institutions in relation Formulation of some subprojects. to their improvement strategies. • Management and implementation of the action plan • Tools for monitoring and evaluation of the action plan (improvement plan). Team of 12 advisors and one coordinator 2010 Advise, support and provide Participative preparation of subprojects in each technical assistance in the work of schools. planning, implementation, monitoring and evaluation of School Improvement Plans. Training and Counseling organized by sub regions about how to realize subprojects responding to the needs. Team of 10 advisors and one coordinator 2011 Advise, support and provide Setting School Improvement Plans, sharing them. technical assistance in the work of planning, implementation, Restructuring the curricula in the core disciplines monitoring and evaluation of on the base of competencies. School Improvement Plans. 35 Training and Counseling organized by sub regions about how to realize subprojects responding to the needs. Support the training done by Corpoeducación on curriculum design Team of 10 advisors and one coordinator Team of 8 advisors and one coordinator plus trainer from Corpoeducación. 2012 •Reframing the subprojects in Visits to the 451 schools. each of the 451 schools (after 8 Diagnosis of institutional needs according to the months without implementation). lines of investment of the Project. • Integrating the components of the Project in the development plan Antioquia la más educada in its line two "Education as transformation engine" • • Critical reading of the curricula. Team of 11 advisors and a coordinator 2013 Characterization of the schools Diagnosis of the status of IE in relation to the with regard to quality commitments from the covenants for Quality. • Assistance in the creation of quality team and the definition Equipment selected. strategies for improvement. • Workshop with teachers on Operational quality improvement plans designed curriculum. • Workshop about opportunities Curriculum with a focus of competencies defined Olimpiadas del Conocimiento in the 451 schools of the Project. “Covenants on quality and • Assistance in the preparation of Lesson plans set in 40% of the schools of the plans for quality improvement. Project. • Workshops with all teachers or Pedagogical models validated in 10% the schools academic councils for review and of the Project. restructuring of the curricula • Dissemination of knowledge networks in particular in the areas of language and mathematics • Training with Codesarollo in the management of classroom in the context of competencies based curriculum and life project. 2014 The schools with lower quality All the 451 schools of the Project have a indicators are prioritized. curriculum based on competencies and Technical assistance of two or corresponding lessons plans. three days is provided. In other schools technical assistance is provided on demand. 36 Technical assistance include: Teachers have appropriate theoretical and comprehensive classroom methodological tools with which to promote the management; lessons planning for learning of students. the curriculum by competencies approach; group analysis of All Teachers of language and mathematics have SABER Olympic testing; theoretical and methodological elements to participatory analysis actual enrich their classes and adjust their plans area in lesson plans focusing attention on reference to the guidelines. language and mathematics • Training workshops in the areas 30% of schools have renovated pedagogical of mathematics and language models. • Workshops on teaching and learning strategies for the 50% of schools have the inputs to adjust their approach by competencies. Institutional Evaluation systems. •Workshops for the validation of the institutional pedagogical models. 30% of the schools have the inputs for the • Review and Guidance for development of tests like SABER. Institutional Evaluation Systems according to the SEDUCA 50% of the schools have systematized their good guidelines practices. • Workshops systematization of good teaching practices as a strategy for quality improvement. •7 pedagogical advisors in the second and a coordinator. Sub-component 3.2. Impact Evaluation and Monitoring Strategy 35. With regards to improving school based management and monitoring results at the school level, this subcomponent financed: (i) in 2010 the design of the Alphasig software to monitor school performance and Project activities; (ii) software installed in 2011 in all 450 schools and staff training in its use; and (iii) in 2014, 100% of the schools used the software to publish their results and to send their information online to SEDUCA (only 65% in 2012). 36. An impact evaluation to conduct a systematic and objective analysis of the results achieved by the Project was not done. Sub-component 3.3. Project Management 37. The Project management team was placed under the authority of SEDUCA and integrated into the Department of Quality Improvement. Initially it consisted of a Coordinator, two specialists to coordinate the technical assistance, a monitoring and evaluation specialist, a procurement specialist, a financial specialist and an accountant. In addition, 12 local advisors were appointed (9 were selected among the regular staff and 3 were contracted). In 2011, the financial management team was strengthened. It consisted of one coordinator and three financial advisors (three were contracted and one was from SEDUCA). 37 List of Uncertified Municipalities Abejorral Giraldo San Rafael Abriaquí Girardota San Roque Alejandría Gómez Plata San Vicente Amagá Granada Santa Bárbara Amalfi Guadalupe Santa Rosa De Osos Andes Guarne Santafé De Antioquia Angelópolis Guatape Santo Domingo Angostura Heliconia Segovia Anorí Hispania Sonson Anza Ituango Sopetrán Arboletes Jardín Támesis Argelia Jericó Tarazá Armenia La Ceja Tarso Barbosa La Estrella Titiribí Belmira La Pintada Toledo Betania La Unión Uramita Betulia Liborina Urrao Briceño Maceo Valdivia Buriticá Marinilla Valparaíso Cáceres Montebello Vegachí Caicedo Murindó Venecia Caldas Mutatá Vigía Del Fuerte Campamento Nariño Yalí Cañasgordas Nechí Yarumal Caracolí Necoclí Yolombó Caramanta Olaya Yondó Carepa Peñol Zaragoza Carolina Peque Caucasia Pueblorrico Chigorodó Puerto Berrío Cisneros Puerto Nare Ciudad Bolívar Puerto Triunfo Cocorná Remedios Concepción Retiro Concordia Sabanalarga Copacabana Salgar Dabeiba San Andrés Don Matías San Carlos Ebéjico San Francisco El Bagre San Jerónimo El Carmen De Viboral San José De La Montaña El Santuario San Juan De Urabá Entrerrios San Luis Fredonia San Pedro Frontino San Pedro De Uraba 38 Annex 3. Economic and Financial Analysis The purpose of this economic analysis is to compare the actual costs and benefits of the Project to the forecasted costs and benefits at Appraisal. The economic analysis carried out at Appraisal,14 however, left out several items that are now considered key and are routinely included in economic analyses in human development projects: i) public health and education externalities due to increased educational attainment, and ii) higher tax revenue for the Government due to increased incomes. To correct for this, the present analysis uses a revised methodology that takes into account and quantifies these factors. The reference point is, thus, the cost-benefit analysis and internal rate of return calculated using the original targets as expressed in the PAD, but under the new methodology. The actual observed cost-benefit analysis uses observed data collected through Project monitoring systems and Government databases. A. Income Profiles The benefits derived from increased income are estimated based on the earnings profile for each of the educational levels included in the analysis. These profiles were estimated using the assumptions listed below. - The income profiles follow the Mincerian equation estimated by Guerra et al (2009). Lnw  0  rS  0 E  1E 2   Where S represents the educational level, E represents age, and r is the rate of return on education. The estimated earnings profile (following a Mincerian quadratic equation) predicts a growth in income as experience (age) increases, but at a decreasing rate. The coefficients estimated by Guerra et al are 0.0450 for 0 and -0.0008 for 1. This means that the initial yearly growth of individual income is 4.5%, but this rate decreases by 0.008 % times age squared every year.15  The expected working life for an individual is 40 years, which, for the purpose of this study, is the 25 to 65 age range. This is in addition to the length of schooling, which varies depending on the type of tertiary education attained.  Entry salaries were estimated based on the earnings of individuals aged 21 - 24, according to data from the 2012 Quality of Life Survey. Salaries were discounted to take into account the Consumer Price Index increase between 2007 and 2012, in order to calculate the income for the first year of Project implementation. These wages are COP561,626 for technical education, COP601,405 for technological education and COP944,638 for university education. 14 The internal rate of return calculated in the original economic analysis is 10.8%, with a total net present benefit of US$11.5 million. The opportunity cost of capital at the time was 8%. 15 Although the original Mincer equation includes experience as the main explanatory variable, in Guerra et al’s work, age is included instead of experience. This does not represent major methodological inconveniences, considering that, in most of the research on this subject, experience is estimated using the following formula: X = E - 18 - S. There, experience (X), is equal to the age (E) of an individual under 18 years of age, which corresponds to the age when he/she presumably enters higher education, minus the time it takes the individual to graduate from higher education, which is assumed to be 6 years. 39 Based on the above assumptions, the estimated earnings profiles for each level of education are shown below. Upper Secondary Technical Technological University B. Individual Benefits The following equation summarizes the costs and benefits (BI) for an individual of continuing on to tertiary education after graduating from upper secondary: =65 = ∑ ( − ) − ( + + ) = Where,  ( − ) represents the increase in annual income, equal to the difference between the initial earnings associated with each educational level minus the earnings of an upper secondary graduate entering the labor market.  represents the cost of tuition for each tertiary degree.  represents the opportunity cost of tertiary education, which is equivalent to the (upper secondary graduate) income lost during the length of the program. 40  represents living expenses and the cost of school supplies. They were estimated based on Gamboa 2007.16 The following table summarizes the different factors considered when estimating the costs and benefits of each educational level and labor market choice. Average labor earnings Tuition Cost Educational Level Mo. income Technical 644,644 Incomplete Upper Secondary 439,932 Technological 721,244 Complete Upper Secondary 469,658 University 1,719,311 Incomplete Technical 471,614 Living expenses Complete Technical 561,626 Semester 1,900,135 Incomplete Technological 485,119 Year 3,800,270 Complete Technological 601,405 Mincer Equation Incomplete University 516,862 Growth rate 0.0450 Complete University 944,638 Rate of decrease 0.0005 The Internal Rate of Return (IRR) is, ultimately, the interest rate that makes the net present value of future cash flows equal to 0, where the net cash flow (I - C) is equal to income minus costs. The following equation shows how the internal rate of return was estimated. ∑( − ) = =0 (1 + ) C. Social Cost-Benefit Analysis of the Project The following Table summarizes the main benefits and costs inherent to the Project. Benefits Costs Individual benefits due to increased access to upper secondary Individual costs (CI) (BAP) Individual benefits due to increased transition to tertiary Additional costs education (BET) (CA) Benefits for the education sector (BSE) Project costs (CP) Benefits related to tax collection (BIm) Externalities - Education (EE) Externalities - Health (ES) The equation used for estimating the costs and benefits of the Project is shown below. =65 =65 = ∑ ( + + + + + ) − ∑ ( + + ) =25 =25 16 Gamboa, Luis Fernando. (2007). Result of Economy Students Consumption Habits Survey, Bogota 41 1. Benefits There are several benefits derived from Project implementation, both individual and social. Individual benefits are related to the increase in individual income, while social include a number of factors often difficult to measure in monetary terms. a) Increase in access and retention (BAP) The following table summarizes the results of increased access and retention, including the projected and the actual results achieved. 2006 2007 2008 2009 2010 2011 2012 2013 FORECAST Gross enrollment rate 46.0% 54.6% 58.9% 53.8% 57.2% 60.0% 66.0% Net enrollment rate 22.0% 23.9% 25.4% 26.5% 27.7% 30.0% 36.0% School age population Poor municipalities 26,691 27,166 27,518 27,790 27,983 28,093 28,115 28,069 School age population Total (117 municipalities) 100,792 102,227 103,201 103,627 103,393 102,568 101,148 99,377 Net enrollment (according to targets) 5,837 6,577 7,059 7,415 7,782 8,435 10,105 Net coverage forecast - Total 30.1% 32.4% 34.8% 37.1% 39.4% 41.8% 44.1% Total enrollment - Net 30,352 33,485 36,042 38,373 40,460 42,260 43,839 Increase in student spots 3,133 2,556 2,331 2,087 1,800 1,579 ACTUAL Total enrollment 45,479 47,512 47,700 52,442 55,635 58,456 56,821 55,242 Increase in student spots 2,221 4,742 3,193 2,821 -1,635 -1,579 The assumptions made for the calculation are shown below: - The projected data is the result of extrapolating the PAD enrollment rate targets to school age population numbers throughout the Project implementation period. - The benefits correspond to increased labor earnings for 11th grade students. - Enrollment data includes only public schools, for the sake of cost-benefit analysis. - These benefits are derived from the difference in earnings between those who finish upper secondary and those who do not. - Negative values shown in the table are a result of using preliminary Project indicator projections. For the purpose of this analysis, the values were set to zero. - The analysis spans a 40 year time period starting in 2014, the end date of the Project. - The discount rate used is 8% per year, which corresponds to the rate used at Appraisal. b) Transition to Higher Education The following table shows the benefits derived from the increase of transition to higher education. The benefits include the increases in individual income derived from graduating from a higher education program. 42 LB 2007 2008 2009 2010 2011 2012 2013 FORECAST Enrollment 11th Grade Private 2,766 3,461 3,098 4,076 3,646 3,274 3,997 Public 19,796 20,200 21,297 22,889 23,036 23,870 23,672 Transition Rate (calculated using UNESCO definition) 23.0% 25.0% 27.2% 28.0% 29.0% 30.0% 31.0% Increase in spots 4,553 0 5,050 5,793 6,409 6,680 7,161 7,338 New student spots (with respect to baseline) 497 1,240 1,856 2,127 2,608 2,785 Distribution of new spots Technical 0 13 32 49 56 68 73 Technological 0 47 117 174 200 245 262 University 0 437 1,091 1,633 1,872 2,294 2,450 Total new student spots 0 497 1,240 1,856 2,127 2,608 2,785 ACTUAL Transition Rate 38.9% 40.7% 49.8% 41.8% 43.8% 44.4% Increase in spots 8,317 8,792 11,706 10,105 10,399 11,277 New student spots (with respect to baseline) 475 3,389 1,787 2,082 2,959 2,000 Distribution of new spots Technical 0 12 89 47 54 77 52 Technological 0 45 319 168 196 278 188 University 0 418 2,981 1,572 1,832 2,603 1,760 Total new student spots 475 3,389 1,787 2,082 2,959 2,000 The previous table shows higher education enrollment as forecasted according to the original transition rate targets compared to the actual numbers achieved, by level of education. Benefits include, in addition to the information displayed in the table, the difference in expected income between students who graduated from their programs and those who did not. Dropout rates reported by SPADIES for 2010 were used in order to estimate differences between those who graduated and those who do not. Dropout rates are the following: 59.2% for technical degrees; 54.5% for technological degrees and 47.0 for university degrees. These rates correspond to dropout by cohort according to the duration of the programs: 4 semesters for technical, 6 semesters for technological and 10 semesters for university degrees. The benefits in terms of higher earnings are given by the income differential between those who graduate from a higher education program and upper secondary graduates. The analysis includes both graduates (complete) and those who do not graduate (incomplete), applying the corresponding wage scales. As was the case for the enrollment in upper secondary, negative values were assumed to be zero (0) in order to carry out the cost-benefit estimates. c) Reduction of School Dropout The benefits for the public education sector are given by the reduction of school dropout (and its associated costs). The reduction of school dropout is understood as the increase of average retention rates. According to the PAD, the Project’s goal is to raise the rate of completion of upper secondary from 78.5% to 85.0%, which is equivalent to an increase of 6.5 percentage points. This 43 increase implies an increase of 3.6 percentage points in retention, equivalent to an annual average rate of 0.59 percentage points. 2006 2007 2008 2009 2010 2011 2012 2013 FORECAST Total enrollment - net Public 24,262 25,019 28,431 31,888 35,047 35,003 35,673 Private 6,090 8,466 7,611 6,485 5,413 7,257 8,166 Total 30,352 33,485 36,042 38,373 40,460 42,260 43,839 Upper secondary completion rate 78.5% 78.2% 74.0% 76.0% 80.0% 82.0% 85.0% Average rate of retention per grade 89% 88% 86% 87% 89% 91% 92% Increase in retention 0.59% 1.18% 1.78% 2.38% 2.99% 3.60% Increase in retention - student equivalent 148 337 568 835 1,046 1,283 ACTUAL Total enrollment - net Public 45,479 47,512 47,700 52,442 55,635 58,456 56,821 55,242 Private 11,415 12,446 15,410 13,957 11,546 9,037 11,780 12,645 Total 56,894 63,110 66,399 67,181 67,493 68,601 67,887 Upper secondary completion rate 78.2% 78.3% 78.5% 74.9% 67.4% 69.0% 72.2% 74.0% Average rate of retention per grade 88% 89% 87% 82% 83% 85% 86% Increase in retention -0.40% -0.81% -1.21% -1.61% -2.01% -2.40% Increase in retention - student equivalent 0 0 0 0 0 0 The following table shows the estimate of the benefits of reducing the dropout rate, expressed in the number of students who would have dropped out in the absence of the Project but do not. The Project’s goal is to reduce the dropout rate from 5.8% to 3.5%, which is equivalent to a reduction of 2.3 percentage points, between the beginning and the end of the implementation period. In annual increments, the above would mean a reduction of 0.39 percentage points. 2006 2007 2008 2009 2010 2011 2012 2013 FORECAST Total enrollment - net Public 24,262 25,019 28,431 31,888 35,047 35,003 35,673 Private 6,090 8,466 7,611 6,485 5,413 7,257 8,166 Total 30,352 33,485 36,042 38,373 40,460 42,260 43,839 Dropout Rate 5.8% 6.6% 7.0% 5.9% 5.0% 4.5% 3.5% Dropout Reduction -0.4% -0.8% -1.2% -1.5% -1.9% -2.3% Dropout Reduction (student equivalent) -97 -220 -369 -539 -672 -820 ACTUAL Total enrollment - net Public 45,479 47,512 47,700 52,442 55,635 58,456 56,821 55,242 Private 11,415 12,446 15,410 13,957 11,546 9,037 11,780 12,645 Total 56,894 63,110 66,399 67,181 67,493 68,601 67,887 Dropout Rate 6.0% 5.8% 5.8% 6.3% 4.0% 3.8% 5.4% 4.5% Dropout Reduction -0.3% -0.5% -0.8% -1.0% -1.3% -1.5% Dropout Reduction (student equivalent) 44 d) Increased tax collection (BIM) While the benefits of the Project are mainly shown as an increase in individual incomes, other effects should also be considered. This is the case of higher tax revenues associated with increases in individual income, for instance. For this calculation, we assume that the increase in tax revenue amounts to 10% of individual income. e) Externalities - Education (EE) Although the Project indicators related to quality do not allow for inferences from a financial standpoint, the calculations carried out here assume the existence of a direct relationship between Project interventions and the development of the student competences. It should also be noted that there is no evidence on the effects of improved quality of education on macroeconomic variables such as economic growth in Colombia. To assess the benefits that a possible improvement of the quality of education could have on individual income we used research by Eric A. Hanushek and Ludger Woessmann (2012) (“Do better schools lead to more growth? Cognitive skills, economic outcomes, and causation”). Using the results of PISA 2009, the authors conclude that an increase of one standard deviation in the skill level of students would cause an increase of 2 percentage points in per capita income. The Project assumes an increase in revenues equal to 1% (0.5 standard deviations). f) Externalities - Health (ES) Likewise, an increase in the earnings of individuals benefiting from the Project would also have an effect on the health system. An increase in labor force formality means an increase in contributions to the health system (the current amount contributed is 12% of income, which is the figure we use in the analysis). While other indirect effects (for example, on the health on beneficiaries) is to be expected, this falls beyond the scope of this analysis, to the extent that there are virtually no studies attempting to measure the value of this kind of externality. 2. Costs a) Costs for individuals This is equal to the sum of the costs incurred by students to complete a tertiary degree. These correspond to the sum of tuition (CM), which varies depending on the degree type; opportunity costs (CO), which is equal to the average annual income of a worker with an upper secondary diploma; and finally, the sum of living expenses and the cost of school supplies. The total cost for all individuals is shown below. = ∑( + + ) 45 b) Additional costs Additional costs are those related to the expansion of coverage. To increase completion rates, more teachers, administrators, and supplies are needed. The cost estimate was made based on the current upper secondary budget and financial framework (COP 1,517,209 per student year). 2006 2007 2008 2009 2010 2011 2012 2013 FORECAST Increase of student spots (Annual) 3,133 2,556 2,331 2,087 1,800 1,579 Increase in retention - Student equivalent 148 337 568 835 1,046 1,283 Dropout reduction (student equivalent) 97 220 369 539 672 820 Net student spots created 2,889 2,000 1,394 713 82 -524 ACTUAL Increase of student spots (Annual) 2,221 4,742 3,193 2,821 -1,635 -1,579 Increase in retention - Student equivalent 0 0 0 0 0 0 Dropout reduction (student equivalent) 0 0 0 0 0 0 Net student spots created 2,221 4,742 3,193 2,821 0 0 c) Cost of the Project The analysis used the actual costs of the Project by component, as shown in the data sheet. 3. Cost - Benefit Analysis The cost-benefit analysis performed using projected targets provides a sound basis to estimate the economic results of the Project. The Project yielded a Net Present Value (NPV) of US$67.7 million (COP172.4 billion), compared to the forecasted US$76.9 million (COP195.9 billion). The actual internal rate of return (IRR) stands at 11.9%, slightly below the projected IRR, which was 13.3%. The NPV was hampered by the modest improvements in retention, but this was more than offset by higher than expected results in transition to higher education. The actual IRR and NPV also compare favorably to those calculated at appraisal (10.8% and US$11.5 million, respectively). Projected (COP) Achieved (COP) Increased access to upper secondary 61,373,561,520 60,949,311,729 Transition to higher education 336,716,851,614 388,984,286,804 Improvements in retention 32,249,537,440 0 Increased tax returns 39,809,041,313 44,993,359,853 Improvements in quality 3,980,904,131 4,499,335,985 Health externalities 47,770,849,576 53,992,031,824 Individual costs (upper sec.) -67,565,202,688 -67,772,231,872 Individual costs (higher ed.) -193,077,418,725 -228,471,575,528 Other costs -31,231,990,454 -50,685,005,280 Project costs -34,125,379,088 -34,125,379,088 Benefits / Costs 195,900,754,638 172,364,134,428 Benefits / Costs (US$) 76,908,677 67,668,436 Internal Rate of Return 13.27% 11.87% 46 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Team Leader/Edu Martha Laverde Senior Education Specialist GEDDR Specialist Operational/Adm. Patricia M. Bernedo Senior Program Assistant LCSHE Support Financial Jeannette Estupinan Sr Financial Management Specialist GGODR Management Operational/Adm. Sonia Levere Language Program Assistant GHNDR Support Operational/Adm. Elsa Coy Team Assistant LCCCO Support Jose M. Martinez Senior Procurement Specialist Procurement Luis Pisani Consultant LCSHE Technical Support Gustavo de Roux Consultant LCSHE Technical Support Carolina Nieto Consultant LCSHE Technical Support Supervision/ICR Team Leader/Edu Martha Laverde Senior Education Specialist GEDDR Specialist Operational/Adm. Patricia M. Bernedo Senior Program Assistant GSPDR Support Claudia Mylenna Cardenas E T Consultant LCSFM Technical Support Garcia Operational/Adm. Elsa Coy Program Assistant LCCCO Support Financial Jeannette Estupinan Sr Financial Management Specia LCSFM Management Operational/Adm. Viviana A. Gonzalez Program Assistant GHNDR Support Operational/Adm. Sonia M. Levere Language Program Assistant GHNDR Support Jose M. Martinez Senior Procurement Specialist ECSO2 Procurement Juan Manuel Moreno Lead Education Specialist GEDDR Education Specialist Operational/Adm. Aracelly Woodall Senior Program Assistant GTIDR Support Keisgner Alfaro Senior Procurement Specialist GGODR Procurement Financial Luz Zeron Financial Management Specialist GGODR Management Alejandra Corchuelo Consultant GSPDR Technical Support Diomedes Berroa Lead Specialist OPSOR Procurement Operational/Adm. Paula Andrea Bernal Team Assistant LCCCO Support 47 Operational/Adm. Antonella Novali Program Assistant GEDDR Support Octavio Medina Junior Professional Associate GEDDR Technical Support ICR Team Marcelo Becerra Lead Education Specialist GEDDR Leader/Edu Specialist William Experton ICR Consultant GEDDR ICR Consultant ICR - Maria Elena Paz Gutzalenko Program Assistant GEDDR Operational/Adm. Support Operational/Adm. Ixora Gonzalez Team Assistant LCCCO Support Janet K. Entwistle Senior Operations Officer GEDDR ICR Team Leader ICR - Jagannath Griffiths Palma E T Temporary GEDDR Operational/Adm. Support (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY06 9 76 FY07 17 91 FY08 35 179 FY09 8 35 Total: 69 381 Supervision/ICR FY09 9 49 FY10 23 101 FY11 21 120 FY12 21 112 FY13 19 90 FY14 20 122 FY15 13 95 Total: 126 689 48 Annex 5. Beneficiary Survey Results (if any) N.A. 49 Annex 6. Stakeholder Workshop Report and Results (if any) N.A. 50 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Project for the Improvement of High-school Education in Antioquia 1. Project Objectives and components The Project for the Improvement of High-school Education in the department of Antioquia, set as primary targets: (i) increasing equitable access and retention in high-school education (ii) improving the quality of high-school education, and (iii) increasing the proportion of graduate students from public high-schools that continue to higher education. These objectives were set up based on specific goals for the poorest zones of the department, using indicators such as: • Increasing gross and net coverage in high-school education. • Reduction of the gap between the results of official and private institutions in the language and mathematics areas. • Increasing the completion rate of high-school education. • Increasing the transition rate of students from high-school education to higher education. Likewise, the Project was structured through 3 components: (i) Equitable access to high-school education and opportunities to continue higher education; (ii) Quality and relevance of high-school education; and (iii) Institutional invigoration, Monitoring and Evaluation and Project Management. 2. Measurement of results The implementation of the Project began in 2008 with a set of 451 educational institutions of the department that provided high-school education. This set of institutions underwent a series of interventions based on the components of the program, which sought to contribute to the achievement of the Project’s core objectives as previously described. In order to measure its development objectives, the Project contemplated clustering the municipalities of Antioquia into 3 main groups based on a set of socio-economic variables. Therefore, the central axis of the intervention in the department took place in 25 municipalities considered to be the ones with the lowest socio-economic levels and mainly located in the sub- regions of Urabá and Lower Cauca. Additionally, 41 municipalities were placed in a middle level and 51 of them were considered as municipalities with high socio-economic conditions. By 2014, 37 of these 451 institutions were located in municipalities that obtained their certification in 2009 or 2010 (Sabaneta, Rionegro and Apartadó), which received a different type of care from the certification of their municipality. The remaining 414 institutions correspond to the three categories of municipalities: 174 are located in wealthy municipalities, 136 in middle level municipalities and 104 in the poorest municipalities. Reviewing the results of the Project in the department regarding coverage, the Project has achieved a gradual increase in the gross and net coverage in high-school education, and municipalities or zones with higher levels of poverty have been able to increase the number of students registered annually at this level. 51 Coverage Rates Antioquia 2007 - 2013 70.0% 65.8% 67.8% 68.3% 60.0% 64.2% 64.9% 61.9% 50.0% 60.0% 37.5% 37.5% 37.3% 40.0% 30.0% 34.5% 36.0% 31.4% 32.5% 20.0% 2007 2008 2009 2010 2011 2012 2013 Tasa Neta Tasa Bruta As far as gross coverage is concerned, this indicator increased from 60% in 2007 to 68.3% in 2013, with a relatively constant increase every year. On the other hand the net rate coverage increased in almost 6 percentage points going from 31.4% to 37.3%. However, these increases have not occurred in the same way across Antioquia. When the gross coverage is analyzed separately with respect to the groups of municipalities, it is clear that the middle socio-economic level group of municipalities greatly increased its coverage rate that went from 52% to almost 70% with an increase of almost 18 points; although it is highlighted that the most significant progress took place during years 2012-13. Gross coverage rate per type of muncipality 2007-2013 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 2007 2008 2009 2010 2011 2012 2013 Ricos Medios Pobres Regarding the wealthy municipalities, it is clear that their coverage level is superior to the other municipalities and that it keeps a relatively stable trend to rise throughout the period going from 71.6% to 78.1% in 2013. On the other hand, gross coverage in poor municipalities, maintains a trend to rise during the period of 2007-09, reaching their maximum of 58% in this last year. From there on, the behavior was variable and as of 2011 it seems to have a reduction in the coverage level at this level. 52 Nevertheless, when the behavior of the total registration at this level is analyzed, the behavior by groups of municipalities allows some additional analyses. Therefore it can be observed that as of 2011 the level of registered students has had reductions in the three groups although it is more severe in the group of wealthy municipalities and to a lesser extent in the group of poorer municipalities. Enrollment evolution in upper secondary education in Antioquia 2008-2014* 30,000 25,000 Registered Ricos 20,000 Medios Pobres 15,000 10,000 2008 2009 2010 2011 2012 2013 2014 Additionally, between 2008 and 2014 the registration volume increased in almost 3,131 students, that is to say, a percentage close to a 6% increase in the total number of registered students at this level. Nevertheless, when this same behavior by groups of municipalities is analyzed, it is clear that poor and middle level municipalities obtained increases in the number of registered students. In this sense, between 2008 and the 2014 poor municipalities increased their registration by about 10%, middle level municipalities increased by about 8% and wealthy municipalities only obtained an increase of 4%; this indicates that a good part of the increase in the department’s registration is explained by the behavior of the poorest municipalities. Thus, it can be seen that there is an important increase in the registration level in the poorest municipalities of more than 1,000 students in the period considered, and considering that there are few municipalities in this group and relatively less educational institutions, the increases can be considered to be significant. 53 Enrollment evolution in upper secondary education in poor municipalities in Antioquia 2008-2014 12,000 11,942 11,634 11,671 Registered 11,500 11,347 11,311 11,384 11,000 10,616 10,500 2008 2009 2010 2011 2012 2013 2014 Pobres Linear (Pobres) The above demonstrates that although the increases in the coverage rate do not show the expected results, there is an important increase in the level of registered students in the poorest municipalities and it will be necessary to make important assessments of the population numbers to explain the behavior of this rate. With regards to quality, the results for the department still show important challenges for both the official sector and the non-official sector; the fundamental objective meant to narrow the current gap between the results obtained. It can be observed that the results compared between official and non-official institutions in the department in the language and mathematics areas do not show important variations especially in the last two years. Public/private relative indicator SABER 11 Language and Mathematics test 2007-2013 1.10 1.05 1.00 0.95 0.90 2007 2008 2009 2010 2011 2012 2013 Matemáticas Lenguaje In the case of mathematics, the results analyzed since 2007 show that between official and non- official institutions there were important differences. Nevertheless, a relative improvement in the results of the mathematics area is seen and, already for 2013, the results for both groups of these institutions tend to be closer and therefore the gap between them has been reduced. 54 In the case of language, it can be seen that although official institutions maintain a superior level during the first period of analysis, it can also be observed that for the last two years, the results of both groups end up being closer and gaps in this group are also reduced. However, it is necessary to note that the SABER 11 test had some methodological changes during 2012 that may have affected the historical trends. Additionally, when analyzing the evolution of the individual results of the 25 poorer municipalities it is clear that important progress was made in a group of municipalities in language and mathematics. 21 municipalities obtained improvements in the language area and 16 of them show differences of more than one point in the SABER 11 results, which can be considered significantly high. In the case of the mathematics the results of these 25 municipalities are more dissimilar, since almost half of them improved their results, while the remaining group maintained or reduced its results, causing non-significant changes in the global results of this group in this area as previously mentioned. Another important aspect is related to the rate of high-school education completion, which during 2007-11 maintained a trend to decline in both official and private institutions, and in all of the municipalities of Antioquia. In general terms, the completion rate measured through the information of c-600 in Antioquia, had a declining trend and during 2007-11 it decreased in a total of 10 points. Nevertheless, it is necessary to clarify that during 2009-10 the process to implement Decree 1290 in charge of the regulation of the students’ institutional evaluation and promotion system was carried out, which indirectly affected the students’ repetition rates and therefore the completion rate was affected during the years of its implementation. Nevertheless, a slight increase was seen in 2011, especially in the poorest municipalities. Completion rate in high-school education 2007-2011 85.0 80.0 75.0 78.3 78.5 74.9 70.0 65.0 67.4 69.0 60.0 2007 2008 2009 2010 2011 Ricos Medios Pobres Total Following there is a brief description of the implementation and the results achieved by the Project in each one of the components and subcomponents of the Project during the 2008-14. Component 1: Equitable access to high-school education and opportunities to continue onto higher education. 1.1 Implementation of flexible pedagogical models 55 The government of Antioquia, within the framework for the implementation of flexible models for high-school education, has concentrated its efforts in the promotion of the rural high-school education by means, not only of the implementation of the model but of the education and qualification of leading teachers in the application thereof, without having the supply of flexible pedagogical models for high-school education compete with traditional education. In this regards the main efforts in the supply of flexible education has concentrated in the rural areas of the department. Students Served with flexible Pedagogical Models for high-school education Flexible 2009 2014 Methodology Post-elementary 32 Tele-high-school 15 Etnoeducation 105 106 SAT presencial 5.234 Rural High- 1.032 2.413 school Total 6.418 2.519 Source: Secretary of Education of Antioquia, Annexes 6A of December 30, 2009 and March 31, 2014 In this front, almost 20 official educational institutions of 12 municipalities of the sub-regions of the low Cauca and Urabá, were accompanied for the implementation in 2013 of the educational model “pensar media” developed and operated by Corpoeducación. Training was carried out in these institutions for the adequate implementation of the flexible model and to guarantee the harmonization between its pedagogical proposal condensed in the Institutional Educational Project – PEI - and the pedagogical model so that its installation in the educational institution and its continuity is guaranteed to focuson the rural population. Additionally, this coaching was supported by a strategy to link off-school students (those students outside the educational system) aiming to achieve the identification of this population in these municipalities and their future entailment to the system through a flexible model mode. 1.2 Direct subsidies to high-school students The program of incentives for permanence had to do with the delivery of direct economic resources to high-school students in the department of Antioquia. The graph below shows the number of students that benefitted from the Project during its implementation, where it stands out that the data for 2012 is lower since it corresponded to the Project’s transition period with the change of Departmental Administration and year 2014 corresponded only to the first half of the year. 56 Beneficiary incentives to retention 2009-2014 6,000 5,142 4,868 5,000 4,496 3,733 3,554 4,000 3,000 2,000 1,587 1,000 - 2009 2010 2011 2012 2013 2014 The incentives for the permanence program were developed during this period, making direct transfers to high-school education students and also contributed, among others aspects, to improve the performance of high-school education students. Therefore, during 2013 a preliminary evaluation of the program was made in order to estimate its impact on the students’ academic results behavior. A comparison of the results through the methodology of differences in differences was carried out in this evaluation, to assess the impact of the program at municipal level, in the results of the common core areas. There, it was found that the program had important effects on the behavior of the language area since it was statistically proven that in the municipalities that participated in the program there were no significant differences in the results in 2000-08, whereas for the years subsequent to the implementation, significant improvements were seen in the municipalities that had a greater number of students who benefited from the program, something that can also be observed in the English and physics areas. 17 1.3 Scholarships conditioned to increase access to higher education The government of Antioquia, has started different strategies that have allowed high-school students access to higher education. Until 2011 and part of 2012, the operation of the Gilberto Echeverri Fund with almost 6,000 million pesos, carried out actions to benefit the students with credits and maintenance subsidies for their access to higher education. Subsequently, the conditioned scholarship fund with the main purpose of supporting young people of low income and excellent academic conditions in the department to access higher education of relevance and good quality.18 The scholarship fund created in 2012, intends to grant full conditioned scholarships for registration fees and support for maintenance (maintenance, transport and/or housing) that allows more young people in the regions of Antioquia to access higher education with a relevant approach for the regions of the department. 17 Evaluation of the high-school education improvement Project in Antioquia and selection of incentives beneficiaries. December 30, 2013 18 Resolution 063642 of October 22, 2012 . 57 Beneficiaries of the Scholarships Fund of the Government of Antioquia 10,000 6,181 4,326 5,000 568 - 2012 2013 2014-1 This has allowed young people in social levels 1, 2 and 3 and under 26 years of age in all the sub- regions of the department to have true opportunities to access higher education in a set of institutions prioritized by the government in in technical professional education, technological and professional university education. Thus, for 2012 the summoning for the first period of 2013 began, resulting in almost 568 beneficiaries who began their studies in 2013. On the other hand, in 2013 two additional summons were carried out resulting in a total of 4,326 beneficiaries and for the first half of 2014 the total beneficiaries corresponded to a total of 6,181 people with the investment of resources coming from both royalties as well as from direct resources of the government of Antioquia for a total of 30,957 million pesos. 1.4 Professional, vocational orientation and project of life Regarding vocational orientation and the project of life, the government of Antioquia implemented different strategies that allowed providing information and assistance to high-school students on the opportunities to continue towards higher education and, generally, for the construction of their project of life. In this order of ideas, several strategies such as Fábrica del Futuro were carried out between years 2010 and 2011, where different higher education institutions among other players, provided guidance to high-school students on financing alternatives, educational programs available as well as assistance on subjects such as sexual and reproductive health, prevention of psychoactive substances consumption and healthy lifestyles. 10 sub-regional high-school shows were carried out in 2013, where 11th grade students from the municipalities of Antioquia, with the exception of Medellín, received information on different higher education programs by the institutions, as well as academic spaces for discussion and financing opportunities that the students of the department could have through the department’s scholarship fund. Additionally, the government of Antioquia, through the summation of efforts between the Secretariat of Education and the Secretariat of Productivity and Competitiveness, started the entrepreneur seed program in 2013,which allowed the education of 9,982 students of grades 10 and 11 in the department of Antioquia in 2012 and 11,103 in 2013 with the aim of stimulating leadership, developing creative ideas and building projects of life in high-school students, that is 58 also linked to the Antioquia E model for the promotion of entrepreneurship and enterprise development in the entire department. Additionally, the Project for the improvement of high-school education, through the sub-projects formulated in an individual way by the educational institutions that belong to the program, handed out resources so that the educational institutions would develop a vocational orientation and project of life program for high-school students. Component 2: Quality and relevance of high-school education. 2.1 Strengthening school work management for the quality of high-school education Under this subcomponent, the design and implementation of a competence-based curriculum that incorporated both, basic competences, as well as scientific, citizen competences and human capital formation in management and leadership, was started in the educational institutions that participated. In this regards, the design of the competence-based curriculum in all the institutions that participated began in 2011, giving a characterization of the institutions and an initial design of the competence-based curriculum. Teachers were accompanied in this intervention in the design of most important curricular elements and later their implementation was accompanied. This is how since 2012 until mid- 2014, all participating institutions have been accompanied in the competence-based curriculum implementation through the on-site presence of a local pedagogical adviser. This coaching has allowed for the refining, fitting or re-contextualizing of important elements of their teleological component, elements of the curricular management (macrocurriculum, microcurriculum and pedagogical model), classroom plan and quality plan, achieving that more than 70% of them have currently made important progress in all the components beyond existence and some of them show significant progress in their implementation in agreement with the information collected by local advisers. This work, was also supplemented with educational strategies for teachers for the implementation of the competence-based curriculum and evaluation by competences carried out in 2013 where almost 2,593 teachers from all the sub-regions of the department participated, of which almost 492 of them belonged to institutions of the department’s poorest municipalities. This teachers’ education, was supplemented with education of teachers in charge of the “Rectores Líderes Transformadores” program during 2013 that also continued in 2014. This education was meant to strengthen the leadership of teachers who direct official educational institutions, which allow the transformation in the medium and long term of the quality of education in their respective institutions. Therefore the education concentrates on aspects related to the director’s personal jurisdiction (in his life and actions) and the invigoration of the director’s competitiveness as a pedagogical leader of their educational institution. In this program, 108 directors of 55 municipalities of the department were certified, of which 9 are from the poorest municipalities in the department. 59 Additionally, the Project carried out an important coaching in the institutions of the department for the implementation of the methodological route of school coexistence, that was developed in all the institutions that participated and that was supplemented with the incorporation of law 1620, which created the national school coexistence and education system for the exercise of human rights, education for sexuality and prevention and mitigation of school violence. In this order of ideas, the project accompanied the institutions in the creation of coexistence committees, collecting and structuring a diagnosis of school coexistence in a participative way, construction of a plan of action articulated to the plan for institutional improvement and the execution thereof, as well as the incorporation in the educational institution of the citizen competences approach for school coexistence, thus achieving that 60% of the Project’s institutions declare to have reached a great part of the objectives set forth in their school coexistence plans by 2013, which demonstrates the progress made in the implementation of the route, in spite of the re- statement that was necessary in 2013 due to the new legislation. In this subcomponent, education strategies for high-school students on basic competences in language and mathematics were also carried out through the program “reinforce your competences” that took place in 2013 and the first half of 2014. Under this program around 16,398 students of grades 10 and 11 were educated. This intervention was focused on the poor municipalities that also had low results in the external tests SABER 11; students of 20 out of the 25 poorer municipalities participated in this intervention. One of the most notorious results of this intervention was observed in the results of the SABER 11 tests in 2013, during which this group of institutions obtained significant progress in the language and mathematics areas. The institutions that participated in the program, managed to increase their average scores in comparison to the 2012 scores in almost 1.7 points, whereas in the mathematics area, although there was a reduction in all the institutions of the department, the results of these institutions do not have significant differences. Another important aspect in the intervention of this subcomponent is related to the resource transfers to the set of educational institutions for their intervention in the institutional sub-projects designed by the educational institutions with support of the local adviser. Such sub-projects had 7 lines of investment, that is to say: 1. Implementation of the competence-based curriculum within the framework of the improvement plan. 2. Education of teachers and school directors in the application of the competence-based curriculum. 3. Coexistence and peace 4. Professional, vocational orientation and project of life 5. Improvement of School environments 6. Pedagogical resources and information technologies 7. School management. 60 On this point, the role of the financial advisers in charge of orienting the educational institutions and their directors in relation to the transparent investment of the resources provided within the framework of the institutional sub-projects, was particularly important. 2.2 Technical high-school education reform (information pending of the evaluation of the supply of technical high-school education and technical education) Component 3: Institutional invigoration, Monitoring and Evaluation and Project Management.  Pedagogical Technical advice and assistance The technical assistance to the municipalities and the educational institutions was one of the main strategies of the Project, in order to strengthen the same institutions and the entire official educational system in the department. Therefore the group of local advisers was responsible for: 1. Supporting the Director and the Board of Directors in the activation of the participation of the Institution’s educational players in relation to the project. 2. Orienting, from a participative process, the identification of needs in goods, works, services and technical assistance for the invigoration of high-school Education. 3. Accompanying the Institutions in the project’s planning, organization, execution, follow- up and evaluation. 4. Orienting new policies and resources management for the project’s financing. 5. Sensitizing municipal authorities, directors, teachers, students, parents and Boards of directors in the Project 6. Reviewing in a participative manner the diagnosis of high-school Education, starting off from the existing baseline 7. Checking the coherence of the sub-projects with the high-school Education Improvement Plan, the PEM, the PEI and the Project’s guidelines. 8. Providing advice, support and technical assistance to the institutions in the design and implementation of the curriculum by competences and in general, the curricular management (macro and micro curriculum and pedagogical model) as well as the teleological component, the classroom plan and the quality or improvement plan. What was achieved by this intervention was that the coaching was made very closely to the directive teams in the educational institutions that participated as well as to the community in general. Aspects that were observed in the behavior of the level of satisfaction of the educational institutions, their teaching and directive staff with the work performed by the local adviser that was satisfactory for almost 97% of the institutions that participated, growing in more than 10 percentage points since 2010, where it was closer to 87%. 61 Level of satisfaction of the teaching and directive staff with the local advisor 100.0% 96.7% 95.0% 90.0% 87.0% 86.7% 85.0% 80.0% 2010 2011 2013 In general, these results show success in the intervention of local advisors and in the generation of installed capacity to the interior of the Secretariat of Education to provide counseling and technical assistance to educational institutions and municipalities regarding educational management and specifically academic management.  Follow-up and assessment System Finally, the Project had designed a follow-up and assessment system that allowed accounting for the interventions performed at all levels in the department, and specifically in the institutions that participated. As support to the follow-up and assessment system there was the Alphasig information system that allowed to directly perform an important management of the information for the topics of incentives to permanence, registration, follow-up for graduated students, follow- up to the execution of institutional sub-projects and indicators. This software was implemented in all the educational institutions that participated and supported the management of the information between the Secretariat of Education and the educational institutions. 3. Lessons Learned Local Pedagogical advice The extensive work developed by local advisors allowed to generate room for improvement in the educational institutions. Therefore, it is important to highlight as a lesson that a direct approach with the institutions generates important challenges with regards to sustainability in time so that periodical revisions and adjustments to elements like the PEI, curricular grid, pedagogical model, classroom plans and quality plans render the effects desired in the quality of the students’ learning. Therefore, it has been observed that the generation of pedagogical spaces for participative work around the academic processes, where the Secretariat of Education is present, will allow the educational institutions to progress in institutional improvement. The effective entailment of teachers and school directors in the design and implementation of all these elements is also important so that its future sustainability and its true incorporation in the classroom is guaranteed, which in turn assures an improvement of the students’ results and a their 62 better development as citizens and in the labor and academic world subsequent to high-school education. Technical education Accounting and financial management of the project The implementation of controls and strategies for the follow-up and control of the accounting and financial information allowed performing a suitable control of the resources, not only those originating in the credit but the own resources of the government of Antioquia invested in the Project. Regarding this aspect, the Secretariat of Education was able to obtain important progress that allowed for the improvement of the quality and opportunity of the financial information for decision making especially during the Project’s closing. These aspects were also confirmed by the results of the last years in the Project’s external audit. Follow-up and assessment System The design of tools that allow a periodic follow-up to the actions developed by the Project allowed an adequate accountability to a large extent of the interventions performed. Nevertheless, it is important that the information management processes be increasingly improved at all levels in the Secretariat of Education, making of this process a continuous aspect that allows the flow of information and that will increase informed decision making on the educational processes in the department’s municipalities and institutions. 63 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders N.A. 64 Annex 9. List of Supporting Documents Secretariat of Education of Antioquia (2013), Evaluación del Proyecto de Mejoramiento de la Educación Media en Antioquia y Selección de Beneficiarios de Incentivos, Final Report, December 30, 2013, Antioquia, Colombia. World Bank (2007), Colombia: Antioquia Upper Secondary Education Project, Loan No. 7492- CO, Project Appraisal Document, October 24, 2007, Washington, D.C. World Bank (2008), Colombia: Antioquia Upper Secondary Education Project, Loan No. 7492- CO, Loan Agreement, March 14, 2008, Washington, D.C. World Bank (2012), Colombia: Antioquia Upper Secondary Education Project, Loan No. 7492- CO, Restructuring Paper, December 18, 2012, Washington, D.C. 65 MAP 66