Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004307 IMPLEMENTATION COMPLETION AND RESULTS REPORT ON A LOAN (8226-PE) IN THE AMOUNT OF US$25 MILLION TO THE REPUBLIC OF PERU FOR A BASIC EDUCATION PROJECT ( P123151 ) August 17, 2018 [This ICR replaces the version published in the Board Operations System on August 22, 2018. This version includes the revised Practice Manager name at approval] Education Global Practice Latin America And Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective March 19, 2018) Currency Unit = Peruvian Soles (PEN) PEN 1 = US$0.31 US$1 = PEN 3.26 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS AOP Annual Operating Plan BEP Basic Education Project CPF Country Partnership Framework DDO Deferred Drawdown Option DEI Directorate of Preschool Education (Dirección de Educación Inicial) Public Debt and Treasury Department (Dirección General de Endeudamiento y Tesoro DGETP Público) Directorate of Education Research and Documentation (Dirección de Investigación y DIDE Documentación Educativa) DIED Teacher Evaluation Directorate (Dirección de Evaluación Docente) General Directorate for Quality School Management (Dirección General de Calidad de la DIGC Gestión Escolar) DIGEDD General Directorate for Teacher Development (Dirección General de Desarrollo Docente) DLI Disbursement-Linked Indicator DPF Development Policy Financing ECE Student Census Assessment (Evaluación Censal de Estudiantes) ECERS-R Early Childhood Environmental Rating Scales – Revised EDI Early Development Indicator FM Financial Management GDP Gross Domestic Product GoP Government of Peru ICCS International Civic and Citizenship Study INEI National Bureau of Statistics (Instituto Nacional de Estadística e Informática) IO Intermediate Outcome IRR Internal Rate of Return MEF Ministry of Economy and Finance (Ministerio de Economía y Finanzas) MELQO Measuring Early Learning Quality and Outcomes MINEDU Ministry of Education (Ministerio de Educación) NPV Net Present Value Office of Monitoring and Strategic Evaluation (Oficina de Seguimiento y Evaluación OSEE Estratégica) PAAC Annual Acquisition and Procurement Plan (Plan Annual de Adquisiciones y Contrataciones) PDO Project Development Objective PELA Learning Outcomes Education Program (Programa Educativo Logros de Aprendizaje) PISA Program for International Student Assessment RBB Results-Based Budgeting RBF Results-Based Financing RF Results Framework SA Social Assessment SEPA Procurement Plans Execution System (Sistema de Ejecución de Plan de Adquisiciones) Integrated System of Financial Management (Sistema Integrado de Administración SIAF Financiera) SPE Strategic Planning Secretariat (Secretaría de Planificación Estratégica) SWAp Sector-Wide Approach Third Regional Comparative and Explicative Study (Tercer Estudio Regional Comparativo y TERCE Explicativo de Evaluación) Learning Quality Measurement Office (Oficina de Medición de la Calidad de los UMC Aprendizajes) Regional Vice President: Jorge Familiar Country Director: Alberto Rodríguez Senior Global Practice Director: Jaime Saavedra Practice Manager: Reema Nayar Task Team Leader(s): Patricia Alvarez, Ines Kudo ICR Main Contributor: Suzana N. de Campos Abbott TABLE OF CONTENTS DATA SHEET .............................................................................................................................1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .....................................11 II. OUTCOME .................................................................................................................... 13 A. RELEVANCE OF PDOs ............................................................................................................13 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................14 C. EFFICIENCY ...........................................................................................................................17 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................18 E. OTHER OUTCOMES AND IMPACTS (IF ANY)............................................................................20 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 21 A. KEY FACTORS DURING PREPARATION ................................................................................21 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................22 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 24 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................24 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................25 C. BANK PERFORMANCE ...........................................................................................................26 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................27 V. LESSONS AND RECOMMENDATIONS ............................................................................. 28 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 31 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 42 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 44 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 45 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 57 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 68 ANNEX 7: DISBURSEMENT LINKED INDICATORS (ACTUAL) .................................................... 71 The World Bank Basic Education Project ( P123151 ) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P123151 Basic Education Project Country Financing Instrument Peru Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency REPUBLIC OF PERU Ministry of Education Project Development Objective (PDO) Original PDO The objective of the Project is to improve MINEDU's capacity to evaluate student learning, instructional practice, and school leadership in Basic Education. Page 1 of 72 The World Bank Basic Education Project ( P123151 ) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 25,000,000 25,000,000 25,000,000 IBRD-82260 Total 25,000,000 25,000,000 25,000,000 Non-World Bank Financing Borrower 158,000,000 161,000,000 161,000,000 Total 158,000,000 161,000,000 161,000,000 Total Project Cost 183,000,000 186,000,000 186,000,000 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 17-Jan-2013 20-May-2014 21-Jun-2016 31-Mar-2018 31-Mar-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 17-Sep-2015 0 Change in Results Framework Change in Components and Cost Change in Disbursements Arrangements Change in Legal Covenants Change in Institutional Arrangements Change in Implementation Schedule Other Change(s) 07-Jun-2017 17.76 Change in Results Framework Change in Disbursements Arrangements Change in Institutional Arrangements Other Change(s) KEY RATINGS Outcome Bank Performance M&E Quality Highly Satisfactory Satisfactory High Page 2 of 72 The World Bank Basic Education Project ( P123151 ) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 03-Apr-2013 Satisfactory Satisfactory 0 02 23-Nov-2013 Satisfactory Moderately Satisfactory 0 03 28-Apr-2014 Satisfactory Moderately Satisfactory 0 04 24-Nov-2014 Satisfactory Moderately Satisfactory 0 05 22-Jan-2015 Satisfactory Moderately Satisfactory 0 06 03-Sep-2015 Satisfactory Moderately Satisfactory 0 07 19-Feb-2016 Satisfactory Satisfactory 5.40 08 08-Aug-2016 Satisfactory Satisfactory 11.25 09 03-Mar-2017 Satisfactory Satisfactory 16.72 10 23-Oct-2017 Satisfactory Satisfactory 17.76 SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Early Childhood Education 20 Public Administration - Education 40 Primary Education 20 Secondary Education 20 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 50 Enterprise Development 50 MSME Development 50 Page 3 of 72 The World Bank Basic Education Project ( P123151 ) Human Development and Gender 50 Education 50 Access to Education 25 Education Financing 25 ADM STAFF Role At Approval At ICR Regional Vice President: Hasan A. Tuluy Jorge Familiar Calderon Country Director: Susan G. Goldmark Alberto Rodriguez Senior Global Practice Director: Keith E. Hansen Jaime Saavedra Chanduvi Practice Manager: Reema Nayar Reema Nayar Task Team Leader(s): Ines Kudo Patricia Alvarez, Ines Kudo Suzana Nagele de Campos ICR Contributing Author: Abbott Page 4 of 72 The World Bank Basic Education Project ( P123151 ) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES 1. The Basic Education Project (BEP, Project), successfully supported critical aspects of the Education Program for Student Learning of the Government of Peru (GoP), whose higher objective was to improve student learning outcomes. The Project was implemented in a rapidly shifting environment, with changing government administrations (five ministers from appraisal to completion), a constant commitment for implementing system-wide reforms that directly affected the conditions for project implementation and facing—in its last year—both devastating impacts of the El Niño phenomenon and a long, difficult teacher strike pushing back on performance evaluations (a project activity). Remarkably, the Project counted upon the Government’s continuous support throughout its implementation. In fact, in one of the two project restructurings processed on time to maintain the Project’s continued relevance to the realities of implementation and the Government’s priorities, the Project’s scope was actually expanded. With this change, the Project not only supported, within the overall context of its Project Development Objective (PDO), the actual implementation of a notoriously complex Teacher reform in any environment. The Project met, and in many cases exceeded, targets for all its carefully selected outcome and intermediate outcome (IO) indicators and completed the originally planned and additional activities introduced through restructuring by the loan’s original closing date. With the World Bank’s loan designed under a Sector-Wide Approach (SWAp), loan funding did not accrue to the different categories of project expenditures. Rather, the main value added of the World Bank’s assistance was the timely, relevant, and continuous technical support delivered from the earliest stages of preparation through completion of this highly successful project. A. CONTEXT AT APPRAISAL 2. At the time of appraisal of the BEP in late 2012, Peru had become one of the fastest growing economies in Latin America, weathering well the impact of the global financial and economic crisis. Gross domestic product (GDP) growth averaged 5.7 percent per year between 2000 and 2011. This strong economic performance enabled a continuous improvement of Peru’s income per capita, which increased by more than 50 percent during the decade, after almost 30 years of stagnation. Particularly strong growth between 2004 and 2011 had resulted in significant improvements in poverty—almost 31 percent of the population were lifted out of poverty—and other social indicators. The GoP was highly committed to tackling the challenge of achieving more inclusive growth and reducing social gaps. Improving Peru’s human capital base was seen as essential for sustaining this strong growth and promoting more equitable poverty reduction and broader social and human well-being. Ensuring access of low-income and otherwise vulnerable children to good-quality basic education was considered essential to unleash skills formation and productivity, as the lack of a core set of skills was evidenced as a binding constraint to labor market insertion for these groups. 3. Peru’s Basic Education System had witnessed improvements in coverage, but these were not matched with improvements in quality. 1 There was near-universal coverage of primary education, with a net enrollment rate of 94 percent; access to preschool education had tripled in the previous five years reaching 70 percent and 80 percent; and the secondary net enrollment rate was above the regional 1Peru’s Basic Education System comprises three levels: preschool (ages 3 to 5 years), primary school (ages 6 to 11 years), and secondary school (ages 12 to 17 years). Page 5 of 72 The World Bank Basic Education Project ( P123151 ) average. Yet, schools graduated high numbers of students who lacked basic mastery of mathematics and literacy skills. Peru finished last among the nine countries in the Latin America and Caribbean Region that had participated in the 2009 Program for International Student Assessment (PISA) exam administered to a national sample of 15-year-old students. The PISA results mirrored results of Peru’s own national assessment of reading and mathematics skills, whereby in 2011, less than a quarter of Grade 2 students had demonstrated adequate reading proficiency and less than half were unable to perform basic mathematics. While outcomes were by far worse among children from rural indigenous communities, gender gaps in both schooling completion and learning were negligible. Still, according to the PISA results, Peru was the country where students’ socioeconomic characteristics had the greatest effect on learning gaps, even when they attended the same schools. Learning outcomes in Peru continued to be far below what was expected based on the country’s per capita income and educational investment as a percentage of GDP. 4. The GoP had designed and launched the Results-Based Budgeting (RBB) Education Program for Student Learning (Programa Educativo Logros de Aprendizaje, PELA) that aimed to raise learning outcomes in public schools. 2 A first phase of PELA had been implemented from 2009–2011 with mixed results; while learning outcomes had improved overall for reading and mathematics, insufficient attention to rural, bilingual education, coupled with unequal, regressive education financing had led to widening social gaps. A redesigned PELA program for 2013–2016, incorporating lessons from the first phase, was approved by the Ministry of Economy and Finance (Ministerio de Economía y Finanzas, MEF) in June 2012. With the objective of improving learning outcomes, the new PELA focused on four ‘products’: (P1) improving schools’ operating conditions and management, (P2) strengthening teaching and curriculum, (P3) improving timely access to quality learning materials, and (P4) establishing a robust national student assessment system to support a standards-led reform program. The 2013 budget allocatation to PELA amounted to PEN 10.2 billion (about US$3.9 billion), 89 percent of the total budget for basic education. 5. Although national assessments provide an important tool for improving learning outcomes, Peru had yet to establish a national student assessment system covering all key grades and subject areas. 3 Peru’s assessment program that consisted only of mathematics and reading tests administered at the Grade 2 level and participation in PISA every three years, was not meeting the need of the Ministry of Education (Ministerio de Educación, MINEDU) for regular information on the student learning program at the end of preschool and across primary and secondary levels. Nevertheless, the Grade 2 census-based tests (Evaluación Censal de Estudiantes, ECE) that had been carried out annually over the previous six years met international standards of quality, and the results had been well-analyzed, broadly disseminated by gender, urban/rural, and geographic locations, and used for policy design. In 2010, over 86 percent of schools received the full package of tailored reports (for parent, teachers, and principals) of the previous year’s test, allowing for more effective decision making and accountability. By providing invaluable feedback to schools, MINEDU, parents, and teachers, the ECE had already shown important improvements in learning outcomes: between 2007 and 2011, the proportion of Grade 2 students reaching the expected level in national reading comprehension assessment increased from 17 percent to 2 The RBB approach was introduced by the MEF in 2008 to prioritize public resource allocation to comprehensive sector-wide strategies and activities for which there was substantiated evidence of impact on the end result (in this case, student learning). 3 Key grades are those that mark the end of a cycle. Basic education in Peru is divided into seven cycles: Cycles I and II for preschool corresponding to ages 0–2 years and 3–5 years respectively; Cycles III, IV, and V for primary corresponding to Grades 1 to 2, 3 to 4, and 5 to 6 respectively; and Cycles VI and VII for secondary corresponding to Grades 7 to 8 and 9 to 11. Key grades are therefore age 5, Grade 2, Grade 4, Grade 6, Grade 8, and Grade 11. Page 6 of 72 The World Bank Basic Education Project ( P123151 ) 30.5 percent, and in mathematics rose from 9.4 percent to 13.2 percent. Based on this positive impact, PELA aimed to scale up the ECE to other grades to analyze issues, inform teacher development programs, ‘anchor’ performance evaluation systems, and provide more extensive feedback to schools to underpin their school development plans. 6. A critical aspect of strengthening teaching was to implement a robust teacher evaluation system —within the newly approved teaching career—that identifies, encourages, and rewards good teaching. Teachers in Peru were drawn from the bottom-third of higher education students, and several content mastery tests for teachers had revealed troubling levels of performance. Classroom observations had documented inefficient use of instructional time, poor lesson preparation, and the preponderance of highly traditional instructional methods, such as copying from the blackboard. MINEDU recognized that substantial upgrades in teacher quality would be a long-term process that would depend on higher standards, higher pay, a more attractive career path, higher quality preservice education, and more effective in-service training, all addressed in an integrated manner. For this, the GoP had passed a Teacher Reform Law to establish a performance-based career that would raise the standards for entry into the profession, regulate promotions across a more remunerative career path based on competency, and implement regular evaluation of teacher performance. 7. Finally, also as part of PELA and the implementation of the teaching career, the GoP’s strategy sought to ensure more effective school leadership through competency-based selection processes and specialized training. In Peru, by law, school principals were appointed for a three-year period, after which they should be evaluated to be confirmed in their posts. However, there had been no evaluations of school principals since 2006, and, as a result, about 75 percent of schools were being managed by acting principals. The new teaching career required the development of a transparent competency-based selection process, followed by the three-year appointment and a new evaluation. Considering that there had been no programs for training teachers on school management or leadership, MINEDU designed a four-month induction program for the school principals appointed through this competitive process, which later would turn into a comprehensive training program in management and instructional leadership. Implementation of this program would address the backlog of around 40,000 unconfirmed principals and vice principals. Theory of Change (Results Chain) 8. The Project’s results chain was designed to focus on three critical aspects of PELA, that together would be expected to have an impact on the Government’s higher-level objective of improving student learning outcomes. The Project activities support PELA products P1, P2, and P4, as its theory of change supported the eventual achievement of the PDO in three areas: (a) scaling up and benchmarking student learning assessments (P4), (b) designing and implementing teacher evaluations (P2), and (c) evaluating and training school leaders (P1). Each of these is described below, and figure 1 depicts the results chain. (a) For scaling up and benchmarking student learning assessments, outputs related to the piloting, expansion, and reporting at the regional and school level of student assessments in additional grades and subjects, together with national surveys carried out to evaluate child development progress and the quality of public preschool education services were expected to result in the Project’s outcome of an increased number of cycles of preschool, primary, and secondary education levels with student learning outcomes analyzed (PDO 1). Outputs Page 7 of 72 The World Bank Basic Education Project ( P123151 ) related to Peru’s participation in international assessments of student learning outcomes, would also lead to (a) the availability of reliable, timely information on student learning outcomes, trends, and patterns across the school cycle being used to inform policy decisions, school management, and teaching, which, in turn, would affect the quality, motivation, and career opportunities of teachers and would contribute to strengthening school leadership, and (b) an education system benchmarked against global standards. The data provided by these assessments would offer school managers and teachers the detailed information needed to focus future policy efforts to improve equity and learning outcomes. 4 (b) For designing and implementing teacher evaluations, outputs related to national studies of classroom instructional practice carried out and reported at the national level would result in having instructional practices analyzed (PDO 2). Outputs related to the design, implementation, and reporting of a system of teacher performance evaluation, together with the design of evaluation instruments to determine entrance to and promotion within the teaching career would also contribute to having instructional practices analyzed. This outcome would lead to an increased quality, motivation, and career opportunities of teachers and to having in place a new teaching career merit-based system of evaluation. As the quality of teaching is a major factor affecting learning outcomes, these activities would be expected to provide a major boost to student learning. 5 (c) For evaluating and training school leaders, outputs related to the design of a school principal evaluation and training system and certifying school principals in school management would permit achievement of having school management positions staffed through the school principal evaluation system (PDO 3). This outcome would lead to strengthened school leadership. International research suggested that principals who combined managerial skills and capacity for instructional leadership at the school level were crucial to improve school management and raise student outcomes. 6 4 De Hoyos Navarro, Rafael E., Alejandro Jorge Ganimian, and Peter Anthony Holland. 2017. “Teaching with the Test: Experimental Evidence on Diagnostic Feedback and Capacity Building for Public Schools in Argentina (English).” Policy Research Working Paper; No. WPS 8261; Impact Evaluation Series. World Bank, Washington, DC. http://documents.worldbank.org/curated/en/488751511886071513/Teaching-with-the-test-experimental-evidence-on- diagnostic-feedback-and-capacity-building-for-public-schools-in-Argentina. 5 Bruns, Barbara, and Javier Luque. 2015. Great Teachers: How to Raise Student Learning in Latin America and the Caribbean. Washington, DC: World Bank. Hanushek, E. A. 2011. “The Economic Value of Higher Teacher Quality.” Economics of Education Review 30: 466–79. 6 Fryer, R.G. 2017. Management and Student Achievement: Evidence from a Randomized Field Experiment. Cambridge, MA: Harvard University. Retrieved on June 2018 from https://scholar.harvard.edu/files/fryer/files/mgmt_20170515_with_tables_figures.pdf. Bloom, N., R. Lemos, R. Sadun, and J. Van Reenen. 2014. “Does Management Matter in Schools.” NBER Working Paper No. 20667. Issued in November 2014. Page 8 of 72 The World Bank Basic Education Project ( P123151 ) Figure 1. Results Chain Project Development Objectives (PDOs) 9. The PDO was to improve MINEDU’s capacity to evaluate student learning, instructional practice, and school leadership in Basic Education. Key Expected Outcomes and Outcome Indicators 10. Achievement of the PDO was to be measured through the following PDO-level results indicators: Page 9 of 72 The World Bank Basic Education Project ( P123151 ) (a) Number of additional grades at the Primary Education and Secondary Education levels in which sound national assessments are administered in at least two subject areas 7 (b) Annual results of national student assessments at the Primary Education and Secondary Education levels reported at the national, regional and school levels (c) Annual results of national studies of instructional practice at the Primary Education and Secondary Education levels reported at the national level (d) Number of school management positions staffed through the school principal evaluation system 11. The Project’s Results Framework (RF) also included five IO indicators (annex 1). Components 12. The Project comprised three components, all aiming to provide support for the implementation of selected activities of the Government’s program for evaluating student learning, instructional practice, and school leadership in basic education, over the course of four years, as described below. • Component 1. Evaluating Student Learning (US$20.245 million). Support for implementation of eligible budget activities supporting the program, including (a) the scale- up of the existing second grade student assessment to cover additional grades and subject areas in primary and secondary education schools; (b) the introduction of internationally validated methodologies for measuring child development outcomes and quality of services in preschool education; and (c) provision of support for continued participation in international assessments managed by the Organisation for Economic Cooperation and Development and United Nations Educational, Scientific, and Cultural Organization to compare trends in national results with international benchmarks. • Component 2. Evaluating Instructional Practice and School Leadership (US$4.249 million). Support for implementation of the eligible budget activities supporting the program, including, activities to strengthen MINEDU’s capacity for monitoring and evaluating the quality of instructional practice at the classroom level, and to implement a competency- based system for selecting and training candidates to school management positions. • Component 3. Strengthening MINEDU’s Implementation Capacity (US$0.506 million). Support for implementation of the eligible budget activities supporting the program, including, activities to strengthen MINEDU’s capacity to improve its education management and monitoring capacity for the activities supported by the program and its operational and fiduciary capacity for the implementation of the program, including the carrying out of independent technical, financial, and procurement reviews. 7 Soundness of the assessments was defined to mean that the tests measure core competencies in the curriculum relevant for the grade and subject being tested, and they do so consistently enough to yield the same results on repeated administration. Page 10 of 72 The World Bank Basic Education Project ( P123151 ) B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 13. The PDO was not revised. The PDO indicators were revised in a Level 2 project restructuring in September 2015 (see below), and the outcome targets for the original PDO Indicator 4 (Number of school management positions staffed through the school principal evaluation management system) were revised to reflect actual progress and changes in the operational plan. With this, the target for 2014 was revised to 15,000 as the original target (5,000) was exceeded; the target for 2015 was revised from 10,000 to 1,000 to reflect the lower number of available positions given that most had been staffed in 2014; and the target for 2016 was revised to 9,000 to complete the Project’s final target of 25,000. A subsequent Level 2 project restructuring in June 2017, revised targets for the same PDO indicator (Number of school management positions staffed through the school principal evaluation management system) from 25,000 to 18,000 for the same reason (Section II B, paragraph 25). The restructuring also revised targets for two IO indicators. Revised PDO Indicators 14. The PDO indicators were revised in the September 2015 restructuring. To better focus on outcomes, two new PDO indicators were introduced to replace the original first three PDO indicators that were downgraded to IO indicators. The two revised PDO indicators were: • PDO Indicator 1: Cycles of Preschool, Primary and Secondary Education levels with student learning outcomes analyzed and • PDO Indicator 2: Instructional practices analyzed. 15. The restructuring also increased the number of IO indicators from 5 to 10 to measure all the key steps critical to achieving the PDO. The three original PDO indicators were converted to IO indicators, and two new IO indicators were added to monitor new activities. Targets for three IO indicators were modified from Yes/No to a number to allow for more accurate tracking of progress. Also, targets were revised to reflect actual progress and changes in the operation’s plan and MINEDU’s organizational structure. 16. The June 2017 restructuring introduced an additional IO indicator (School principals certified in school management), and minor changes to indicator descriptions, verification protocols, and task responsibilities. Revised Components 17. The September 2015 restructuring revised the Project’s Component 1 (Evaluating Student Learning) to include the borrower’s participation in the International Civic and Citizenship Study (ICCS) in 2015, managed by the International Association for the Evaluation of Educational Achievement, and Component 2 (Evaluating Instructional Practice and School Leadership) to include three new subcomponents: (a) Teacher performance evaluation, (b) Design of the evaluation system to determine merit-based promotions within the teaching career; and (c) Design of the evaluation system to determine merit-based entrance to the teaching career. Page 11 of 72 The World Bank Basic Education Project ( P123151 ) Other Changes 18. The September 2015 restructuring also revised the following: (a) The Project’s implementation schedule from the original four calendar years (2013–2016) to five (2013–2017), mainly to allow time to complete additional activities that were incorporated in the new subcomponents that were to be completed in 2017, (the loan’s closing date was not extended, however—it remained March 31, 2018). (b) Targets in the RF to be non-cumulative to ensure consistency with the disbursement-linked indicator (DLI) tables. (c) Data source and responsibility for data collection of the core indicator from Integrated System of Financial Management (Sistema Integrado de Administración Financiera, SIAF) to school census and from the MEF to the Strategic Planning Secretariat (Secretaría de Planificación Estratégica, SPE), to reflect actual responsibilities and data sources. (d) A legal covenant to state that an independent verification agent (instead of agency) should be hired not later than three months after restructuring (instead of six months after effectiveness) to introduce flexibility to allow hiring of a firm or an individual and to allow additional time to hire the independent verification agent after this flexibility is introduced (e) Disbursement arrangements to assign equal total value to all DLIs and incorporate the three new teacher evaluation components. (f) Disbursement estimates to reflect implementation progress, the fulfillment of DLIs, and the availability of eligible expenditures and the Project’s latest operational plan. (g) Adjustments to reflect revised cost estimates and the inclusion of the three new subcomponents. (h) The Project’s institutional arrangements (section IIIB). (i) The definition of ‘Eligible Budget Activities’ in the legal agreement to include training. (j) The period covered by progress reports, covenants in the legal agreement (including the inclusion of a covenant to reflect the borrower’s responsibility for allocating adequate budgetary resources to the implementing units for the achievement of DLIs), the Project’s technical analysis to include discussion of the evidence on the importance of teacher evaluation and an increase in the project’s stakeholder risk from the original Substantial to High, given the risk that the Teachers’ Union would oppose the teacher evaluations. 19. The June 2017 restructuring also revised (a) the Project’s disbursement arrangements, to reallocate amounts assigned to two DLIs and include a new DLI (annex 8); (b) disbursement estimates to reflect implementation progress; (c) institutional arrangements to include the name of the unit responsible for carrying out the second preschool assessment under Component 1; and (d) the definition of ‘Basic Education’ to ‘preprimary, primary, secondary, special, alternative, and technical-productive Page 12 of 72 The World Bank Basic Education Project ( P123151 ) education,’ to broaden the definition to basic education in all its modalities, which broadly comprise regular basic education, basic education in specialized centers for students with severe disabilities, basic education for adults and working youths, and technical-vocational education. These are all relevant for achieving the PDO, but their inclusion was subject to interpretation in the original definition. Rationale for Changes and Their Implication on the Original Theory of Change 20. The modifications to the project description introduced in the September 2015 restructuring were aimed at strengthening and better focusing the Project on the Government’s priorities. Changes in Component 1 to reflect the borrower’s participation in the ICCS in 2015, would contribute to enhancing the borrower’s capacity to evaluate student learning in a third subject area (citizenship) and yield internationally comparable results for potentially benchmarking Peru’s own assessment in the subject. The inclusion of three new subcomponents focused on teacher evaluation under Component 2 were in response to MINEDU’s priority to implementing the Teacher Reform Law, adopted in late 2012, which effectively integrated all tenure teachers into the same career path (they were previously divided under two regimes). The new career path determines remunerations, promotions, other incentives, and sanctions based largely on performance, which according to the law must be evaluated regularly. As a result, a range of new evaluation instruments and processes were required to make all teacher recruitment and promotion merit-based, and to initiate a process of regular performance evaluation and feedback for all teachers. The modifications did not affect the theory of change, but, given the proven impact of the quality of teachers on learning, actually strengthened the instruments with which to achieve the PDO, and the higher-level objective of improved student learning outcomes. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 21. The PDO continues to be relevant to the World Bank’s FY17–FY21 Country Partnership Framework (CPF) for Peru. 8 Specifically, the CPF’s Objective 3, Facilitate absorption of skills and technology especially by small- and medium-size business, under Pillar 1, Productivity for Growth, highlights that technology absorption and managerial capacity of businesses in Peru is handicapped by scarce skills, which should begin to be built at early stages. The CPF elaborates that demand for secondary and tertiary workers continues to outpace supply, and that Peru needs to start developing skills and human capital from the early stages of education, including both primary and secondary. It further mentions that the low and heterogeneous quality of education contributes to the insufficient human capital in the workforce. In this respect, the Project is fully consistent with the CPF’s central objective of contributing toward long-term productivity in Peru. The PDO is expected to lead to raising student learning outcomes in basic education and contribute to closing the gaps in performance across regions and income levels, and is, therefore, fully consistent with the CPF. Through its restructuring, the PDO maintained its relevance to the GoP’s changing needs throughout implementation. The relevance of the PDOs is rated High throughout its implementation. 8 World Bank. 2017. Country Partnership Framework for the Republic of Peru for the Period FY17–FY21. Report No. 112299-PE. Page 13 of 72 The World Bank Basic Education Project ( P123151 ) B. ACHIEVEMENT OF PDOs (EFFICACY) 22. The PDO was to improve MINEDU’s capacity to evaluate student learning, instructional practice, and school leadership in basic education. As such, achievement of the PDOs is evaluated based on the following three objectives: to improve MINEDU’s capacity to evaluate (a) student learning in basic education, (b) instructional practice in basic education, and (c) school leadership in basic education. The Project has achieved (in several cases exceeded) its objectives despite challenging and changing implementation conditions, which is a reflection of the long-term institutional commitment to system- wide reforms that has survived administration turnover. These activities are being continued and scaled up by MINEDU beyond the Project’s implementation. Improve MINEDU’s Capacity to Evaluate Student Learning in Basic Education • PDO Indicator 1 (revised to IO Indicator 2): Number of additional grades at the Primary Education and Secondary Education levels in which sound national assessments are administered in at least two subject areas; Target: 5 • PDO Indicator 2 (revised to IO Indicator 3), with slightly revised wording): Annual Results of national student assessments at the Primary Education and Secondary Education levels reported at the national regional and school level; Target: YES • Revised PDO Indicator 1: Cycles of Preschool, Primary and Secondary Education levels with student learning outcomes analyzed; Target: 2 in 2015 (for example, Cycles II and V), 1 in 2016 (Cycle VI), and 1 in 2017 (Cycle IV) - Achieved 100 percent 23. This objective was achieved. MINEDU’s capacity to assess student learning outcomes has been strengthened, as measured by the number of cycles of preschool, primary, and secondary education levels with student learning outcomes analyzed (Revised PDO Indicator 1). After first piloting student learning assessments, MINEDU assesses, analyzes, and reports, at the regional and school level, student learning outcomes at the end of Cycle II (Grade 2, which MINEDU had already been implementing), Cycle IV (Grade 4), Cycle V (Grade 6) and Cycle VI (Grade 8) (PDO 1, Intermediate Results Indicators [IRI 1 and IRI 2]). Grade 4 and Grade 8 assessments are being repeated every year, and in 2016, MINEDU for the first time carried out three census-based assessments. The reports were distributed to regional and local authorities, schools, and parents. The Project had initially contemplated expanding the assessment to Grade 11 (the last grade of basic education) as well and had piloted a set of instruments based on a framework similar to that of the earlier grades. Nevertheless, the target (for the original PDO Indicator 1) had to be modified (to the revised PDO Indicator 1) because of a major reform to the higher education sector, and based on evolving information needs, MINEDU is rethinking the potential uses for and scope of an assessment in the last year of secondary education. The development of a suitable framework is under way, and the design and piloting of new instrument are scheduled for 2018. A first survey was carried out to evaluate student development progress and the quality of preschool education services was carried out in 2014 using the Early Development Instrument and the Early Childhood Environment Rating Scale; this was followed by a survey using the Measuring Early Learning Quality and Outcomes (MELQO) in 2017 (IRI 4). The report is scheduled to be completed in October 2018 and MINEDU intends to use the findings of these evaluations to revise the early learning curriculum and to feed into training for preprimary teachers. MINEDU also participated in several international assessments of student learning outcomes (IRI 5). In Page 14 of 72 The World Bank Basic Education Project ( P123151 ) 2013, students in Grades 3 and 6 participated in the Third Regional Comparative and Explicative Study (Tercer Estudio Regional Comparative y Explicativo, TERCE) in mathematics, Reading and Writing, and Natural Sciences (Grade 6 only) to assess their learning in these subjects. This was followed by participation of 15-year-old students in PISA, and a PISA pilot in 2017. Further, for the first time, MINEDU participated in the ICCS to evaluate how well students are prepared to meet their obligations as citizens. MINEDU will participate in PISA 2018 and is planning to implement the census evaluations to Grades 4 and 8 in 2018, and a sample-based assessment to Grade 2 as well. The first multiyear assessment plan elaborated by MINEDU to be formally approved includes the assessment of Grade 11 in the coming years. Improve MINEDU’s Capacity to Evaluate Instructional Practice in Basic Education • PDO Indicator 3 (revised to IO Indicator 7): Annual Results of national studies of classroom instructional practice reported at the national level; Target: YES • Revised PDO Indicator 2: Instructional practices analyzed; Target: YES – Achieved 100% 24. This objective was achieved. MINEDU’s strengthened capacity to monitor instructional practice is reflected in the publication of a comprehensive report analyzing pedagogical practices over the course of four national studies of classroom instructional practice (PDO 2). Another study was prepared to analyze pedagogical practices from the data produced by the performance evaluation of Preschool teachers conducted by MINEDU in 2017. MINEDU used the Stanford Research Institute Classroom Observation System (Stallings Classroom Snapshot Instrument) to measure teachers’ use of instructional time in primary (2012 and 2014) and secondary (2013 and 2015) education (IRI 6). The findings were analyzed and distributed to MINEDU’s Regional Education Directorates (IRI 7). With support of the Project, MINEDU assigned new priority to implementing the Teacher Reform Law adopted in 2012 (paragraph 19). MINEDU developed the evaluation instruments and processes to make all teacher recruitment and promotion merit-based and initiated a process of regular performance evaluation and feedback for all teachers (IRI 9). The first round of performance evaluations was delayed so that appropriate risk mitigation measures could be implemented, including consultation and outreach to teachers, unions, and local authorities and training and certification of selected school principals to lead the evaluation of teachers in their schools. Evaluations began with preschool teachers in 2017 and the results were reported in January 2018 (IRI 9). This was the most sensitive of all teacher evaluations, and was achieved despite strong opposition from important factions of the Teacher Union, who held a strike for several months demanding, among other things, the suspension of the performance evaluation. MINEDU handled the crisis, which led to the resignation of a minister, without compromising on the evaluation, offering information training for all participant teachers across the country, disseminating the rubrics and instruments, and meeting with teacher representatives as needed to explain the entire process. Evaluations follow a process whereby those teachers that do not perform well on the classroom observation processes can participate in an extra evaluation following participation in a professional development program. MINEDU also designed, piloted, and implemented instruments for recruitment and merit-based promotion (IRI 10). The instrument for recruitment was implemented in 2015 and 2017 and resulted in the recruitment of 19,069 teachers out of a total 400,423 candidates in both years. The instrument for merit-based promotion was implemented in 2015, 2016, and 2017, and resulted in the promotion of 49,763, out of 410,429 teachers evaluated. For 2018, there are currently six evaluations in place: the second part of the performance evaluation for preschool teachers, the first performance evaluation of school’s principals, selection of teachers into the teaching career, and evaluation for Page 15 of 72 The World Bank Basic Education Project ( P123151 ) promotion within the career, in addition to evaluations for access to school management and local specialist positions. This is all a positive sign of sustainability and institutional strengthening. Improve MINEDU’s Capacity to Evaluate School Leadership in Basic Education • PDO Indicator 4: Number of school management positions staffed through the school principal evaluation system; Target: 25,000 • Revised PDO Indicator 3: School management positions staffed through the school principal evaluation system; Target: 25,000, revised to 18,000 in second project restructuring – Achieved 104 percent 25. This objective was overachieved, based on the revised PDO Indicator 3. MINEDU’s strengthened capacity to evaluate school leadership resulted in 18,477 school management positions staffed through the competitive evaluation of over 53,699 candidates (PDO 3). In 2014, managerial incumbents were evaluated to retain their positions; in 2014 and 2016, high-ranking teachers (the fourth level in the teaching scale) that sought leadership positions were evaluated. The principal evaluation system linked to a competitive selection process is in place, supported by related software. It includes a centralized phase whereby a unified national exam is given to candidates and a decentralized phase whereby an instrument to assess their professional growth is assessed (IRI 8). Nevertheless, the original target for staffing of school managers was reduced from 25,000 to 18,000, due to factors beyond MINEDU’s control. With the first two rounds of school principal selection processes in 2014, 15,461 positions were staffed after evaluating 51,563 candidates for 20,000 positions. In the selection process carried out in 2016, 12,562 positions were open and a total of 13,421 candidates were evaluated, of which 9,113 were qualified for appointment. Yet, in January 2017, only 3,016 positions were staffed by the Local Education Management Units due to the following unforeseen issues: (a) the results of the evaluation and verification requirements; (b) an imbalance in the demand of specific positions, some of which concentrated several applicants while others had none; and (c) legal disputes that left several positions ‘unavailable’ after being publicly announced as ‘available for competitive placing.’ Thus, the indicator could be disaggregated in two: (a) number of qualified candidates selected for opened positions, which would be 24,584 so far (3,028 + 12,443 + 9,113) and (b) number of those candidates accepting the offers, which is the originally reported number of 18,447 so far. Furthermore, MINEDU has already launched a third selection process to staff 9,000 additional school management positions, and about 50,000 teachers have registered to apply. The process will be completed in 2018, and with this, the original target of 25,000 is expected to be met or even exceeded. In addition, MINEDU has initiated the first performance evaluation of school principals (those selected in 2014). Again, there are strong signs of strengthened institutional capacity and sustainability, as the same directorate oversees these two processes and the three teacher evaluations (see above) in the same year, something never done before. 26. To strengthen the skills of selected school managers, MINEDU implemented and closely monitored the design and delivery of a school leadership training program carried out by qualified training institutions. The training program in school management went beyond the originally planned 4-month induction to incorporate two additional phases: a one-year certification (Diploma) and a one-semester complementary second degree. A total of 13,536 of the 15,461 of the teachers that were selected to managerial positions in 2014 have received diplomas in school management (IRI 11). Page 16 of 72 The World Bank Basic Education Project ( P123151 ) Justification of Overall Efficacy Rating 27. Overall efficacy is rated High, based on High efficacy before the first restructuring, Substantial efficacy after the first restructuring and before the second restructuring, and High efficacy after the second restructuring until completion (tables 2 and 3). C. EFFICIENCY 28. The economic analysis prepared at appraisal considered only the impact of the Project’s largest component—actions to expand and deepen the student learning and evaluation system. The analysis was conservative because it did not consider potential benefits from improvements in education quality and results linked to other components. It concluded that the Project’s full costs could be economically justified by the benefit stream of Component 1 alone, with an internal rate of return (IRR) of 10.1 percent and a positive net present value (NPV) equal to US$417 million. 29. The Project’s ex post economic analysis indicates an IRR between 10.7 percent and 13.8 percent (annex 4). 9 The lower bound of the range for this indicator is based on conservative estimates of the expected impact of the largest component of the Project—scaling up student’s evaluations for basic education students. They do not consider the benefits from results linked to the Project’s other components, such as evaluating instructional practice and actions oriented to improve the managerial performance of school’s principals, due to limited evidence of the impact for these types of interventions on student achievement in Latin America and Caribbean. However, research analyzing the eight top school systems in the world, on the one hand, and the U.S. school system, on the other, emphasize that the quality of teachers is the single most important factor for learning outcomes (Hanushek 2011). 10 Also, recent evidence shows that better school management systems have a significant impact on student performance (Bloom et al. 2014). 11 Given this, it is expected that the Project’s Component 2 (teacher evaluations within a merit-based career and competitive selection and training of school principals) would have at least a similar impact on cognitive skills as that of Component 1. Assuming the impact of the multiple intervention package (scaling up students’ evaluations, evaluating instructional practice, and improving managerial performance) on cognitive skills is double that of Component 1 of the project, the IRR might increase up to 13.8 percent. 12 30. The Project will affect education outcomes by improving quality of service delivery and student learning through (a) more abundant and robust comparative information on student learning results that will increase individual schools’ and regions’ accountability for education performance and (b) better information on student learning progress will aid in designing and implementing more effective and better-targeted education policies. 9 The NPV of the project ranges between US$339 million and US$1.35 billion. 10 Barber, M., and M. Mourshed. 2007. How the World’s Best-Performing School Systems Come Out on Top. London: McKinsey. http://mckinseyonsociety.com/downloads/reports/Education/Worlds _School_Systems_Final.pdf.. 11 Romero, M., J. Sandefur, and W.A. Sandholtz. 2017. Can a Public-Private Partnership Improve Liberia’s Schools? Center for Global Development. https://www.cgdev.org/sites/default/files/can-public-private-partnership-improve-liberias-schools.pdf. 12 E. Schiefelbein, L. Wolff, and P. Schiefelbein. 1998. “Cost-Effectiveness of Education Policies in Latin America: A Survey of Expert Opinion.” This assumption is also in line where experts assign an estimated increase in achievement that doubles that consisting of only testing for Grade 4 students. Page 17 of 72 The World Bank Basic Education Project ( P123151 ) 31. The economic analysis rests on the assessment that the Project increased MINEDU’s success and impact in implementing the evaluations for basic education students, evaluating instructional practice, and improving managerial performance. To the extent that these interventions have been recently applied and there is still no formal evidence on their results, this assessment makes use of meta-analysis that estimates an average effect of these types of interventions on student achievement, based on diverse country experiences. Also, it is expected that improved learning performance will translate into positive payback to participants in terms of increased earnings as they enter the labor market. Furthermore, it will also provide a positive payback to society in terms of overall economic gains and likely improvements in living standards resulting from such gains. 32. The Project’s design and implementation was efficient. Despite initial delays in loan effectiveness, the GoP proceeded to implement the Project with budget funding, and these delays did not have an impact on implementation of activities in the Project as approved. Project design allowed for an extra year before closure to report on implemented activities, which proved useful also to complete additional activities that had not originally been contemplated, including an extended specialized training for selected school principals, three new teacher evaluation subcomponents, and the borrower’s participation in the ICCS. The results-based financing (RBF) mechanism, built upon a carefully designed RF, was instrumental in ensuring that activities were completed in a coordinated manner to produce the outputs needed to achieve PDO objectives. In fact, MINEDU mostly complied with the scheduled indicators in advance of the estimated dates. The Project was completed in March 2018, as originally scheduled, without any closing date extension. The final allocation of loan proceeds, in comparison to the allocation estimated at appraisal is presented in table 1. 33. Further, the Project’s overhead costs were low. Designed as a SWAp, the Project’s various activities were implemented by the responsible units in MINEDU, supported by consultants as needed. The only project management costs related to the coordination provided by two central agencies, one responsible for technical coordination and the other for administrative aspect. In all, the costs of these two centralized agencies were very reasonable, amounting to only about US$1.9 million, or 1.04 percent of the total project cost of US$186 million; that is, half of the cost projected at appraisal. Table 1. Estimated Versus Actual Total Project Costs Estimated at Actual Appraisal Component US$, US$, % % million million 1. Evaluating Student Learning 148.316 81 127.712 69 2. Evaluating Instructional Practice and School Leadership 31.127 17 56.418 30 3. Strengthening MINEDU’s Implementation Capacity 3.642 2 1.939 1 Contingencies 13.036 7 — 0 TOTAL PROJECT COSTS 183.000 100 186.000 100 D. JUSTIFICATION OF OVERALL OUTCOME RATING 34. The Project’s implementation was never at risk. The Project’s Implementation Status and Results Reports consistently rated Development Objectives as Satisfactory throughout, and Implementation Page 18 of 72 The World Bank Basic Education Project ( P123151 ) Progress as Satisfactory or Moderately Satisfactory in the period leading up to the first project restructuring. The ICR Guidelines require that when the scope of a project has been narrowed, a split rating is required. While the project restructurings actually expanded, the Project’s scope in several areas (section IB), the actual target for PDO Indicator 3 (School management positions staffed through the school principal evaluation system) was reduced from 25,000 to 18,000 in the June 2017 restructuring. In view of this performance, the outcome for PDO 3 is rated Substantial for the second phase (until the second restructuring was approved). Resources allocated to achieving this PDO (30 percent) were less than half of those allocated to achieving PDO 1 (68 percent). However, more importantly, the outcome for the two other components that were equally or even more important for achieving the PDO were High, the outcome for Phase 2 is still High. Therefore, a split rating has been prepared, as presented in tables 2 and 3. Based on this split evaluation, the Project’s overall outcome is rated Highly Satisfactory. In retrospect and counting upon the Project’s continued high relevance and the high ex post efficiency analysis, and, more importantly, with the now available results of the continuation of project activities beyond the closing date, the final outcome rating is justified. While Peru’s results in the 2015 PISA improved in all areas—science, mathematics, and reading—it would not be appropriate to attribute these results to activities supported by the Project. The next PISA is scheduled for 2018 and it is expected that this trend toward improvement in achievement and improved outcomes will be continued (and measured), especially in view of the Project’s support for the full implementation of the Teacher Reform Law, an activity not envisaged when the Project was approved. Table 2. Outcome Rating Before and After Project Restructurings Phase 1 Phase 2 Phase 3 (July 1, 2013– (September 17, 2015– (June 07, 2017–March September 17, 2015) June 07, 2017) 31, 2018) Relevance of Objectives High Efficacy High High High (1) PDO 1 - Evaluate Student High High High Learning (2) PDO 2 - Evaluate High High High Instructional Practice (3) PDO 3 - Evaluate School High Substantial High Leadership Efficiency High Overall Outcome H/H/H= H/H/H= H/H/H= Highly Satisfactory (6) Highly Satisfactory (6) Highly Satisfactory (6) Table 3. Weighted Project Outcome Rating Phase 1 Phase 2 Phase 3 Total 1 Rating HS HS HS 2 Rating value 6 6 6 3 Total disbursed (US$, million) 0 17.76 25 Total disbursed/final disbursed 4 0 71 100 amount (in %) 5 Weight value (2 X 4) 0 4.26 1.74 6 6 Final outcome rating 6=HS Note: 1 - HU = Highly Unsatisfactory; 2 - U = Unsatisfactory; 3 - MU = Moderately Unsatisfactory; 4 - MS = Moderately Satisfactory; 5 - S = Satisfactory; 6 – HS = Highly Satisfactory. Page 19 of 72 The World Bank Basic Education Project ( P123151 ) E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 35. The Project was targeted at the entire student population, all teachers and school leaders. Despite gender parity not being a corporate goal at appraisal, the project did have it in mind by producing gender- disaggregate data over time that could help inform, design, and assess gender-focused interventions. For the student assessment component, results were disaggregated by gender at the national, regional, and local level, and by urban/rural areas. This provided very useful policy information for addressing gender gaps in learning outcomes where they may remain. Information provided by the Project notes that national tests and in PISA, girls perform slightly higher than boys in reading and slightly lower in mathematics (this is true in Grades 2, 4, and 8), although differences tend to be small. In terms of career opportunities for teachers, MINEDU reports that the merit-based promotions and access to school management positions—supported through the Project—have increased the share of female teachers being promoted to higher levels in their career, including into management positions. From 2013 to 2018, the number of female teachers promoted or hired to school management positions increased by 26 percent, with the highest growth at the level of deputy director (107 percent). This growth exceeds that of male teachers promoted or hired into management positions, which was 21 percent and 26 percent, respectively during the same period. This is expected to benefit not only the winning candidates but also junior female colleagues who now can foresee a more promising professional path ahead, with more young girls seeing women in leadership positions and advancing their careers at higher rates than before. Further, the age distribution of all new school managers has fallen—56 percent of managers are above age 50 while 44 percent are between ages 40 and 49. Had this process not been carried out, and the managers in place in 2013 remained in their positions, 84 percent of all managers would now be over the age of 50. So, not only women recently hired or promoted, but all new managers are now offered the opportunity to progress in their careers, providing an incentive for good performance and ensuring sustainability of the existing cadre given their younger age distribution. Institutional Strengthening 36. The institutional strengthening of MINEDU was at the center of the PDO. Its impact in this area is described in section IIB. Mobilizing Private Sector Financing Not applicable. Poverty Reduction and Shared Prosperity 37. The ultimate beneficiaries of MINEDU’s strengthening will be the 5.7 million students in Peru’s public school system, which include children of the lower-income population, including those living in poverty. The Project did have a significant impact on ensuring that indigenous people (which are on average poorer than non-indigenous, 31 percent versus 18 percent) could fully benefit from the Project in a culturally appropriate manner through the implementation of technical assistance (section IVB). A study on potential biases in tests administered to indigenous students both in the indigenous language and Spanish as a second language was carried out, based upon which MINEDU improved the tests. Going beyond what was agreed, MINEDU distributed reading materials for children in four indigenous languages Page 20 of 72 The World Bank Basic Education Project ( P123151 ) and was preparing them in two more languages to promote reading among children in contexts where availability of such literature is rare or nonexistent. This was in addition to the distribution in 2016, of official classroom materials in 19 indigenous languages to 20,000 schools and the approval of the first National Bilingual Intercultural Education Plan for 2021. The plan was subject to prior consultations, as mandated by law, and includes a specific strategy to ensure that student assessment mechanisms are pertinent to intercultural bilingual education. The strategy consists of designing and implementing a set of methodological tools and instruments to adequately assess learning outcomes in schools where intercultural bilingual education is provided. MINEDU has, for several years, carried out communications campaigns on local radio and television in the four indigenous languages being assessed to inform parents and communities of the upcoming tests and their purpose, as well as when to expect results. It also produced videos in indigenous languages in which these campaigns are carried out and is currently preparing an even more comprehensive communications strategy targeting indigenous peoples. In the final area of technical assistance, contributing to the selection of school principals whose cultural and linguistic profiles are considered appropriate for the communities where they are working, MINEDU has conducted a major effort to evaluate teachers in their mastery of indigenous languages, providing them a certification in their level of proficiency for each language. This certification has been used as a prerequisite for appointing school principals to schools that serve indigenous communities. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 38. Government commitment. The Government was committed to improving student outcomes in basic education even before project preparation began, as reflected by the first phase of its PELA Strategy implemented between 2009–2011, targeted at preschool and Grades 1 and 2. It was fully aware of the country’s dire educational outcomes, especially in relation to the its income per capita level and educational investment efforts, measured as a percent of GDP. Supported to a great extent by the Project’s preparation, a new preliminary design for a subsequent phase of PELA was introduced in 2012 for the 2013–2016 period. Its objective was to raise student learning achievement in public schools, and it allocated approximately 89 percent of the total budget for basic education to its financing. Therefore, preparation counted upon the Government’s full commitment throughout. Currently, the third phase of PELA (2016-2018) is being implemented. 39. PELA’s RBF design. The Project’s RBB approach had been introduced by the MEF in 2008 to prioritize public resource allocation to comprehensive sector-wide strategies and activities, including PELA. PELA was comprehensive and organized around four products with system-wide budget activities, including (a) basic operating conditions for schools (infrastructure and personnel), (b) adequate curriculum and teaching, (c) adequate teaching materials, and (d) evaluation. An additional project component consisted of support for program management and piloting of targeted activities. The GoP’s budgetary framework linked the financing of these activities to expected results in terms of improvements in student learning. The Project financed a small, but nevertheless very important, subset of PELA. Following a SWAp, the Project was designed to cofinance selected activities within eligible budget activities. Responsibility for implementation of project activities was entrusted to the responsible units within MINEDU. The Project’s design followed the RBB approach of the Government’s program, whereby MINEDU would be expected to achieve an agreed set of DLIs that would trigger loan disbursements. Page 21 of 72 The World Bank Basic Education Project ( P123151 ) 40. Lessons of experience. The Government’s program and the Project’s design both considered several lessons of experience. The first phase of PELA had produced mixed results, especially because insufficient attention to rural, bilingual education, coupled with unequal, regressive education financing had led to widening social gaps. A causal analysis conducted by MINEDU identified as an additional factor contributing to low student achievement: the fact that education services were not tailored to address the specific needs of students in higher social vulnerability situations or with special conditions for learning. PELA’s subsequent phase addressed this issue directly. Further, the Project addressed lessons of experience of previous World Bank-financed projects in Peru and in other countries, as well. These included (a) selecting activities that were central to the Government’s strategy and supported by global evidence, (b) focusing on institutional capacity building, (c) supporting capacity development during preparation, (d) emphasizing the use of data for polity and accountability, and (e) building public awareness and confidence. The Project’s design (a) included activities related directly to evaluation of student learning, instructional practice, and school leadership, which evidence supported as being directly related to PELA’s objective; (b) provided technical support to build MINEDU’s institutional capacity in three key areas—student assessment, monitoring of teacher instructional practice and school directors’ leadership capacity, and monitoring of overall system progress; (c) counted upon significant capacity development through the World Bank Group’s program of non-lending technical assistance, trust funds, and South-South learning resources to deliver cutting edge knowledge on global best practice; (d) included activities to increase the potential of student assessment data to inform education policy, such as disseminating information, making it available to schools, parents, journalist, researchers, education administrators, and the general public; and (e) supported MINEDU in establishing technically sound systems and cleanly implementing the systems to measure student learning, school directors’ performance, and teacher practice, and disseminating the findings to the media, policy makers, education officials, school, and parents to help build public awareness and confidence in the results. 41. Consistency with World Bank Group Country Partnership Strategy. The Project was fully consistent with the World Bank Group’s Peru Country Partnership Strategy for FY12–FY16 (Report No. 66187-PE), considered by the Executive Directors on March 13, 2012. It contributed directly to the CPF’s first Strategic Objective—Increased access and quality of basic services for the poor—and, specifically, to its Results Area 1.3—Fostering an efficient and results-oriented education system. It built upon earlier World Bank- financed projects, including the Rural Education Project (P055232, Loan No. 7176-PE), and followed the then ongoing technical assistance to MINEDU and the National Education Council (Consejo Nacional de Educación) in the dissemination of its student test results, videoconferences, study tours, and workshops with international experts to learn from relevant experiences in the region, and was approved almost in parallel with an operation to strengthen accreditation and quality assurance systems for higher education. With this, the World Bank had developed a strong partnership with MINEDU, that made preparation a natural continuation of earlier and ongoing assistance. B. KEY FACTORS DURING IMPLEMENTATION 42. The US$25 million World Bank loan for the BEP was approved on January 17, 2013, signed on July 1, 2013 and became effective on May 20, 2014. The Project was never considered to be at risk. MINEDU and the World Bank implementation support team worked harmoniously throughout to address issues and priorities and required modifications to the Project and its legal and administrative framework. Page 22 of 72 The World Bank Basic Education Project ( P123151 ) 43. Early implementation. The extended period between loan approval and effectiveness was the result of delays in approving the Project Operational Manual. While this delay affected disbursements from the World Bank loan, it did not affect the Project’s implementation, especially because the allocation of loan funding to different activities was not relevant as the loan was structured as a SWAp. During this period, MINEDU began implementation of several key project activities with budget funding following GoP procurement rules (section IVB, Environmental, Social, and Fiduciary Compliance), and more importantly, with the World Bank’s technical support. In fact, by the time the loan became effective, MINEDU had made important progress in its capacity to evaluate student learning. In addition to administering the second grade national assessment that it had carried out since 2007, MINEDU had already extended the student assessment in mathematics and reading to an additional grade (Grade 6) through a sample evaluation in 2013, following a pilot assessment at that level. It had also conducted a pilot for the assessment of Grade 11, and distributed reports on the 2012 and 2013 assessments to regional and local authorities, schools, and parents. National studies to monitor instructional practice in primary and secondary education had been implemented since 2012, and the first report disseminated. Further, the first evaluation for the competitive selection of school principals had been carried out, through which 2,900 school principal positions had been staffed. In sum, the Project was off to a strong start and was well on its way to meeting the targets of several PDO and IO indicators. 44. Institutional changes. In August 2013, MINEDU requested a change in the Project Technical Coordination Unit. While the Project was originally to be implemented through existing technical and administrative line units, the technical management and coordination of its activities would be the responsibility of a Technical Coordination Team within the ministry’s Learning Quality Measurement Office (Oficina de Medición de la Calidad de los Aprendizajes, UMC). The change would transfer the Project’s coordination to the SPE. Unlike the UMC, the SPE was hierarchically above all the directorates implementing project activities, which would facilitate its coordination. After confirmation of this request by MINEDU’s new leadership that took office in 2013, this was reflected in the 2015 restructuring, that also included changes to the project description and RF reflecting the new leadership’s priorities. 45. Government commitment. The new Minister of Education who assumed office in October 2013 gave high priority not only to the activities under the SWAp, but to implementing the far-reaching teacher policy reform adopted in late 2012 under the Teacher Reform Law (section IA). All the elements of this reform had been included as part of the Project’s original design, but were dropped during negotiations due to concerns at the highest levels that associating external support to the introduction of merit-based processes for the recruitment, promotion, and regular performance evaluation of Peru’s 25,000 teachers would only exacerbate teachers’ resistance to the measure. Instead, the Project concentrated on processes for the selection of school directors and the ministry’s capacity to carry out regular observation of teachers’ classroom practice. With a renewed determination to implement the Teacher Reform Law, MINEDU requested that the Project be expanded to include financing for the design of the new evaluation instruments and processes required to make all new teacher recruitment and promotion merit-based, and to initiate a process of regular performance evaluation and feedback for all teachers. The Government’s commitment to the Project and its objectives were unwavering throughout implementation, despite three changes at the ministerial level in MINEDU. 46. Suspension of school activities. In 2017, school activities were suspended in seven of Peru’s regions, in some cases for up to one and a half months, due to the effects of El Niño. This was compounded by a national teachers’ strike from June to September of the same year that affected the carrying out of Page 23 of 72 The World Bank Basic Education Project ( P123151 ) teachers’ evaluations and cancelation of the student assessment that was scheduled for November. The teachers’ strike was to demand, among other things, that the teachers’ evaluation that was scheduled for preschool teachers be cancelled. Nevertheless, the evaluation went forward, and the strike served to highlight the subject of evaluation at the policy level and mobilize popular support for it. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 47. The UMC (later the SPE) had primary responsibility for tracking progress related to the Project’s outcomes and results, based on inputs gathered from the different implementing units within MINEDU. The SPE was to prepare project reports on the progress made under the Project’s different components, and progress toward targets established in the RF. The Project was designed as a SWAp, that would co- finance selected activities within eligible budget activities included in the Government’s PELA under an RBF framework. The World Bank’s Specific Investment Loan would disburse against compliance with a set of DLIs. The defined DLIs were fully compatible with the Project’s RF: four DLIs were PDO indicators and the other four were key IO indicators, and disbursements against all of these were sequenced to match the Project’s implementation time table to keep activities on schedule and to trigger disbursements in each year there was a target to meet. Project design contemplated two technical reviews of progress toward compliance with DLIs that were to be carried out by an independent verification agency. The indicators selected to monitor progress and trigger disbursements were the right indicators to measure its impact. Nevertheless, section IIB disaggregates indicators with additional evidence that has now become available. 48. Further, the Project, by design, supported the establishing of monitoring and assessment systems (for example, student assessment, teacher and principal evaluation systems, and so on), considered fundamental toward the broader outcome of informing policy decisions, school management, and teaching (section IIB). M&E Implementation and Utilization 49. M&E was implemented routinely, and without issues, and the structure of the Project’s RF and DLIs provided a well-defined yet flexible framework with which to monitor implementation progress and disburse. MINEDU found the structure of the monitoring reports to be concise and simple, with well- defined indicators, which facilitated the process of both monitoring and disbursement. The implementation teams adjusted the RF dynamically throughout implementation to perfect the definition and structure of indicators in the Project’s results chain, to introduce new indicators to reflect MINEDU’s request for additional subcomponents and activities, and to reflect progress and expectations of future implementation. The Project’s restructuring in September 2015 introduced several changes to the RF to better focus on outcomes and to measure all the key steps critical to achieving the PDO (section IB). It also introduced modifications to the DLIs to assign an equal value to all DLIs, with a unit value determined based on the DLIs’ final targets. It also introduced flexibility to allow the hiring of a firm or an individual to verify compliance of indicators against protocols (section IB). Further adjustments to the Project’s RF and to corresponding DLIs were introduced in the June 2017 restructuring, but these were mostly to adjust Page 24 of 72 The World Bank Basic Education Project ( P123151 ) targets to reflect implementation progress. M&E played a central role in the Project’s implementation, as it was used to trigger disbursements upon compliance with DLIs and also to verify compliance by an independent external entity. Justification of Overall Rating of Quality of M&E 50. The quality of overall M&E is rated High, based on the solid design of the RF, its continuous adjustment to reflect implementation progress and new priorities, and that it allowed a results-based implementation support, including for disbursement purposes. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 51. The Project was implemented by MINEDU, through its existing technical and administrative units. Overall technical management and coordination was provided first by the UMC, and following restructuring, through SPE (section IIIB, Key Factors During Implementation). The existing Executing Unit 118: Improving Preschool Education Program within MINEDU, later renamed as Executing Unit 118: Improving Quality of Basic Education through a dedicated SWAp Administrative Coordination Team, was responsible for the Project’s fiduciary management. The Project Operational Manual, adjusted throughout implementation to reflect modifications to the Project, included detailed guidelines for the implementation of the Project’s components, M&E, procurement, financial management (FM), and safeguards. 52. Safeguards. The Project was classified as Category C for environmental purposes, because it did not include construction or rehabilitation of infrastructure. However, the Indigenous Peoples Policy (OP/BP 4.10) was triggered because the SWAp would apply across all public schools in Peru. During preparation, a Social Assessment (SA) and Indigenous Peoples Planning Framework were prepared, consulted, and disclosed, as required. The SA identified three areas where further technical assistance was required to ensure that indigenous people could fully benefit from the Project in a culturally appropriate manner while at the same time addressing potential adverse risks: (a) minimizing biases in standardized tests administered to indigenous students whose native language is not Spanish and whose culture is different from the mainstream; (b ) ensuring indigenous families and communities have timely access to meaningful, culturally pertinent reports of the results to be informed about their children’s learning outcomes and how to support them; and (c) contributing to the selection of school principals whose cultural and linguistic profiles are considered appropriate for the communities where they are working. The Project carried out each of these activities, going well beyond original plans (section IIE). 53. Fiduciary. The Project’s fiduciary matters were handled by UE 118 (Executing Unit 118 ‘Improving Quality of Basic Education’), a unit with ample experience in the administration of World Bank—and other multilateral-financed loans, and compliance with World Bank FM and procurement policies and requirements was satisfactory, with risk mitigation measures established. Still, the Project’s FM risk was considered Substantial as its design included several components and subcomponents implemented by different directorates within MINEDU, and the RBF approach presented challenges from the FM perspective. During preparation, specific arrangements and strengthening measures were agreed to strengthen MINEDU’s FM capacity, including, using Peru’s well-functioning systems for budget formulation, execution, and reporting through Peru’s SIAF, and recruiting and strengthening FM and technical teams. Further, the Project’s legal agreement required MINEDU to contract an independent Page 25 of 72 The World Bank Basic Education Project ( P123151 ) verification agency 13 to carry out technical reviews of agreed results during implementation, particularly focused on the compliance with the DLIs. Given the delay in effectiveness, MINEDU began implementing project activities using budget funding and following local procurement legislation. As a result, the World Bank could not disburse the amounts related to DLIs that had been complied with because there were no eligible expenditures to support disbursements against the DLIs. The World Bank’s implementation support team worked with the Government to identify expenditures that could be considered eligible. With this, disbursements trailed implementation and compliance with DLIs, but rapidly caught up to appraisal estimates by the end of 2016. The management of procurement processes by MINEDU was overall Satisfactory. Overall, application of the World Bank’s procurement policies and procedures facilitated contracting of consultants, but placed a high burden on the implementation support team to review each and every phase of contracting numerous consultants. Three ex post procurement reviews were performed in June 2015, May 2016, and May 2018. The first two ex post reports and the Supervision Mission Aide Memoires included recommendations which were shared with the Project, to improve the management of procurement processes. These recommendations were included in the action plans of the ex post reports which were, in general, complied with during Project implementation. C. BANK PERFORMANCE Quality at Entry 54. The World Bank’s performance in ensuring Quality at Entry is rated Satisfactory. As highlighted by MINEDU in its Completion Report, the World Bank’s team played a central role in the Project’s preparation. The World Bank’s task manager had been involved in the preparation and supervision of several projects implemented by MINEDU (by earlier administrations), and, as a result, had a clear definition of the ministry’s plans and requirements which were reflected in the PDOs. The team applied lessons of cross-sectoral experience: the World Bank-financed Results in Nutrition for Juntos SWAp (P117310) was the first RBF project in Peru, and lessons from its implementation fed into the Project’s design, including having MINEDU’s units manage implementation directly. The RF was concise, and the DLIs were well-integrated into the PDO indicators and IO indicators. Procedures for reporting and claiming disbursements were easy to comply with. The team also had an important role in bringing technical expertise into the project design, using specialized instruments, especially in innovative areas such as the evaluation of preschool education and studies in classroom practice. The Project was reasonable in terms of the activities it included and the objectives which it strived to achieve, within its given implementation period. The team built flexibility into the implementation schedule that became useful when additional subcomponents were included to support implementation of the Teacher Reform Law (the Project’s completion date was extended by one year, but did not require a closing date extension given the leeway in the design of its implementation schedule). The Project was ready for implementation upon approval, so much so that MINEDU began implementing activities with its own resources. Were it not for the requirement that the Operational Manual be approved, the loan could have become effective earlier to support initial disbursements. 13 The requirement to contract an independent verification agency by three months after effectiveness was modified in the 2015 Project Restructuring to ‘contract an Independent Verification Agent no later than three months after restructuring,’ to introduce flexibility to allow hiring of a firm or an individual. Page 26 of 72 The World Bank Basic Education Project ( P123151 ) Quality of Supervision 55. The World Bank’s Quality of Supervision is rated Satisfactory. Throughout implementation, the World Bank’s implementation support team, led by the same country office-based task manager who was responsible for preparation, maintained open, fluid, and continuous dialogue with the teams in MINEDU. The team was pragmatic, results-focused, effective, and constantly on top of emerging issues. When the World Bank’s loan became effective, and it became clear that MINEDU had begun implementation following local procurement legislation (section IVB), the team worked closely with the client to identify alternatives for recognizing these for financing as far as possible, and then reprogramming scheduled disbursements. The team was proactive in working with MINEDU to reflect institutional changes and MINEDU’s new priorities (for example, the teacher evaluation subcomponents) in the Project, and proactively adjusted its RF both to improve it to reflect outcomes, to reflect new priorities and activities that expanded the Project’s scope, and continuously to adjust targets to reflect implementation progress. Project restructurings, often delayed in many projects, were processed expeditiously. The midterm review in June 2016 was held on time, and comprehensively, including a thorough review of the Project’s progress in addressing the three areas where further technical assistance was required to ensure that indigenous people could fully benefit from the Project in a culturally appropriate manner. The Project’s Implementation Status and Results Reports were timely, clear, and results-focused and provided a strong basis for understanding the issues that the Project’s implementation was facing and the progress it was making toward its PDO. D. RISK TO DEVELOPMENT OUTCOME 56. Risk to Development Outcome is considered Modest. It is unquestionable that the Project helped establish and implement important tools that have strengthened MINEDU’s capacity in the three areas covered by its PDO. Some of these, such as the instruments to implement the Teacher Reform Law in areas of contracting, evaluation, and promotion are anchored in legislation. While it is not inconceivable that there may be further protests against the implementation of these instruments, the Government has shown thus far that it has the wherewithal to hold the ground. Similarly, the program for training of school managers has been put in practice, with the two programs formalized for a managerial path. Student assessments have been carried out, with several implemented concurrently, and participation in international assessments has started a second cycle of implementation. MINEDU is developing a Plan de Evaluaciones to cement future plans for student assessments at different educational levels. As a SWAp, the instruments and processes for all these achievements were carried out by staff of MINEDU, and therefore internalized in the ministry’s own procedures. The capacity to sustain these is in place. The main risks to the development outcome relate to the political will to adhere to the standards and continue to apply the instruments developed under the Project, and related to this, that MINEDU will be provided adequate budget and other resources to do so in the future. So far, this has been the case and MINEDU continues to perfect and implement the instruments it developed under the Project. The World Bank is supporting further reforms in primary and secondary education under the ongoing Boosting Human Capital and Productivity Development Policy Financing (DPF) with a Deferred Drawdown Option (DDO). 14 The DPF-DDO aims to support measures targeting key productivity constraints by enabling, among others, the improvement of the quality of human capital through the following expected results that directly 14World Bank, Program Document, Peru Boosting Human Capital and Productivity Development Policy Financing with Deferred Drawdown Option, Report No. 99251-PE, dated January 16, 2016. Page 27 of 72 The World Bank Basic Education Project ( P123151 ) reinforce the Project’s objectives: (a) an increased share of programmed merit-based promotions and appointments for teachers and school principals with the teaching career outside Lima and Callao from 60 percent in 2014 to 75 percent in 2017, (b) increase the number of secondary schools that implement the new school-day model from 0 in 2014 to 1,500 in 2017, and (c) include learning outcomes in the stated targets of the performance-based financial incentive mechanism for subnational entities from 0 percent in 2014 to 100 percent in 2017. V. LESSONS AND RECOMMENDATIONS 57. Projects that are designed within the framework of a Government’s institutional structure and budget and implemented by the teams normally responsible for the activities being financed, have an impact beyond that of the Project alone. The Project, designed as a SWAp, complemented Government funding with that of the World Bank and was implemented by the staff of MINEDU’s various units responsible for the respective activities. This not only leveraged the impact of the Project but ensured a high degree of sustainability given the staff’s experience in carrying out project activities, with technical support from project-financed consultants. One comment made by the executing units that is of relevance to these types of operations is that it would be useful to have more clarity ex ante on the duration of each of the activities and processes to plan accordingly. This is especially important for projects implemented by separate units but whose activities depend on the outputs of other units. The other comment made by staff of executing units is that the annual project planning cycle should coincide with the Government’s yearly budget cycle. Although it did not in the end affect final outcomes, it did delay activities at the beginning of each calendar year. The Project was challenged by the different schedules of the Government’s budget cycle (preparation in April–May) and the preparation of the Project’s Annual Operating Plan (AOP, November–January). This posed additional demands on the Project’s administrative and technical units, who had to prepare a plan twice a year with either incomplete or dated information. To the extent possible, future projects should consider adapting preparation of their annual programming to coincide with that of budget preparation. 58. A borrower’s commitment to a project and its objectives is always an influential factor in its eventual outcome, but this commitment is even more imperative for implementation of difficult reforms, such as the reform of the teaching career, and the World Bank can be a trusted partner in the implementation of these difficult reforms. Although originally a part of the Project’s design, support for implementation of the Teacher Reform Law was dropped in the final stages of preparation given concerns by the Government regarding its implementation. Nevertheless, sensing both the need and opportunity for counting on expert advice financed by the Project, the Government later requested that it be included throughout the project restructuring. The process was not easy, as expected, and a national teachers’ strike affected the carrying out of the student evaluations and resulted in the cancelation of a student assessment that was scheduled for November. MINEDU delayed the teacher performance evaluation so that appropriate risk mitigation measures could be implemented, including consultation and outreach to teachers, unions, and local authorities and training and certification of selected school principals to lead the evaluation of teachers in their schools. However, given the Government’s strong commitment to implementation of the law, the evaluation went forward, and the strike served to highlight the subject of evaluation at the policy level, and mobilize popular support for it. Page 28 of 72 The World Bank Basic Education Project ( P123151 ) 59. World Bank loans should not include approval of Operational Manuals (or similar documents required for implementation) as a condition of effectiveness. Given characteristics of the Project, designed as a SWAp where funding for individual activities was not required to begin implementation, the delay in approval of the Operational Manual did not affect implementation. Nevertheless, had this not been the case, implementation could have suffered initially given the cumbersome, bureaucratic procedures for the manual’s approval that were made even more difficult given the changes in administrations. This lesson was internalized by the World Bank’s team during implementation when needed revisions to the Operational Manual were not required as a condition of the amendment’s effectiveness. 60. Timely project restructurings not only facilitate implementation, but also have an important impact on final outcomes. This lesson is especially relevant for projects at risk, although the Project was never officially at risk. 15 Nevertheless, the timeliness of the Project’s restructurings was especially important to maintain its relevance to the borrower’s priorities (that is, the first restructuring that introduced new activities related to the Teacher Reform Law) and to adjust targets of number of teachers, directors, and so on to implementation progress. Although good estimates were available at appraisal, actual implementation can have an impact on progress toward targets, and the early restructurings, helped ensure required adjustments to reflect implementation experience, and very positive outcomes based on defined indicators and targets. 61. Projects that have qualified staffing as an outcome can be affected by several factors, and constant adjustment of plans may be required to ensure sustainability. Under the Project, staffing of qualified school directors was affected by availability of interested, qualified staff to be evaluated, as well as to take the available most remote posts. Because many interested directors had been qualified in the first round, it became difficult to qualify others in the final year (and the target was adjusted). In addition, it was difficult to recruit directors (and teachers) to rural areas, and some thought may be warranted as to possible additional incentives that will compensate for this. Finally, staffing normally involves issues regarding staff age and imminent retirement of existing qualified directors. This issue was identified by MINEDU, and, to capitalize on the Project’s investment, the ministry is considering a program of using new retirees (who underwent full training) either in a program of training the trainers or as mentors or resource teachers to new directors. 62. To be successful, there are choices to be made when starting a process of reform of teachers and directors. For teachers, there was initially a tendency to be lenient in evaluations, and standards were not set impossibly high. This is because there is no alternative market for teachers, so the Project needed a critical mass of qualified teachers on board. The standards were set so that a majority could pass the evaluations, but information would be gathered as to training needs to be addressed. Also, for the evaluation process, it was important to disseminate the process, make everything public and transparent to avoid perceptions of a black market. It was helpful that the criteria were disseminated in advance, and the process was done in progression. Further, the Project carried out a successful communications campaign regarding the evaluation process that was geared to the population to secure citizens’ support. 15 Although the Project was never rated Moderately Unsatisfactory or Unsatisfactory for Development Objectives or Implementation Performance in its Implementation Status and Results Reports, it was flagged for low disbursements in its initial implementation. Page 29 of 72 The World Bank Basic Education Project ( P123151 ) Finally, for parents’ input on teachers, the Project considered cultural differences in parents’ responses to inquiries to adjust for their different cultural values. . Page 30 of 72 The World Bank Basic Education Project ( P123151 ) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: The Project’s Development Objective was to improve the MINEDU’s capacity to evaluate student learning, instructional practice and school leadership in Basic Education. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDOI1. Cycles of Preschool, Number 0.00 4.00 4.00 4.00 Primary and Secondary Education levels with student 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 learning outcomes analyzed. Comments (achievements against targets): Targets: 2 in 2015 (e.g. Cycle II & V); 1 in 2016 (e.g. Cycle VI); and 1 in 2017 (e.g. Cycle IV). Achieved (100%): Cycle II, IV, V and VI. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDOI2. Instructional practices Yes/No N Y Y Y analyzed 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 Page 31 of 72 The World Bank Basic Education Project ( P123151 ) Comments (achievements against targets): Achieved (100%) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDOI3. School management Number 0.00 25000.00 18000.00 18477.00 positions staffed through the school principal evaluation 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 system. Comments (achievements against targets): Original Target 25,000; revised to 18,000 in 2nd project restructuring. Achieved (104%). 2018 projection: with 9,000 positions opened for recruitment and 50,0000 applicants registered, it is likely the original target of 25,000 will be also met and exceeded this year. A.2 Intermediate Results Indicators Component: Evaluating instructional practice and school leadership Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI6. National studies of Number 1.00 4.00 4.00 4.00 classroom instructional practice in Primary Education 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 and/or Secondary Education levels, as the case may be. Comments (achievements against targets): Target met (100%). Target: 2 (secondary) 2013 and 2015; 2 (primary) 2012 (baseline) and 2014. Page 32 of 72 The World Bank Basic Education Project ( P123151 ) Achieved: 2 primary; 2 secondary. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI7. Results of national Number 0.00 4.00 4.00 4.00 studies of classroom instructional practice 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 reported at the national level. Comments (achievements against targets): Achieved (100%) 1 in 2013; 2 in 2015; 1 in 2016 Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI8. School principal Text 0 National Framework School principal System in place evaluation system designed. for Good School evaluation system supported by software Leadership approved; designed. evaluation instruments, grading criteria and software designed and piloted; School Leadership Training Program designed Page 33 of 72 The World Bank Basic Education Project ( P123151 ) 31-Jul-2012 30-Sep-2015 31-Mar-2018 31-Mar-2018 Comments (achievements against targets): Achieved (100%) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI9. Teacher performance Text 0 0 Teacher performance Teacher performance evaluation system designed, evaluation system is evaluation system is implemented and results being implemented being implemented, reported. starting with preschool teachers. 31-Jul-2012 30-Sep-2015 31-Mar-2018 31-Mar-2018 Comments (achievements against targets): No original target (indicator added after restructuring). Achieved (100%). Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI10. Evaluation instruments Text 0 0 Evaluation Evaluation to determine entrance to instruments to instruments to and promotions within the determine entrance to determine entrance to Teaching Career designed and promotions and promotions within the Teaching within the Teaching Career designed Career designed, piloted and implemented. Page 34 of 72 The World Bank Basic Education Project ( P123151 ) 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 Comments (achievements against targets): No original target (indicator added after restructuring). Target exceeded: while the end target was "Evaluation instruments to determine entrance to and promotions within the Teaching Career designed," the Borrower has designed, piloted and used the instruments for actual recruitment and promotion of teachers nationwide. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI11. School principals Number 0.00 0.00 10000.00 13536.00 certified in school management. 31-Dec-2012 31-Mar-2018 29-Dec-2017 31-Mar-2018 Comments (achievements against targets): No original target (indicator added after restructuring). Exceeded (135%). Component: Evaluating Student Learning Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI1. Student assessments for Number 0.00 4.00 4.00 4.00 new grades of Primary Education and Secondary 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 Education levels validated in a pilot study. Comments (achievements against targets): Achieved (100%) Page 35 of 72 The World Bank Basic Education Project ( P123151 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI2. Additional grades at the Number 0.00 4.00 3.00 3.00 Primary Education and Secondary Education levels 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 in which sound national assessments are administered in at least two subject areas. Comments (achievements against targets): (e.g. 8th) in 2015; 1 (e.g. 4th) in 2016; 1 (e.g. 11th) in 2017. End-target revised from 4 to 3 total in second restructuring. Partially achieved original target; revised target 100% achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI3. Results of national Number 0.00 3.00 3.00 3.00 student assessments in new grades at the Primary 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 Education and Secondary Education levels reported at the regional and school level. Comments (achievements against targets): Achieved (100%). (e.g. 6th) in 2015; 1 (e.g. 8th) in 2016; 1 (e.g. 4th) in 2017. Page 36 of 72 The World Bank Basic Education Project ( P123151 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI4. National surveys carried Number 0.00 2.00 2.00 2.00 out to evaluate child development progress and 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 the quality of public Preschool Education services. Comments (achievements against targets): Target achieved: 1 in 2014; 1 in 2017; Targets were reworded in the second project restructuring. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI5. Borrower’s (through Text Last participation in TERCE, PISA, and ICCS TERCE, PISA and ICCS TERCE, PISA and ICCS. MINEDU) participation in PISA 2012 End target met. international assessments of Pilot TERCE student learning outcomes. 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 Comments (achievements against targets): Exceeded: TERCE 2013; Pilot PISA 2014; PISA and ICCS 2015. Unlinked Indicators Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 37 of 72 The World Bank Basic Education Project ( P123151 ) Target Completion Direct project beneficiaries Number 0.00 0.00 5.70 5.70 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 Female beneficiaries Percentage 0.00 0.00 50.00 50.00 31-Jul-2012 31-Mar-2018 31-Mar-2018 31-Mar-2018 Comments (achievements against targets): Achieved (100%) Page 38 of 72 The World Bank Basic Education Project ( P123151 ) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: Improve MINEDU’s capacity to evaluate student learning in Basic Education 1. Number of additional grades at the Primary Education and Secondary Education levels in which sound national assessments are Outcome Indicators administered in at least two subject areas (number); Revised to: Cycles of Preschool, Primary and Secondary Education levels with student learning outcomes analyzed (number). 1. Yearly assessments for selected grades of Primary and Secondary Education levels validated in a pilot study (YES/NO); Revised to the following three indicators: (a) Student assessment for new grades of Primary and Secondary Education levels validated in a pilot study (number); (b) Additional grades at the Primary Education and Secondary Education levels in which sound national assessments are administered in at least two subject areas (number); (c) Results of national student assessments in new grades at the Primary Education and Secondary Education levels reported at the regional and school Intermediate Results Indicators level (number). 2.Number of surveys carried out to evaluate child development progress and the quality of public Preschool Education services (number); Revised to: National surveys carried out to evaluate child development progress and the quality of public Preschool Education services (number). 3.Borrower’s (through MINEDU) participation in international assessments of student learning outcomes (text); Revised to: (target adjusted). Page 39 of 72 The World Bank Basic Education Project ( P123151 ) 1. Design, validation and administration of national assessments in new grades and subjects of primary and secondary education. 2. Results of national assessments in primary and secondary education reported at the national, regional, and school levels. 3. Design, validation, administration of national evaluations of quality Key Outputs by Component and outcomes of preschool education. (linked to the achievement of the Objective/Outcome 1) 4. Results of national evaluations of preschool education quality and outcomes reported at the national level. 5. Borrower’s participation on international assessments (PISA, TERCE, ICCS). 6. National results on international assessments reported at the national level. Objective/Outcome 2: Improve MINEDU’s capacity to evaluate instructional practice in Basic Education 1.Annual results of national studies of classroom instructional practice Outcome Indicators reported at the national level (YES/NO); Revised to: Instructional practices analyzed 1.Number of national studies carried out to monitor instructional practice in Primary and Secondary Education levels as the case may be (number, non-cumulative); Revised to the following two indicators: (a) National studies of classroom instructional practice in Primary and Secondary Education levels, as the case may be (number); and (b) Intermediate Results Indicators Results of national studies of classroom instructional practice reported at the national level (number) 2.New indicator: Teacher performance evaluation system designed, implemented and results reported (text) 3. New indicator: Evaluation instruments to determine entrance to and promotions within the Teaching Career designed (text) Key Outputs by Component 1. Teacher performance standards designed; proposal of evaluation (linked to the achievement of the Objective/Outcome 2) instruments defined; first round of regular performance evaluations Page 40 of 72 The World Bank Basic Education Project ( P123151 ) carried out; and first round of exceptional performance evaluations carried out. 2.Evaluation instruments to determine entrance to Teaching Career designed; evaluation instruments to determine promotion in Preschool designed; evaluation instruments to determine promotion in primary designed. Objective/Outcome 3: Improve MINEDU’s capacity to evaluate school leadership in Basic Education 1.Number of school management positions staffed through the school Outcome Indicators principal evaluation system (number) Revised to: School management positions staffed through the school principal evaluation system. 1.School principal evaluation system designed (text, non-cumulative) Intermediate Results Indicators 2.New indicator: School principals certified in school management (number). 1.National Framework for Good Leadership developed and approved. 2.Evaluation instruments, grading criteria, and software designed and Key Outputs by Component piloted. (linked to the achievement of the Objective/Outcome 2) 3.School leadership training programs designed and implemented. 4.Software to support the school principal evaluation system designed. Page 41 of 72 The World Bank Basic Education Project ( P123151 ) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Ines Kudo Senior Education Specialist and Task Team Leader Barbara Bruns Lead Education Specialist Patricia de la Fuente Hoyes Senior Financial Management Specialist Lourdes Linares Senior Financial Management Specialist Maria Virginia Hormazabal Financial Officer Ana Karina Rozas Operations Analyst Mariana Montiel Senior Counsel Jean Jacques Verdeaux Procurement Specialist Selene del Rocio la Vera Procurement Specialist Maria Elena Paz Gutzalenko Program Assistant Sara Burga Program Assistant Peter Lafere Social Development Specialist German Freire Social Development Specialist Raul Tolmos Environmental Specialist Supervision/ICR Patricia Alvarez, Ines Kudo Task Team Leader(s) Juan Carlos Martell Rivera Procurement Specialist Patricia De la Fuente Hoyes Financial Management Specialist Raul Tolmos Environmental Safeguards Specialist Maria Elena Paz Gutzalenko Program Assistant Sara Burga Program Assistant Carlos Tomas Perez-Brito Social Safeguards Specialist Luciana Beatriz Velarde Arrisueno Analyst Page 42 of 72 The World Bank Basic Education Project ( P123151 ) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY11 .100 48,712.87 FY12 58.142 410,574.24 FY13 53.830 210,589.40 FY14 0 2,906.56 FY15 2.091 13,057.15 FY16 0 9.25 FY18 0 3.63 Total 114.16 685,853.10 Supervision/ICR FY13 0 624.78 FY14 23.689 90,661.85 FY15 34.758 125,128.92 FY16 46.574 100,439.05 FY17 30.357 97,095.55 FY18 25.528 136,170.17 FY19 1.140 3,960.16 Total 162.05 554,080.48 Page 43 of 72 The World Bank Basic Education Project ( P123151 ) ANNEX 3. PROJECT COST BY COMPONENT Actual at Project Amount at Approval Percentage of Approval Components Closing (US$, (US$, millions) (%) millions) Component 1. Evaluating 20.24 10.90 80.96 Student Learning Component 2. Evaluating Instructional Practice and 4.25 13.87 17.00 School Leadership Component 3. Strengthening MINEDU’s Implementation 0.50 0.23 2.36 Capacity Total 25.00 25.00 100.00 Page 44 of 72 The World Bank Basic Education Project ( P123151 ) ANNEX 4. EFFICIENCY ANALYSIS 1. This annex focuses on the economic rationale, estimated costs, and projected benefits of the Project’s activities to strengthen MINEDU´s capacity to evaluate student learning. First, research evidence on educational quality and economic growth as well as impact of student learning assessments on learning outcomes was reviewed. Second, primary and secondary data were used to estimate the costs and make assumptions about the benefits of strengthening Peru’s national assessment system. Finally, a cost-benefit analysis was performed based on projected impacts of the expanded student assessment program. Educational Quality and Economic Growth 2. Both, macroeconomic growth accounting and Mincerian analysis of the labor market returns to schooling have proved that education is a key determinant of economic growth and that education spending is an investment with positive returns. Influential research (Hanushek and Woessmann 2007) has shown that the economic and social benefits of education investments are less closely correlated with education spending (as a share of GDP) and educational attainment (enrollment rates and the average years of schooling)—the two sources of data traditionally used to estimate the quality of education—than they are with average student learning performance on internationally benchmarked tests. Analyzing international test data for 50 countries over 50 years, Hanushek and Woessman established that countries whose students learn more have higher long-run rates of economic growth. Their estimates suggest that an increase of one standard deviation in average performance on internationally validated tests such as PISA (a 100-point difference or approximately the difference between Norway and Mexico) will increase annual long-run GDP growth by 1–2 percent. Their evidence also suggests that returns to quality may be even larger for developing than developed countries, due to the high economic returns to cognitive skills in these countries. 3. Microeconomic analyses show that expected wages for students depend on their level of schooling completed and their learning outcomes. Lazear (2003) finds evidence that suggests that an increase of one standard deviation in test performance yields an increase of 12 percent in individual expected income. Conversely, a lack of quality primary education, including basic literacy skills, greatly diminishes a student’s future possibilities to accumulate human capital and develop career opportunities (Psacharopoulos and Patrinos 2004). Widespread access to basic education and high-quality schooling produce also positive externalities such as a more productive workforce, higher innovation and faster dispersion of ideas throughout society, a better-informed citizenry, and lower rates of population growth and healthier and better-educated children. In sum, countries with high-quality, relevant education systems are capable of producing abundant human capital, which is needed for economic production, innovation, and competitiveness in a globalized market. 4. Research also emphasizes that returns to education are most significant at the primary and preschool levels, both due to the longer time horizon to recoup educational investments and because skills acquired in the early years facilitate learning and skill formation throughout an individual’s subsequent education (Cunha et al. 2006; Heckman 2000). These general findings are strongly supported by recent research in Peru which has correlated the development of basic cognitive and interpersonal skills early in life with individuals’ subsequent success in completing basic education, accessing higher education, the probability of graduation, employment, and income generation. (Castro, Yamada, and Arias 2011). Page 45 of 72 The World Bank Basic Education Project ( P123151 ) 5. Available evidence for Peru shows high heterogeneity in returns to education; the estimated returns per year of schooling are 3 percentage points higher in Lima (metropolitan area) than in the rest of the country and this gap has widened compared to two decades ago. Returns also vary by family income, averaging 9.1 percent for individuals from the lowest income decile in the country versus 12.2 percent for the individuals from the top income decile. Finally, the returns to education by the level of schooling completed, as shown in figure 4.1, vary as well. Tertiary education has a much higher payoff than secondary education, and the returns to completed primary education show a decline relative to incomplete primary, given the rarity of the latter in Peru today. The returns to primary and secondary education in Peru have also declined over time in Peru. Since the 1980s the returns to a complete secondary education have fallen by almost half, a pattern observed in other Latin American countries as well. Given the rapid expansion of secondary education coverage in the region over the past several decades, the share of the labor force with a complete secondary education has increased tremendously. On the other hand, the supply of tertiary graduates remains relatively scarce and has not kept pace with the demand for highly skilled workers. This provides additional force to the idea that investment in education can help reduce inequality and drive economic prosperity. Figure 4.1. Annual Returns - Peru 2004 Source: Yamada 2007. Note: All of the coefficients are statistically significant at 99 percent. The Impact of Learning Assessments on Education Quality and Learning Outcomes 6. National assessments of student learning are an important tool for improving education system quality. They help schools benchmark their performance and allow policymakers to analyze the impact of education policies and programs. Linn and Herman (1997) state that “assessments play a pivotal role in standards-led reform, by: communicating the goals, providing targets, and shaping the performance of educators and students. Coupled with appropriate incentives and/or sanctions—external or self-directed— high quality assessments 16 can motivate students to learn better, teachers to teach better, and schools to be more educationally effective.” 16 To guarantee these positive impacts, assessment data must meet high standards of quality and validity in and generate reliable Page 46 of 72 The World Bank Basic Education Project ( P123151 ) 7. Information from tests can be used to improve learning outcomes (that is, by designing better teacher training programs and informing curriculum development). Nasser (2007) states that “when effectively used, large-scale testing carries positive implications for students, teachers, administrators, policymakers and parents (Stecher 2002) as they ensure accountability, quality control, instructional diagnosis and allow identifying needs and allocating resources.” 8. When assessment results are disseminated to parents and the community in a user-friendly manner, a second channel to improve school system performance is activated: parents become more empowered in holding schools and teachers accountable, driving to higher levels of educational achievement. In Mexico, for example, accountability systems (student testing, school rankings, and school report cards) were found to have a strong, positive, and significant impact on learning outcomes (Alvarez et al., 2007). A comparison of 15 countries that participated in Trends in International Mathematics and Science Study (TIMSS) found that “students in countries with centralized examination systems scored 16.1 points higher in mathematics and 10.7 points higher in science. Students in schools where external examinations or standardized tests had a lot of influence in determining the curriculum had test scores 4.3 points higher in mathematics” (Woessman 2003). Evidence from the United States is less consistent, given the differences in the quality of student assessment systems across states, but there is some evidence that test-based accountability systems introduced during the 1990s positively influenced student achievement levels (Hanushek and Raymond 2004). 9. Peru’s national system of student assessment was very limited before the project: it only tested one grade. Thus, strengthening and expanding the system is expected to have had positive impacts on the quality of education through its influence on various stakeholders, as outlined in table 4.1. Table 4.1. Impact of National Student Assessments on Stakeholder Groups Students Nasser (2007) paraphrasing Stecher (2002) indicates that “through identifying learning achievements that are below expectations, supplying important information regarding students' knowledge, and conveying signals regarding what to study, large-scale testing results motivate many students to work harder, smarter and more effectively.” The potential beneficiaries of the student evaluations in three additional grades in at least two subjects are numerous. According to MINEDU, approximately 5.8 million students were enrolled in preschool, primary, and secondary public schools in Peru in 2017, representing the most direct beneficiaries of improvements in quality resulting from an expanded, high-quality assessment systems. Parents Nasser (2007) states that “large-scale testing might have positive effects on parents. They can manifest greater parental involvement in their children's education by encouraging and motivating them to put more effort into schoolwork. In another vein, large-scale testing results possibly will prompt parents to bring pressure to bear on ineffective schools.” Vegas and Petrow (2007) find evidence in Mexico that suggests that the involvement of parents in the administration of education can be effective and cost efficient. Informed and involved parents will consider the past performance of schools when deciding where to register their children, bringing additional pressure on teachers and administrators to improve the quality of their schools. information for policy makers, teachers, principals, parents, and students. (Braun and Kanjee 2006). This utility of the data generated depends on the quality and relevance of the assessment, as well as the expertise of those charged with the analysis, interpretation, reporting, and dissemination of results. Page 47 of 72 The World Bank Basic Education Project ( P123151 ) Teachers Nasser (2007) states that “large-scale testing has potentially positive effects for teachers too.” The results from large-scale testing provide teachers important information that can help them to identify strengths and weaknesses of students and in the curriculum (Elliot and Branden 1997;). Teachers can use test results to improve their diagnosis of students' individual needs and to allocate resources for students with greater needs. Proper interpretation and effective use of testing results can motivate teachers to work harder, smarter, and more effectively by aiding them in selecting better curriculum and/or more effective teaching methods. Moreover, useful information from large-scale testing allows teachers to align instruction with standards, and encourages them to engage in professional development to improve their instruction. In sum, teachers can effectively use assessment results as a basis for making changes in instructional practice and thus enhance student learning and achievement (Stecher 2002). Administrators Nasser (2007) concludes that “administrators can effectively use test results to examine school policies related to curriculum and instruction, and consequently perform changes in school policies that are designed to make schools more effective.” The results from assessments can aid administrators in judging the quality of their programs and guide actions for improvement. Furthermore, administrators can use assessment results to better direct the allocation of resources required for improving student learning (for example, after-school classes for students in need, teacher professional development). Policy makers Policy makers use assessment results to inform decision making regarding educational policies and to enhance their ability to monitor the quality of school system performance. Analysis of assessment results can promote better allocation of state educational resources and facilitate targeting of resources to schools that are in need. Thus, assessment results can be used as a road map for improving outcomes. High-quality assessments can also raise public confidence in education and improve the relationship between the education system, its beneficiaries, and the legislature. Countries with educational systems that include assessments and measures of quality are better able to inform parents, teachers, principals, and policy makers of potential strengths and weaknesses, and allow them to make better-informed decisions regarding education policies, reforms, and allocation of resources. Improved learning outcomes in turn are correlated with increased labor productivity and income, and higher long-term rates of national income growth. 10. On the other hand, there is limited evidence in Latin America and Caribbean countries on the impact of evaluating instructional practice and on actions oriented to improve the managerial performance of school’s principals. However, on one hand, research analyzing the eight top school systems in the world, and on the other, the U.S. school system, emphasize that the quality of teachers is the single most important factor for learning outcomes (Barber and Mourshed 2007; Hanushek 2011). Also, recent evidence shows that better school management systems have a significant impact on student performance (Bloom et al. 2014; Romero, Sandefur, and Sandholtz 2017). Given this, it is expected that the Project’s Component 2 (teacher evaluations within a merit-based career, and competitive selection and training of school principals) would have at least a similar or higher impact on cognitive skills than that of the Component 1. Cost Benefit Analysis 11. While there is a growing body of literature on the impact of cognitive ability (as measured by test scores) on earnings (Glewwe 1996; Green and Riddell 2001; Moll 1998; Murnane et al. 1995; and Patrinos and Sakellariou 2007), there is limited rigorous evidence on the impact of introducing or expanding student testing systems on students’ cognitive ability. In current literature, impact of learning assessments on school accountability is the most commonly investigated; greater awareness of parents and communities of education outcomes increases their ability to pressure school systems for improvement. Emerging work has Page 48 of 72 The World Bank Basic Education Project ( P123151 ) shown that programs which generate and disseminate information to promote school accountability can be very cost-effective. In a rigorously evaluated, randomized program in Pakistan, gains in student learning in public schools and reductions in the prices of local private schools substantially outweighed the costs of a program to measure student learning and transmit relative school results to parents through ‘school report cards.’ (Bruns et al. 2011). In Mexico, the national student assessment program was compared to other interventions likely to improve school accountability and raise student learning and was found to have the lowest costs per student (Alvarez et al. 2007). Cost-Benefit Analysis of the Proposed Project Benefits 12. This analysis mainly builds on various studies showing that the introduction of student learning assessments—with dissemination of their results—can improve school-level accountability and effectiveness of system management, and that both channels can improve student learning. Results vary, of course, with the context, the quality of the assessment information, the degree to which it is disseminated, and the scope for local stakeholders to influence school management (for example, the degree of voice in school governance or their scope for choice among competing schools). It also considers evidence showing that improving both, teacher and managerial performance have a positive impact on the delivery of cognitive skills to students. 13. Schiefelbein, Wolff, and Schiefelbein (1998) is the only meta-analysis that estimates an average effect of universal testing of 4th graders on student achievement, based on diverse country experiences. For assessment of 4th graders, Schiefelbein, Wolff, and Schiefelbein (1998) report an average impact of 12.3 percent (of standard deviation 1) given a 62.5 percent probability of adequate implementation of the assessment program. While it is reasonable to expect that the introduction of student testing at multiple grades, as under the present Project, instead of only Grade 4 could compound these effects, the Schiefelbein study suggests no parameters for this. Therefore, this estimate of the overall impact on learning outcomes is retained (12.3 percent of standard deviation 1) for this study’s core projections. 14. Although there is no evidence for the impact on cognitive skills of a mix of interventions as those supported by the project, some studies provide references for the impact of a package of multiple interventions and for the impact of improving teachers’ performance. Schiefelbein, Wolff, and Schiefelbein (1998) reports that multiple interventions that include testing, training, school-based management, and learning packages can yield 26.8 percent average impact, with a 45 percent probability of being adequately implemented. Although, this estimate cannot directly be translated to the activities of the project, it provides a reference for the impact of the whole project. Hanushek (2011), reviewing results for several studies in the United States, reports that “a teacher who is 1 standard deviation above the mean distribution of teachers in terms of quality (i.e. comparing the 84th percentile teacher to the 50th percentile teacher) is estimated to produce marginal learning gains of 0.2–0.3 standard deviations of student achievement compared to the average teacher.” As for improvement in managerial abilities, such estimates are lacking. Considering this evidence as the reference, it is assumed that the multiple intervention project under the analysis, will have, as maximum, an average impact on cognitive skills that will double that of only testing Grade 4 students. 15. A second dimension of the analysis is the expected impact of the Project on MINEDU´s ability to implement the assessment program with the degree of integrity required for robust results. The Schiefelbein, Wolff, and Schiefelbein (1998) meta-estimate is that institutional execution of Grade 4students assessment Page 49 of 72 The World Bank Basic Education Project ( P123151 ) (Component 1 of the program) will be improved from a 62.5 percent probability of adequate implementation to 72.5 percent probability. In the case of this Project, the differential was larger, due to the incremental financial resources and capacity-building activities that the Project supported, which complemented and reinforced MINEDU’s existing capacity efficiently. It is expected that these activities will continue to be performed in a relatively efficient manner in the following years. As a result, it was estimated that the proposed Project raised the probability of adequate implementation of the expanded testing program from 62.5 percent to 87.5 percent. 17 In the case of the whole project (Component 1 and Component 2), it was estimated that the Project raised the probability of adequate implementation from 45 percent to 70 percent. 16. The third step in the estimation of benefits is to calculate the expected impact of improvements in student learning on students’ future earnings. Based on the meta-study ‘Returns to Education in Developing Countries’ by Patrinos and Psacharopoulous (2010) the average estimated return to a standard deviation 1 increase in cognitive skills across different countries is between 0.17 and 0.22. For the analysis, a moderate estimate of 0.20 was adopted. 17. The Project’s impact on future earnings is calculated multiplying the impact of the expanded national assessment on learning (12.3 percent in the case on Component 1 of the Project and 24.6 percent for Components 1 and 2 together), by the increased probability of adequate implementation with support of the Project (0.875 minus 0.625 in the case of Component 1 and 0.70 minus 0.45 for Components 1 and 2, together), and by the impact of estimated learning increases on future wage returns (0.2). Under these parameters, the proposed Project will increase future earnings by a factor between 0.006 and 0.017. Table 2 summarizes the calculations. Table 4.2. Expected Impact of the Project C1a C1+C2b (A) Impact of national assessments on student achievement in terms of SD 0.1230 0.2460 (B) Probability of adequate implementation without the Project 0.6250 0.4500 (C) Probability of adequate implementation with the Project 0.8750 0.8000 (D) Expected impact of the Project on student achievement (A × (C − B)) 0.0310 0.0860 (E) Estimated returns to standard deviation 1 increase in cognitive skills 0.2000 0.2000 (F) Expected impact on earning of the Project: (D × E) 0.0062 0.0172 Note: a. Reflects the expected impact of the expanded assessment of student learning (Component 1 of the Project). b. Reflects the expected impact of the expanded assessment of student learning (Component 1 of the Project), together with the improvement of teacher and managerial quality (Component 2 of the Project). 18. Project beneficiaries are the students exposed to the new evaluation system during their school years finishing at least secondary education in public schools and entering the labor market. According to MINEDU, the completion rate of secondary in Peru for 2015 is 72.2 percent: 46.5 percent retained that level, 12 percent pursued and completed non-university higher education, while 13.7 percent graduated from university. The 17 This probability is considering that the programmed student’s evaluations for Grades 2, 4, and 8 were not carried out in 2017. MINEDU took this decision in a context of delays at the start of the school year, due to a national given the effects of the El Niño weather phenomenon and strikes by school teachers. However, evaluations will be carried out this year and budget allocations have already been made with that purpose. Page 50 of 72 The World Bank Basic Education Project ( P123151 ) analysis follows the student cohort starting primary school in 2014 18 until 2033 and then calculates how many of these students will enter the labor market who have at least completed secondary school. These calculations are based on population projections and labor participation rates of the National Bureau of Statistics (Instituto Nacional de Estadística e Informática, INEI) and enrollment and completion data provided by MINEDU.19 Table 4.3. Number of Beneficiaries for the Cohorts Starting School between 2014 and 2023 Year Entering Number of Students Workers Entering the Workers Entering the Workers Entering the in Primary Enrolled in Public Labor Market as High- Labor Market as Labor Market as School Schools school Graduates Technical Graduates University Graduates 2014 392,108 124,532 38,254 40,987 2015 417,369 132,554 40,719 43,628 2016 429,920 136,540 41,943 44,940 2017 422,333 134,131 41,203 44,147 2018 418,826 133,017 40,861 43,780 2019 415,429 131,938 40,529 43,425 2020 412,600 131,040 40,253 43,129 2021 410,433 130,351 40,042 42,903 2022 408,624 129,777 39,865 42,714 2023 407,031 129,271 39,710 42,547 2024 405,513 128,789 39,562 42,389 2025 403,927 128,285 39,407 42,223 2026 402,283 127,763 39,247 42,051 2027 400,675 127,252 39,090 41,883 2028 399,089 126,749 38,935 41,717 2029 397,516 126,249 38,782 41,553 2030 395,942 125,749 38,628 41,388 2031 394,383 125,254 38,476 41,225 2032 392,847 124,766 38,326 41,065 2033 391,311 124,279 38,176 40,904 19. With the number of students entering the labor market from 2014 until 2029, future annual earnings are predicted based on an estimated logarithmic earnings function. 20 The function has the following explanatory variables: X1: Years of primary school 18 An additional conservative assumption was made that the Project affects only students entering primary in 2014 (the first year of the Project) and not those already in school, even though a significant share of current students will be exposed to the assessment program as it is phased into the targeted grades over the next five years. 19 It is assumed that enrollment and completion rates do not change over time. This assumption is conservatory because it ignores a potential impact of the Project: The intervention could increase completion rates (decrease dropout rates) and the later entry into the labor market with increased aggregate earnings, based on current calculations of rising returns to education in Peru. 20 The following assumptions were made: (a) earnings function does not change over time, except for the impact of the Project and (b) children do not enter the labor market and begin earning wages until age 18 years. Page 51 of 72 The World Bank Basic Education Project ( P123151 ) X2: Years of high school X3: Years of superior education - technical X4: Years of superior education - university Z: Potential experience (age minus years of schooling minus primary entry age) 21 Z2: Potential experience squared. = 0 + 1 1 + 2 2 + 3 3 + 4 4 + θZ + δZ2 + ϵ According to Castro and Yamada (2010), the values for the parameters are the following: Constant 0 = 0.3 Years of primary school 1 = 0.051 Years of high school 2 = 0.07 Years of superior education − technical 3 = 0.089 Years of superior education − university 4 = 0.16 Potential experience θ = 0.02 Experience squared δ = −0.0002 20. Using this equation, future annual earnings are calculated for the 2014 student cohort up to the 2033 cohort. It is assumed that the effects of testing on earnings will last 20 years. Also, it is considered that there is a probability of being unemployed. That probability is approximated with the average unemployment rate between 2007 and 2016, according to the level of studies. 21. The Project benefits are calculated comparing the situation with the Project with the situation in absence of the Project. The benefits in the situation with the Project are calculated multiplying the estimated future earning by the impact factor in two scenarios: (a) conservative - the impact of the Project on earnings is 1.0062, as it considers only the benefits of Component 1 of the project (b) comprehensive - the impact of the Project on earnings is 1.0172, as it considers the effects of Components 1 and 2. Thus, the benefits of the Project are calculated as the difference of the estimated future earnings with and without the Project. Table 4.4. Project Total Benefits (US$) Conservative Scenario (Discounted Comprehensive Scenario (Discounted Cohort Earnings) Earnings) 2014 41,041,382 82,082,763 2015 41,644,040 83,288,080 2016 40,891,845 81,783,690 2017 38,293,095 76,586,190 2018 36,200,543 72,401,086 2019 34,229,052 68,458,104 2020 32,407,346 64,814,693 2021 30,730,723 61,461,447 2022 29,165,591 58,331,183 2023 27,694,380 55,388,760 21 The age at which entry to primary school occurs is 6 years. Page 52 of 72 The World Bank Basic Education Project ( P123151 ) Conservative Scenario (Discounted Comprehensive Scenario (Discounted Cohort Earnings) Earnings) 2024 26,301,777 52,603,554 2025 24,974,684 49,949,369 2026 23,710,733 47,421,467 2027 22,512,390 45,024,781 2028 21,375,483 42,750,966 2029 20,296,307 40,592,614 2030 19,271,277 38,542,554 2031 18,298,405 36,596,810 2032 17,375,411 34,750,823 2033 16,498,723 32,997,445 Costs 22. For this analysis the total Project costs by year are used, based on the amounts already executed for the Project, until 2017. For 2018 and 2019, the budgeted amounts by MINEDU have been included. In addition, estimates are made of the incremental recurrent costs of carrying out the evaluation systems and disseminating its results nationally every year (to parents and individual schools in the case of census-based tests and at the national level and to school districts in the case of sample-based tests). Table 4.5. Project Total Costs (in US$) Year Project Cost Incremental Recurrent Cost Total Discounted Cost 2014 21,252,974 19,862,593 2015 45,106,148 39,397,457 2016 68,184,230 55,658,642 2017 30,553,249 23,308,928 2018 17,346,927 12,368,119 2019 3,625,266 10,146,409 9,176,649 2020 17,978,407 11,196,048 2021 9,621,761 5,599,952 2022 16,751,930 9,111,940 2023 10,891,950 5,536,915 2024 16,227,282 7,709,465 2025 10,146,409 4,505,127 2026 17,978,407 7,460,400 2027 9,621,761 3,731,485 2028 16,751,930 6,071,670 2029 10,891,950 3,689,480 Cost-Benefit Indicators 23. In the conservative scenario, comparing the total discounted benefits of the Project for the 2014 to Page 53 of 72 The World Bank Basic Education Project ( P123151 ) 2033 student cohorts with the total discounted costs, an Economic Rate of Return for the Project equal to 10.7 percent has been obtained. The NPV is equal to US$339 million. 22 In the comprehensive scenario included in the analysis, it was determined that the IRR was positive and might increase up to 13.8 percent and an NPV of approximately US$ 900 million. Therefore, the Project is expected to render a positive reward in economic terms: NPV of estimated and expected costs is lower than the NPV of expected benefits. Table 4.6. Project Economic Indicators Conservative Comprehensive Scenario Scenario IRR 10.7% 13.8% NPV (US$) 339,218,406 902,821,682 Additional Parameters 24. Finally, the potential impact of the Project on several additional parameters considers poverty, gender, fiscal revenues, and the public/private education interface. Poverty 25. Many education investments aim at expanding coverage, which typically has strong, progressive impacts on the educational opportunities for the poorest students and direct links to poverty reduction. Given that Peru already has extremely high rates of education enrollment, equally large impacts on the poorest students was not the goal of the Project. There is no research evidence that suggests that the introduction or expansion of a student testing system has significant differential effects on the learning outcomes, employment, and long-term earnings of students from lower versus higher income deciles per se. However, clearly one of the key channels through which testing can affect average learning outcomes is by providing policy makers with disaggregated information on student performance that can support better-targeted interventions. In fact, data from the student assessments has been used this way over the past several years. The evidence of large gaps in learning outcomes between rural low-income and urban students inspired the design of the educational accompaniment program, which has been targeted to low-income rural students. Student assessment results for 2011 show that schools receiving the assistance have made faster progress than other schools. However, the average gap in student results between urban and rural low-income students remains significant: in 2016, the share of rural Grade 2 students achieving satisfactory performance in mathematics and reading was 17.3 percent and 16.5 percent respectively, compared with 36.6 percent and 50.9 percent for urban students and 34.1 percent and 46.4 percent for all students. It is reasonable to expect that an important share of the overall learning gains that are expected because of expanded student testing and school performance information will derive from the impact of better-targeted programs for students in lagging schools and regions. This, in turn, implies a differential—and positive—impact of the program on the poorest students. There is unfortunately no robust basis for estimating the size of this impact. 22 The NPV was calculated at a rate of 7 percent. Page 54 of 72 The World Bank Basic Education Project ( P123151 ) Gender 26. Education participation rates at all levels in Peru are currently higher for girls than boys. There is also no gender gap in learning outcomes disfavoring girls; girls’ performance on student assessments—both on the national Grade 2 test and on regional and international tests such as PISA—is in many cases higher than that of boys. There is however, still a bias in the labor market, with average wages for females approximately 29 percent lower than that for men. 23 While the Project is expected to contribute to improvements in student learning, there is no basis for projecting that these will rise differentially faster for girls (in fact, the opposite might be the case). Nor is there a basis for projecting that relative wages for girls will be affected significantly even if the learning advantage that girls currently hold were to grow stronger, as the labor market factors contributing to the current wage differential appear significantly influenced by factors other than academic achievement. On the other hand, there is no basis for projecting any measurable negative impacts on girls. The most likely outcome is that the Project could have marginally positive impacts on boys’ learning achievement, relative to girls. Fiscal Impact 27. The major route through which the Project is expected to have an impact on the Government’s fiscal position is through higher taxes generated from the projected increases in average wages for students benefitting from this Project. These incremental earnings are sufficient to justify the proposed investment but are not sufficiently large to constitute an important stream of government fiscal revenues. Anyhow, the stream of fiscal earnings is just a transfer from the workers to the Government. Those transfers are already included in the estimated higher earnings of students. On the other hand, there is no basis for projecting any negative long-term impacts of this Project on the Government’s fiscal position. Therefore, fiscal benefits were not explicitly incorporated into the Project cost-benefit analysis (but of course, include all projected fiscal costs). Public Sector Rationale 28. The central intervention undertaken by this Project is strengthening MINEDU’s capacity to measure the learning progress of Peruvian children and to disseminate these results to schools, parents, communities, and the Peruvian public more broadly. Such information is a public good that research has shown can increase accountability pressures on public school systems and stimulate them to improve. A second potential benefit of this information is to equip parents to make better choices among competing school options for their children. As MINEDU’s testing program covers students in private as well as public schools, the information it provides, generates the potential to increase competition among public and private school providers and to provoke shifts in enrollments across these sectors. The private school sector in Peru is already relatively large for Latin America and has been growing rapidly in recent years. Approximately 25 percent of basic education enrollments in Peru in 2017 were in private schools: 27 percent at the preschool level, 25 percent at the primary level, and 24 percent at the secondary level. Many factors appear to be driving the growth of private enrollments, most notably, the teacher strikes in the public system have kept schools closed for significant periods during the last several school years. There is a near-complete lack of empirical data to support estimates of either the potential size of enrollment shifts in response to expanded information about the 23INEI (2016) estimates that. in 2015, women earned on average 28.6 percent less than men. According to Ñopo (2012), the gap is approximately 20 percent when controlling by education, age, urban or rural location, presence of children, and other attributes. Page 55 of 72 The World Bank Basic Education Project ( P123151 ) comparative performance of public and private schools, or the potential efficiency gains from such shifts. The most that can be observed is that such shifts in parental choice are likely to be welfare-enhancing for families and to some extent efficiency-enhancing for the education system as a whole. Because these benefits cannot be estimated with any specificity, they are not included in the cost-benefit analysis. However, it is reasonable to hypothesize that the school choice impacts that may derive from the Government’s investment in the public good of high-quality comparative school performance information will positively affect the Project’s benefits. Page 56 of 72 The World Bank Basic Education Project ( P123151 ) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS This is a summary of the “Reporte de Cierre de la Implementación del Programa SWAP – Educación” (Contrato de Préstamo N° 8226-PE) translated into English by the Bank. Any mistakes are the sole responsibility of the World Bank. Assessment of the Project’s Objective, Design, and Implementation Objective 1. The objective of the Education SWAp Program (hereinafter, the ‘Education SWAp’) is to strengthen the capacity of MINEDU to assess student learning, pedagogical practices, and school leadership in basic education to contribute to the ultimate target of improving student learning outcomes. 2. Since 1996, MINEDU—through the UMC—has assessed student learning using censuses and samples, including assessments led by international bodies. Considering the UMC’s extensive experience, the agency required the advice and services of international experts to expand this experience, especially with regard to innovative issues. 3. On the other hand, following the implementation of the Ley de Reforma Magisterial—Law 29944, approved in 2012, MINEDU had to conduct a range of teacher assessments within a short period. To build the capacity of the Teacher Development General Directorate (Dirección General de Desarrollo Docente, DIGEDD) while meeting high-quality standards, it was also necessary to hire international advisers with experience in these types of evaluations. Design 4. The Education SWAp was designed in 2012 to support the sector strategy of PELA, the objective of which was to improve students’ learning outcomes in regular basic education. 5. The design of the Education SWAp used the lessons learned in the implementation of the SWAp program ‘Closing Gaps in Priority Products in the Articulated Nutritional Program’ (Nutrition SWAp), the objective of which was to support the efforts of the Articulated Nutritional Program to reduce chronic child malnutrition in the country. The Nutrition SWAp was implemented by the MEF through the Sector Loan Coordination Unit, in coordination with the sectors involved. Based on this experience, it was decided that the Education SWAp should be implemented directly by MINEDU. Given that this was a nontraditional loan with an innovative design, its implementation involved a learning curve for both MINEDU and the MEF. Original Design 6. The design of the Education SWAp was based on three components: (a) Evaluating Student Learning, (b) Evaluating Instructional Practice and School Leadership, and (c) Strengthening MINEDU’s Implementation Capacity, with activities that are part of PELA. 7. The subcomponents included in Component 1 were census, sample, and international learning assessments. This component also included the assessment of preschool education, which does not directly Page 57 of 72 The World Bank Basic Education Project ( P123151 ) measure children’s learning, but rather the progress in their development and the quality of their educational environment. Component 2 included the evaluation of pedagogical practices through the application of the Stallings Classroom Snapshot (‘Stallings’) instrument, and only one of the teacher assessments: assessment of access to management positions. Component 3 of the SWAp involved the program’s management, which was expected to help improve MINEDU’s implementation capacity. Technical Areas 8. The UMC was responsible for the subcomponents ‘Student Census Assessment,’ ‘Student Sample Assessment,’ and ‘Participation in International Assessments’ under Component 1 of the Education SWAp. The Preschool Education Directorate (Dirección de Educación Inicial, DEI), which reports to the Regular Basic Education General Directorate, oversaw measuring childhood development progress and assessing the quality of the learning environment. Because the DEI was essentially judge and jury in the assessment, it was decided during the implementation of the Education SWAp that this responsibility should be entrusted to the Strategic Monitoring and Evaluation Office (Oficina de Seguimiento y Evaluación Estratégica, OSEE). 9. The Education Research and Documentation Department (Dirección de Investigación y Documentación Educativa, DIDE) was originally in charge of using the Stallings instrument for the ‘assessment of the effective use of classroom time and other educational quality attributes’ as part of Component 2. This technical area was eliminated from the organizational structure of MINEDU when its Regulations on Organization and Functions were amended in 2015, and the task was assigned to OSEE. Most of the team from DIDE that had handled the application of the Stallings instrument was absorbed by the team from OSEE, which meant that the lessons learned by MINEDU in the use of this tool were not lost. 10. The DIGEDD was responsible for the subcomponents of Component 2 related to teacher evaluations, and the activities included were assigned to its Teacher Evaluation Directorate (DIED). The ‘Evaluation of Access and Training of Principals’ subcomponent involved the assessment of teachers’ qualifications for holding leading positions, entrusted to the DIED, but under the responsibility of the DIGEDD, and the training of the selected candidates for their new role, under the responsibility of the School Management Quality General Directorate (Dirección General de Calidad de la Gestión Escolar, DIGC) and entrusted to its School Management Strengthening Directorate (Dirección de Fortalecimiento de la Gestión Escolar - DIF). The decision making, coordination, and bureaucratic formalities would have gone more smoothly if these subcomponents had been the direct responsibility of the DIED and the DIF. Technical Coordinator 11. In the original design of the Education SWAp, the UMC was chosen to be the program’s technical coordinator. This was decided not only because the UMC is one of the technical units with the most experience in assessment, but also because it has a great deal of experience in procurement processes, because the assessments it performs are quite large in scale. 12. Before the Project was effective, however, it was decided that the SPE would act as the technical coordinator of the Education SWAp. On the one hand, it was viewed as advantageous for the technical coordination to be assigned to a unit that was not directly responsible for the program’s activities, while on the other, it was deemed advisable for this unit to be hierarchically above those that were responsible for such activities. Page 58 of 72 The World Bank Basic Education Project ( P123151 ) Administrative and Fiduciary Coordination 13. Given that one of the benefits of the Education SWAp is the use of World Bank Guidelines in the procurement processes, the best option was to entrust the fiduciary management of the program to EU 118, which was already implementing a program with the support of international cooperation based on the procurement guidelines of the Inter-American Development Bank, which are similar to those of the World Bank. 14. To implement the Education SWAp, EU 118 hired a team of procurement specialists with expertise on the use of international cooperation guidelines, so that they could be tasked exclusively with the procurement process for this program. Programming and Planning 15. During the first stage for the design of the Education SWAp, an AOP was prepared for 2013, 2014, 2015, and 2016 for a value of PEN 114,739,951, corresponding to eligible expenses. Although the Loan Agreement was signed on July 1, 2013, the Project was not effective until May 20, 2014, after the approval of its Operational Manual, as required in the Loan Agreement. As such, although the activities for the Education SWAp were implemented in 2013, none were procured using the World Bank’s guidelines, and only four World Bank procurement processes were carried out in 2014. 16. According to the program’s Operational Manual, at the start of each year, the Technical Coordinator of the Education SWAp must submit the AOP to the World Bank for no-objection. To do so, it had to approve and consolidate the AOPs submitted by the technical units and EU 118 in November of the previous year. For this reason, the annual AOP preparation process for the Education SWAp did not match up with the annual activity planning and budget preparation process, which starts in May of the previous year. Because of this, the majority of the activities programmed in the AOP did not have a budget allocated at the beginning of the year, which meant that to begin the procurement processes it was necessary to make budget modifications. One lesson learned from the program is that it is necessary to follow a single budget preparation schedule. 17. On the other hand, the AOPs prepared by the technical units include the approximate cost of their activities, based on which their budget is defined. On many occasions, the calculation was not accurate, which meant that the budget allocated for their activities was either insufficient or excessive. One lesson learned from the Education SWAp is that AOP preparation should involve the unit with expertise in designing activity cost structures and pricing. Program Monitoring and Results Framework 18. The Education SWAp RF included three PDO indicators and eleven IRIs. The annual targets for each indicator were monitored on an ongoing basis. As the activities progressed, some of these targets proved to be too ambitious and had to be modified. Program Restructuring 19. Over the course of the Education SWAp, the program underwent two restructuring processes that were established in addendums to the Loan Agreement. The first addendum was signed on November 9, 2015, Page 59 of 72 The World Bank Basic Education Project ( P123151 ) to reflect the institutional changes made to MINEDU with the approval of the ministry’s new Regulations on Organization and Functions, and to include the following teacher assessments: (a) teacher performance evaluation, (b) public teaching career entry evaluation, and (c) promotion evaluation. Second Restructuring 20. The second addendum to the Loan Agreement was signed on July 7, 2017. In this addendum, OSEE was tasked with measuring childhood development progress and assessing learning environment quality using the MELQO instrument, rather than having the DEI apply the early development indicator (EDI) and Early Childhood Environmental Rating Scales – Revised (ECERS-R) for the same purpose. 21. In conclusion, the design of the Education SWAp was sufficiently flexible to be modified based on the institutional changes in MINEDU and the lessons learned during its implementation, as well as the needs of the technical areas. The main weakness in its original design was the failure to include the teacher assessments. Implementation 22. The Education SWAp had no central management, but two coordination units instead: one in charge of the technical area, and the other in charge of program management. While the coordination between the two units was fluid, which allowed for the efficient development of the Education SWAp, at times there was a need for a body that was hierarchically superior to them both to take responsibility for the program’s development and handle final decision making. 23. The annual work of the Education SWAp was based on the consolidated AOP approved by the SPE, without no-objections from the World Bank. The technical units were responsible for preparing the terms of reference or technical specifications for the services, with the support of the program’s technical and administrative coordinators. Additionally, the review by the World Bank before granting its no-objection to these documents assured their quality. This process required some learning on the part of the technical units, given that the traditional method of defining service requirements—that is, according to National Procurement Guidelines—is focused on making them as narrow as possible. The World Bank’s Procurement Guidelines, on the other hand, encourage greater flexibility in service provider requirements, to expand the field of providers and make the procurement process more competitive. 24. The technical units, EU 118, and the SPE directly evaluated the proposals through their representatives to the Assessment Committees. World Bank Procurement Guidelines value the discretional powers of Assessment Committee members, as well as the qualitative—and not just quantitative— assessment of the proposals. This also involved a learning process, because it represented a change in customary assessment practices under the National Procurement Guidelines. 25. The technical coordinator for the Education SWAp was the liaison between the technical units and EU 118. As such, it was agreed that all communications between these parties should be sent through the SPE. This agreement to centralize information, which was not formalized in the Operational Manual, caused no delays in the procurement processes, merely resulting in more organized information. This was due mainly to the fluid communication among the technical units, EU 118, and the SPE. Under different circumstances, however, this additional step could possibly hold up processes. Page 60 of 72 The World Bank Basic Education Project ( P123151 ) 26. Along with making it possible to hire international consultants and service providers with the same ease as their domestic counterparts, the World Bank Procurement Guidelines used by the Education SWAp offered the advantage of precluding any contestation to the results of the procurement processes. Additionally, the implementation of the contracts signed as part of the Education SWAp, including the payment of the agreed services, was punctual. This helped the program earn a good reputation among consultants and providers of services other than consulting, which then attracted more competition and better candidates. As a result, not only did the quality of the services improve, but their cost was reduced. 27. Given that the consulting activities and services carried out as part of the Education SWAp were innovative by their very nature, the search for consultants and providers was a core part of the procurement process. The program would have had better results if the market study preparation process had been clearly detailed and designed, as noted above. Externalities 28. External factors that could not have been foreseen in the design of the Education SWAp ended up influencing its implementation. Such was the case with the approval of the Licensing Regulations for public and private universities, as part of the University Reform Act, on November 20, 2015. According to these regulations, universities cannot offer new academic programs until they obtain an institutional license from the National Superintendency of University and Higher Education. These regulations were not considered when the Second Specialization Program in School Management with Pedagogical Leadership was designed in 2014, nor in the process of contracting with universities to offer this course, which was implemented throughout 2015. As a result, MINEDU had to carry out new procurement processes to cover the areas that were under the responsibility of those universities that failed to obtain a license in time. 29. During 2017, in particular, there were many unforeseeable events that ended up affecting the progress of the Education SWAp. First, the coastal El Niño phenomenon that hit during the first few months of the year resulted in the suspension of classes in seven regions of Peru, in some cases for up to a month and a half. On the other hand, the national teachers’ strike between June and September of that year also resulted in the interruption of educational activities. The suspension of classes due to these two causes affected the development of some activities included in the Education SWAp, and made it impossible to comply with all of the program’s contractual obligations. 30. Due to the cumulative effects of the suspended classes during the year, MINEDU decided in September 2017 to cancel the ECE that was scheduled for November. While this did not affect the fulfillment of the programmed targets of the Education SWAp, its cancelation did have administrative implications for the program. The decision was made after a contract for the design and reading of the optical sheets for the ECE had been signed, and MINEDU decided to cancel the assessment in October. In response, the company hired to perform this service initiated arbitration proceedings before the Lima Chamber of Commerce, in accordance with the specific contract conditions. 31. On the other hand, it should be noted that one of the demands of the teacher strike was the cancellation of the teacher performance assessments. In 2017, preschool teachers were scheduled to undergo this assessment, which was considered one of the targets of the Education SWAp. The strike came to an end without canceling the assessment, although the conditions for its implementation were changed slightly. One positive result is that the teacher assessment became a major topic of political debate and gained prominence Page 61 of 72 The World Bank Basic Education Project ( P123151 ) in the public eye. 32. The political crisis unleashed by the teachers’ strike resulted in a series of changes in the Cabinet of Ministers, with Marilú Martens losing her job as Minister of Education and Idel Vexler taking her place. Minister Martens was the second minister to leave the position during the implementation of the Education SWAp. The program was initially implemented, starting in May 2014, under the administration of Minister Jaime Saavedra, who stepped down in December 2016 after being censured by the Congress of the Republic. Minister Daniel Alfaro succeeded Minister Vexler, filling the position following the change of the Cabinet of Ministers in April 2018, after President Pedro Pablo Kuczynski resigned from office. Considering that the Education SWAp was designed during the administration of Minister Patricia Salas, the program has thus existed under five different ministers. However, this did not prevent it from achieving its objective. Operations Results Assessment vs. Established Objectives 33. The Education SWAp was designed to achieve the development objective, to be reached through the following intermediate results: PDO Indicator 1: Cycles of Preschool, Primary and Secondary Education levels with student learning outcomes analyzed 34. As part of the Education SWAp, an assessment has been performed on Cycle II (preschool education), Cycles IV and V (primary education), and Cycle VI (secondary education) of the Peruvian education system. The following IRIs contributed to achieving this indicator: • IRI 1: Student assessments for new grades of Primary Education and Secondary Education levels validated in a pilot study. • IRI 2: Additional grades at the Primary Education and Secondary Education levels in which sound national assessments are administered in at least two subject areas. • IRI 3: Results of national student assessments in new grades at the Primary Education and Secondary Education levels reported at the regional and school level. The Education SWAp sought to expand the grades evaluated by MINEDU, as well as the subjects to be evaluated. During the program’s implementation, Grade 6 of primary, Grade 2 of secondary, and Grade 4 of primary were assessed for the first time. All of these assessments were validated by a pilot study, and reports with the results of their final application were distributed nationwide. • IRI 4: National surveys carried out to evaluate child development progress and the quality of public preschool education services. In 2014, as part of the Education SWAp, the first preschool education assessment was performed using the EDI and the ECER-S; while in 2017, the second assessment was performed using the MELQO. Page 62 of 72 The World Bank Basic Education Project ( P123151 ) • IRI 5: Borrower’s (through MINEDU) participation in international assessments of student learning outcomes. In 2013, MINEDU participated in the TERCE, which assessed the performance of students in Grades 3 and 6 of primary school in mathematics, reading and writing, and natural sciences, in the case of Grade 6. One approximate gauge of the results for the education system are the assessment results of PISA that is taken by 15-year-old students. Through the Education SWAp, contracts were signed with the entity responsible for the final PISA test in 2015, and in 2017 for the 2018 PISA assessment pilot. Furthermore, as part of the program, MINEDU participated for the first time in the ICCS to assess how well youths are prepared to assume their duties as citizens. PDO Indicator 2: Instructional Practices Analyzed 35. OSEE prepared a document presenting the results of the four studies on classroom time on task, which is one aspect of pedagogical practices, conducted between 2012 and 2015. The following IRIs contributed to achieving the indicator for this objective: • IRI 6: National studies of classroom instructional practice in Primary Education and/or Secondary Education levels, as the case may be. • IRI 7: Results of national studies of classroom instructional practice reported at the national level. With technical support from the World Bank, MINEDU used the Stallings instrument to measure classroom time on task during 2013 and 2015 (in primary education) and during 2014 and 2016 (in secondary education). The results were analyzed by the teams from the DIDE and OSEE, and then elaborated upon in a series of research papers and brief articles. These latter articles were distributed to all of the regional departments of education in the country. 36. There is also a report in which the DIED analyzes the results of the first teacher performance assessment. These are the IRIs that contribute to achieving the indicator for this objective: • IRI 9: Teacher performance evaluation system designed, implemented and results reported. MINEDU has designed a teacher performance assessment system, which will be progressively implemented starting with preschool teachers. The teachers take a regular exam, for which the DIED has designed a classroom observation rubric. Only those who fail this exam must take the special assessment, after participating in a professional development program under the responsibility of MINEDU. Those teachers who do not pass the first special assessment must once again complete the professional development program and are then evaluated in a second special assessment. In 2017, the first teacher performance assessment was performed for preschool teachers, and the results were reported in January 2018. • IRI 10: Evaluation instruments to determine entrance to and promotions within the Teaching Career designed. Page 63 of 72 The World Bank Basic Education Project ( P123151 ) In 2015 and 2017, MINEDU organized admissions exams for the public teaching career. In 2015, 2016, and 2017, it performed promotion assessments for the public teaching career. The DIED designed the relevant instruments. PDO Indicator 3: School management positions staffed through the school principal evaluation system 37. Between 2015 and 2017, 18,647 leading positions were competitively assigned using the School Principal Selection Evaluation. In 2014, those teachers who held the position of school director or principal were evaluated to be confirmed for the position. Likewise, those teachers who ranked at level four or higher on the career ladder and who were interested in holding a leading position were evaluated based on two entry exams for leading positions in 2014 and 2016. • IRI 8: School principal evaluation system designed. The School Principal Selection Evaluation consists of a series of steps designed to ensure a competitive selection process to fill these positions, carried out in two stages. The first is a centralized stage, in which applicants take the standardized national exam; while the second, decentralized stage involves the application of the Professional Career Path Evaluation instrument. • IRI 11: School principals certified in school management. Under the Teacher Reform Law, MINEDU is responsible for training those teachers who are promoted to school management positions through a National Training and Education Program. MINEDU designed this program in 2014, dividing it into three stages: induction, specialization, and enrichment. As part of the Education SWAp, the specialization stage was implemented through the Diploma Program in School Management and the Second Specialization in School Management and Pedagogical Leadership, imparted by universities selected through a competitive process. Of the 15,461 teachers who were promoted to leading positions following the assessments in 2014, 13,536 held diplomas in school management. As of the closing date of the Education SWAp, the second specialization is still under way. 38. The fulfillment of the intermediate results and the development objectives of the Education SWAp made it possible to reach the program objective to ‘strengthen the capacity of MINEDU to assess student learning, pedagogical practices, and school management in basic education’, which is expected to help improve student learning outcomes. 39. The Education SWAp progress reports, which have been submitted periodically to the World Bank, have detailed the progress made in the fulfillment of program targets. The contents of these reports were validated through an outside review of indicators entrusted to an independent consultant, thus complying with the provisions established in the Loan Agreement. The observations raised by the consultant on indicator target fulfillment helped clarify the approach underlying some of the program’s objectives and indicators. Page 64 of 72 The World Bank Basic Education Project ( P123151 ) Assessment of Borrower Performance during Operational Implementation, with a Special Emphasis on Lessons Learned that Will Be Useful in the Future 40. Both the technical areas and the administrative and technical coordinators of the Education SWAp were fully committed to ensuring the smooth implementation of the program. One of the benefits of the Education SWAp was the access it provided to specialized consultants and services in support of innovative activities. The technical coordinator and the program’s procurement specialists worked together with the technical areas to identify the best service providers and consultants, taking full advantage of World Bank procurement guidelines. To do this, they used the experience of MINEDU, the World Bank’s experience in Peru and elsewhere in the world, and different channels to promote the calls for bids, such as the United Nations Development Business web portal and local media outlets. Although significant efforts were made in this regard, as noted above, the Education SWAp would have benefited from a well-defined strategy for the performance of market studies. 41. The MEF’s Public Debt and Treasury Department (DGETP) also played a role in the implementation of the Education SWAp. The department was responsible for monitoring the program’s development, and the portfolio review meetings held by this department, MINEDU, and the World Bank were a very useful tool in this regard. The restructuring processes for the Education SWAp, on the other hand, were designed in coordination with the DGETP, whose director general signed the addendums to the Loan Agreement formalizing these restructuring plans. Lastly, this department was also responsible for defining the annual debt ceiling for the different sectors, as well as approving disbursement requests in accordance with the Debt Operations Disbursements and Debt Management Guidelines. Financial Implementation 42. The financial implementation of the program was successful, as may be observed by the level of expenditures executed each year. 24 It can thus be concluded that the administrative management of the Education SWAp was well designed. Nevertheless, the program’s contribution to improving the administrative management capacity of the technical units cannot be verified. Disbursements 43. The total loan amount granted by the World Bank for the implementation of the Education SWAp was US$25 million, which was disbursed as the targets were met for certain indicators within the program’s RF. Under the original design of the Education SWAp, the first disbursement was scheduled for September 2014. Due to delays in the start of the program’s implementation, the disbursements were reprogrammed, and the first—based on the value of the indicator targets for 2013 and 2014—was paid out in December 2015. After that, MINEDU requested the disbursements on time, even moving up the disbursement requests for the value of those targets fulfilled, where permitted by the debt ceiling. 44. The simplicity of the DLI fulfillment reports facilitated the disbursement execution process. 24 98.4 percent (2014), 99.1 percent (2015), 99.8 percent (2016), and 96.1 percent (2017). Page 65 of 72 The World Bank Basic Education Project ( P123151 ) Performance Assessment of the World Bank during the Preparation and Implementation of the Operation, with a Special Emphasis on Lessons Learned 45. Throughout the entire implementation of the Education SWAp, fluid communication was maintained between MINEDU—through its technical coordinator—and the World Bank, thus facilitating the achievement of its objectives. The World Bank supervision mission meetings, held twice a year, were a very useful space for evaluating the program’s development and suggesting any changes necessary to guarantee its success. 46. At the beginning of the program, the advisory services of the World Bank’s specialists in other parts of the world were very important in the conceptualization of innovative activities. Such was the case with the design of the Item Bank 25, the application of the preschool education assessment instruments (EDI, ECER-S, and MELQO), and the use of the Stallings instrument to analyze classroom time on task. The Education SWAp would have benefited from the advice of international experts in the following stages, and experts from the World Bank’s education area who work in the region, which would have enabled the program to establish connections with thematic experts and specialized entities able to handle specialized consulting processes. 47. The procurement specialists for the Education SWAp received support from the World Bank’s procurement specialist in charge of the program. However, due to the number of projects for which she was responsible, she was not always available to provide support exactly when needed. In any case, the World Bank team that participated in the Education SWAp was highly committed to making the procurement processes more flexible, and in the most important cases, its no-objection replies to requests submitted were issued quite quickly. The definition of deadlines for the review of the documents and the granting of no- objection by the World Bank was highly useful in the development of the Education SWAp. This definition did not occur until the middle of the program’s implementation. One lesson learned is to do this at the beginning. 48. The use of the World Bank’s online system to monitor the Annual Acquisition and Procurement Plan (PAAC) for the original program (Procurement Plans Execution System [Sistema de Ejecución de Plan de Adquisiciones]) went smoothly, because this system had been used extensively by the procurement specialists of the Education SWAp. During the final stage of the program’s implementation, the World Bank decided to migrate its online system for the tracking and monitoring of PAACs in all its projects, and began using Systematic Tracking of Exchanges in Procurement. Description of Arrangements for the Project’s Future Operations 49. Given that MINEDU and the World Bank evaluated the successful implementation of the Education SWAp in each World Bank supervision mission meeting, the World Bank suggested to the ministry’s senior officials in early 2017 that this program be expanded in 2018. While the expansion of its projects currently under way does not involve any complex processes for the World Bank, this does mean a new debt transaction for the GoP. As such, the decision on whether or not to expand involved the MEF, whose new administration decided to reduce loans of the kind that support programs such as the Education SWAp. As a result, it was decided not to expand the program, instead concentrating support to the education sector by international cooperation on higher education. 25 An electronic repository of validated items (questions) to be selected for standardized tests. Page 66 of 72 The World Bank Basic Education Project ( P123151 ) 50. In any event, the MINEDU team’s capacity for assessing learning and pedagogical management, which has been strengthened under the program, will continue to serve as a resource for the ministry in the coming years. On the other hand, as noted above, one of the achievements of the Education SWAp was the validation of innovative assessment instruments, which will also be useful in the coming years. Page 67 of 72 The World Bank Basic Education Project ( P123151 ) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) Álvarez, Jesús, Vicente García Moreno and Harry Anthony Patrinos. 2007. Institutional Effects as Determinants of Learning Outcomes: Exploring State Variations in Mexico. Policy Research Working Paper; No. 4286. World Bank, Washington, DC. © World Bank. https://openknowledge.worldbank.org/handle/10986/7475 Barber, M., and M. Mourshed. 2007. How the World’s Best-Performing School Systems Come Out on Top. London: McKinsey. http://mckinseyonsociety.com/downloads/reports/Education/Worlds _School_Systems_Final.pdf. Bloom, N., R. Lemos, R. Sadun, and J. Van Reenen. 2014. “Does Management Matter in Schools.” NBER. Working Paper 20667. Braun, Henry and Anil Kanjee. 2006. Using Assessment to Improve Education in Developing Nations, in Braun, Henry, Anil Kanjee, Eric Bettinger and Michael Kremer (eds.) Improving Education Through Assessment, Innovation, and Evaluation, Cambridge MA., American Academy of Arts and Sciences. Bruns, Barbara, Deon Filmer and Harry Anthony Patrinos. 2011. Making schools work: new evidence on accountability reforms. World Bank, Washington, DC. © World Bank. http://siteresources.worldbank.org/EDUCATION/Resources/278200- 1298568319076/makingschoolswork.pdf Castro, J. F. and G. Yamada. 2010. “Educación superior e ingresos laborales: estimaciones paramétricas y no paramétricas de la rentabilidad por niveles y carreras en el Perú”, DD/10/06, Universidad del Pacifico. Castro, Juan F., Gustavo Yamada and Omar Arias. 2014. Higher Education Decisions in Peru: On the Role of Financial Constraints, Skills, and Family Background. Centro de Investigación de la Universidad del Pacífico. Documento de Discusión DD/11/14. Cunha, F., J.J. Heckman, L.J. Lochner and D.V. Masterov. 2006. Interpreting the evidence on life cycle skill formation, in: Hanushek, E.A. and F. Welch (eds.) Handbook of the Economics of Education, Amsterdam, Elsevier, 697-812. E. Schiefelbein, L. Wolff, and P. Schiefelbein. 1998. “Cost-Effectiveness of Education Policies in Latin America: A Survey of Expert Opinion.” Elliot, S. and P.J. Branden. 1997. Educational assessment and accountability for all students: Facilitating the meaningful participation of students with disabilities in district and statewide assessment programs. Wisconsin Department of Public instruction. Madison, Wisconsin. Glewwe, Paul. 1996. The relevance of standard estimates of rates of return to schooling for education policy: A critical assessment. Journal of Development Economics, 1996, vol. 51, issue 2, 267-290. Green, David and W. Craig Riddell. 2001. Literacy, Numeracy and Labour Market Outcomes in Canada. Canadian International Labour Network Working Papers. Hanushek, E. A. 2011. “The Economic Value of Higher Teacher Quality.” Economics of Education Review 30: 466–79. Hanushek, Eric A. and Ludger Woessmann. 2007. The Role of Education Quality for Economic Growth. Policy Research Working Paper; No. 4122. World Bank, Washington, DC. © World Bank. Page 68 of 72 The World Bank Basic Education Project ( P123151 ) https://openknowledge.worldbank.org/handle/10986/7154 License: CC BY 3.0 IGO. Hanushek, Eric A. and Margaret E. Raymond. 2004. Does School Accountability Lead to Improved Student Performance? NBER Working Paper No. 10591. http://www.nber.org/papers/w10591 Heckman, James J. 2000. Policies to foster human capital. Research in Economics, Elsevier, Elsevier, vol. 54(1), pages 3-56, March. INEI (Instituto Nacional de Estadística e Informática). 2016. Perú: Brechas de Género, 2016: Avances hacia la igualdad de mujeres y hombres. INEI Lazear, Edward P. 2003. Firm Specific Human Capital: a skill-weights approach. NBER. Working Paper 9679. http://www.nber.org/papers/w9679 Linn, Robert L. and Joan L. Herman. 1997. A Policymaker's Guide to Standards-Led Assessment. Office of Educational Research and Improvement (ED), Washington, DC. Moll, Peter G. 1998. Primary Schooling, Cognitive Skills and Wages in South Africa. Economica, Volume65, Issue258, May 1998, Pages 263-284. Murnane, Richard, John B. Willett and Frank Levy. 1995. The Growing Importance of Cognitive Skills in Wage Determination. The Review of Economics and Statistics, 1995, vol. 77, issue 2, 251-66. Nasser, Ramzi N. 2007. Does Subjective Class Predict the Causal Attribution for Poverty? Journal of Social Sciences 3 (4): 197-201, 2007. ISSN 1549-3652. Ñopo, Hugo. 2012. New Century, Old Disparities: Gender and Ethnic Earnings Gaps in Latin America and the Caribbean. Latin American Development Forum; Washington, DC: Inter-American Development Bank and the World Bank. © Inter-American Development Bank. https://openknowledge.worldbank.org/handle/10986/11953 License: CC BY-NC-ND 3.0 IGO. Patrinos, Harry Anthony and Chris Sakellariou. 2008. Quality of Schooling, Returns to Schooling and the 1981 Vouchers Reform in Chile. Policy Research Working Paper No. 4617. World Bank, Washington, DC. © World Bank. https://openknowledge.worldbank.org/handle/10986/6721 Patrinos, Harry and George Psacharopoulos. 2010. Returns to Education in Developing Countries. International Encyclopedia of Education. 305-312. 10.1016/B978-0-08-044894-7.01216-1. Psacharopoulos, George and Harry Patrinos. 2004. Returns to investment in education: a further update. Education Economics, 2004, vol. 12, issue 2, 111-134. Romero, M., J. Sandefur, and W.A. Sandholtz. 2017. Can a Public-Private Partnership Improve Liberia’s Schools? Center for Global Development. https://www.cgdev.org/sites/default/files/can-public-private-partnership- improve-liberias-schools.pdf Stecher, Brian M. 2002. Consequences of large-scale, high-stakes testing on school and classroom practice, in Hamilton, Laura S., Brian M. Stecher and Stephen P. Klein (eds.) Making Sense of Test-Based Accountability in Education. RAND Corporation. http://www.jstor.org/stable/10.7249/mr1554edu.11 Vegas, Emiliana and Jenny Petrow. 2007. Raising student learning in Latin America: the challenge for the twenty-first century (English). Latin American development forum. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/709361468277132582/Raising-student-learning-in-Latin- America-the-challenge-for-the-twenty-first-century Page 69 of 72 The World Bank Basic Education Project ( P123151 ) Woessmann, Ludger. 2003. Schooling Resources, Educational Institutions and Student Performance: The International Evidence. Oxford Bulletin of Economics and Statistics, Volume65, Issue2, May 2003, Pages 117- 170. World Bank, Project Appraisal Document, Basic Education Project, Report No. 66918, dated December 14,2012. World Bank, Basic Education Project, Implementation Status Reports and Mission Aide Memoires (2102–2018). World Bank, Project Paper, Basic Education Project, First Project Restructuring, Report No. 13494, dated September 17, 2015. World Bank, Project Paper, Basic Education Project, Second Project Restructuring, Report No.27794, dated June 7, 2017. World Bank, Country Partnership Framework for the Republic of Peru for the Period FY17–FY21, Report No. 112299-PE dated April 4, 2017. Yamada, Gustavo. 2007. Retornos a la educación superior en el mercado laboral: ¿Vale la pena el esfuerzo?. Documento de Trabajo 78 CIUP /CIES. Page 70 of 72 The World Bank Basic Education Project ( P123151 ) ANNEX 7: DISBURSEMENT LINKED INDICATORS (ACTUAL) Disbursements Linked to DLIs Baseline and DLI # Indicator 1 5 6 7 8 Final Target 2 3 4 December September November April 2018 July 2018 April 2016 July 2016 March 2017 2015 2017 2017 PDO Indicator 3: School US$1,875,000 US$375,000 Baseline: 0 management positions 1 Final Target: staffed through the school 25,000 15,000 3,017 principal evaluation system. IRI 1: Student assessments US$1,562,500 US$1,562,500 for new grades of Primary Baseline: 0 2 Education and Secondary Final Target: 4 Education levels validated in 2 2 a pilot study. IRI 2: Additional grades at the Primary Education and US$781,250 US$781,250 US$781,250 Secondary Education levels Baseline: 0 3 in which sound national Final Target: 4 assessments are administered in at least two 1 1 1 subject areas. IRI 3: Results of national student assessments in new US$1,041,670 US$1,041,670 US1,041,670 grades at the Primary Baseline: 0 4 Education and Secondary Final Target: 3 Education levels reported at 1 1 1 the regional and school level. IRI 4: National surveys carried out to evaluate child US$1,562,500 US$1,562,500 Baseline: 0 5 development progress and Final Target: 2 the quality of public Preschool Education 1 1 Page 71 of 72 The World Bank Basic Education Project ( P123151 ) Disbursements Linked to DLIs Baseline and DLI # Indicator 1 5 6 7 8 Final Target 2 3 4 December September November April 2018 July 2018 April 2016 July 2016 March 2017 2015 2017 2017 services. IRI 5: Borrower’s (through US$1,041,670 US$2,083,330 Baseline: 1 MINEDU) participation in 6 (PISA 2012) international assessments of Final Target: 3 TERCE PISA/ICCS student learning outcomes. IRI 6: National studies of classroom instructional US$1,041,670 US$1,041,670 US$1,041,670 practice in Primary Baseline: 1 7 Education and/or Secondary Final Target: 3 Education levels, as the case 1 1 1 may be. IRI 7: Results of national US$191660 US$589,590 US$1,562,500 US$781,250 US$10 studies of classroom Baseline: 1 8 instructional practice Final Target: 4 reported at the national 0.75 0.25 2 1 level. IRI 11: School principals Baseline: 0 US$1,656,230 9 certified in school Final Target: management. 12,000 13,536 Total estimated disbursement ( US$) 5,400,000 5,850,000a 5,468,750 1,041,670 2,604,170 1, 041,670 2,031,240b 1,562,500 Note: a. A single disbursement report was submitted by the client providing evidence for the achievement of US$11.25 million worth in Disbursement-Linked Indicators (DLSs) in years 2013 and 2014. At the time (December 2015), the client presented over US$5 million in eligible expenses, thus the disbursement from December 2015 was completed in April 2016 when the client presented the equivalent amount in eligible expenses. However, due to some system rounding up, the actual disbursement was US$10 less than what the client requested. b.US$10 correction for undisbursed amount requested in April 2016. Page 72 of 72