ICRR 13486 Report Number : ICRR13486 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/17/2011 PROJ ID : P088728 Appraisal Actual Project Name : Mexico Education US$M ): Project Costs (US$M): 345.65 350.0 Quality Country : Mexico Loan/ Loan /Credit (US$M ): US$M): 240.0 240.0 Sector Board : ED US$M ): Cofinancing (US$M): 0 0 Sector (s): Secondary education (40%) Primary education (40%) Pre-primary education (18%) Central government administration (2%) Theme (s): Education for all (40% - P) Participation and civic engagement (20% - S) Managing for development results (20% - S) Access to urban services and housing (20% - S) L/C Number : L7347 Board Approval Date : 12/13/2005 Partners involved : Closing Date : 07/31/2009 12/31/2009 Evaluator : Panel Reviewer : Group Manager : Group : Judith Hahn Gaubatz George T. K. Pitman IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: The project was the first phase of a longer term three-phase Adjustable Program Loan (APL), in which the ultimate objective is improving learning outcomes over the period 2005-2015. According to the Project Appraisal Document (PAD, page 6), the project objective was "to improve public basic school management through strengthening and expanding the Government's Quality Schools Program (Programa Escuelas de Calidad - PEC)." The Loan Agreement articulates the project objective as follows : "to support the Borrower's efforts to increase the quality of basic public education through the strengthening of the Borrower's schools' autonomy ." As the PAD objective statement is more specific and thus more monitorable, it is used as the basis for this review . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): tota l project cost : US$324 (1) School Grants (Appraisal total 324 .7 million; Actual : not available . The Bank planned to US$ 324. 234 .49 million; Actual was US$ 238. contribute US$ 234. 238 .75 million ): This component financed the federal share of the PEC school grant program. The PEC program was intended to empower school communities (including teachers, parents, and principals) and promote school autonomy by providing school improvement grants . School communities were expected to establish participatory school management practices by designing and overseeing a Strategic School Transformation Plan (Plan Estrategico de Transformacion Escolar - PETE) that responded to the needs of the school and its students . US$ 15. (2) Program Monitoring and Oversight (Appraisal total : US$1515 .1 million; Actual : not available . The Bank neither planned nor contributed to this component ): This component financed program oversight an d information dissemination by the Secretariat for Public Education (Secretario de Educacion Publica - SEP). Activities included maintenance of the national management information system (Sistema de Informacion del Programa Escuelas de Calidad - SIPEC), implementation support to State Education Agencies (Autoridades Educativas Estatales - AEE) such as supervision visits and technical seminars, and program dissemination through media and special publications. US$ 5.8 million; Actual : not available . The (3) Policy Development and Evaluation (Appraisal total project cost : US$5 Bank planned to contribute US$ 4.91 million; actual was US$ 0.65 million ): This component financed policy d evelopment and evaluation activities including the design and field test of evaluation methodology, a baseline survey, and piloting of strategies to encourage applications from lower -income urban areas. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost : The ICR reports that Borrower financial obligations were fulfilled . The project team reports that the actual project cost was US$350.0 million, with states contributing approximately 25% of the total cost. Only US$80,000 was used for Component 3 and the remaining amount was reallocated to Component 1. No explanation is given in the ICR . Financing : The Bank's Loan was fully disbursed . Dates : The project closing date was extended from July 2009 to December 2009. According to the project team, this was done largely to allow the program time to complete studies and finalize on -going technical assistance . It also helped coordinate with the Mexican fiscal year and ensured the continuation of the project staff before the new project (Phase II) was launched. All of the major disbursement had been completed . Triggers between phase I and phase II : Number of schools participating in the Program (including schools that graduate ) will increase by 50 Percent Program schools (including schools that graduate ) as a percentage of all public basic education schools will increase to 15 percent Completion of baseline data collection for impact evaluation Government approval of the Program expansion plan with specific targets for expansion . 3. Relevance of Objectives & Design: The relevance of the project objectives is rated Substantial . Although the country had made significant progress in expanding overall access to education, access was still lower in rural and indigenous communities, and the quality of education was inadequate as reflected by low achievement levels . The Government's National Development Plan for 2007-2012 prioritizes efforts to promote social inclusion and specifically endorses the PEC program . The Bank's Country Partnership Strategy for FY 2008-2013 points to improving the quality of education as a key strategy for promoting social inclusion and reducing poverty . The relevance of the project design is rated Substantial . The Government introduced the PEC program in 2001 through a pilot program of 2,000 schools, and subsequently requested the Bank's support to expand the program nationwide. The program design was based on extensive analytic work which found that increased school autonomy and increased accountability through community involvement were important factors that affected learning outcomes . By having communities identify needed school improvements, prepare and oversee the implementation of school improvement grants, the program aimed to improve school climate and ultimately, learning outcomes . This project was the first phase of a three -phase APL, in which the first phase would increase participation in the program; the second phase would fine tune the model and start showing improvements in intermediate outcomes such as reduced drop-out rates; and the third phase would see maturation of the program and actual improvements in learning outcomes. The results framework appropriately identified longer -term outcomes for the third phase of the program (such as learning improvements) as well as intermediate outcomes for the first phase such as increased coverage of the program and increased community participation (although, as discussed in the ICR, the indicator for measuring the latter outcome was inadequate given the Project's goal to increase focus on poorer schools ). It also included the establishment of baseline data as a trigger for the second phase . PEC originally adopted a top-down almost “cookie-cutter� approach that required undifferentiated application of the Program’s systems and processes across all levels . In the roll out and adoption of SIPEC to all the AEEs, it soon became apparent that their capacities varied greatly and there would be a need to adjust the one -size-fits-all approach. As a result many states were not able to adopt SIPEC (ICR page 19) and the Government proposed to allow states to use existing or develop new school -based information systems. Thus an accreditation system for individual state-level MIS was financed to ensure that these alternative systems ensured minimum requirements with respect to school-based reporting. One issue in the Project’s design that had not been contemplated was the large financing for school grants in the first six months of the project with nearly half of the loan had been disbursed by mid -2007. As a result, loan disbursements were out of synchronization with the PEC ’s capacity to monitor and evaluate the Project ’s progress, and thus disbursement had to be slowed down while "while agreeing upon changes internally in PEC ’s management structure and a plan to bring monitoring back on track in an accelerated manner " (ICR page 20). 4. Achievement of Objectives (Efficacy): To improve public basic school management through strengthening and expanding the Government's Quality Schools Program : Substantial . Achievement of the objective is rated substantial, due to a substantial expansion of the program and some evidence of strengthening of the program. (Note: Although the ICR does not provide specific information on the surveys conducted in PEC schools, the project team reports that the survey covered 820 schools in 12 (out of 32) states, including a range of education level and target population (e.g. indigenous schools or technical schools ). Expanding the School Program 40,790 basic education schools were participating in PEC by 2009, a doubling from 21,500 schools in 2005/06 (target was a 50% increase). The ICR reports (page 25) that the total amount financed in the school grant component was US$ 238.75 million. The ICR notes that because recipients of grants spent most on infrastructure, PEC is changing its Operating Rules under PEC II to ensure that a larger share of school grants are used to acquire education inputs . The proportion of schools participating in PEC, out of the total number of basic education schools, increased from 10.3% in 2005/06 to 20.35% in 2009, surpassing the target of 15.8%. Of these, almost 69% were located in areas of medium, high or very high marginalization . (While PEC has made a strong effort to reach poor schools through its focus on targeting, it did not directly address the issue of equity during Phase I . A total of 68.58% of PEC schools are in highly marginalized or marginalized localities compared to 77.37% of schools as a whole. Likewise, 31.42% of PEC schools are located in areas with low or very low levels of marginalization, compared to 22.63% of all schools. However, when analyzing this data on a per -student (as opposed to school) basis, the distribution of students in PEC and non -PEC schools becomes quite similar because schools in poor areas tend to be substantially smaller than schools in other areas, reducing concerns about equity .) Although the Project did not specifically target the indigenous, it exceeded the goals specified in the Indigenous Peoples Development Plan (IPDP). At the time PEC was launched in 2001, there were only 27 indigenous centers (schools with 70% or more of indigenous students ) participating in the Program, out of a total of 2,240 PEC schools. The IPDP aimed that the PEC’s indigenous centers would comprise 4% of PEC schools. By 2009, PEC covered a total of 40,790 schools, of which 2,950 were indigenous centers, equivalent to 7% of the total PEC schools. Strengthening the School Program A baseline survey for the impact evaluation was completed . According to the project team, seven studies were conducted, including "Design of Material to Promote Social Participation " and "Developing a New Model for the Program's Fiduciary Arrangements." The percentage of community members that were familiar with their PETE increased from 62% to 76.8% by the project's end. The percentage of community members that had participated in the design or adjustment of their PETE increased from 51% to 74.2% by the project's end. The percentage of community members that observed participatory decision -making among parents, teachers, and principals about school affairs increased from 82.0% in 2005 to 90.2% in 2009. 75% of states were providing timely reports of the program through SIPEC or other acceptable, comparable systems. No baseline figure was available at appraisal . The ICR reports (page 15) that scores in Enlace (an achievement test at the national level ) have been increasing faster in PEC schools than in non -PEC schools. No specific data are provided, although the project team reports that "it is a well-established fact that (a) ENLACE scores have been increasing nationwide and (b) PEC schools have seen a faster increase than non -PEC schools." The project team also reports that "PEC is now regarded as the premier school autonomy program in Mexico and is taken seriously in the education sector (this was not the case in 2005)". However, The indicator intended to measure commitment to the goals and activities of the PETE - local financial contributions to the schools - was not met. The average local contribution per school decreased from 30,168 pesos in 2005 to 8,170 pesos. The ICR reports that this was due to PEC's focus on incorporating poorer and smaller schools and cites the increase in community participation in the design of the PETE as a more appropriate indicator of commitment. However, increased participation reflects only one dimension of improved school management and there is limited evidence provided for other dimensions . The percentage of community members who observed teachers encouraging and supporting student performance decreased from 95% to 72% by the project's end. The ICR reports that there was a change in definition since the baseline was established to measure actual observation of positive practices by teachers . This change led to a "more conservative estimate of community observation of teachers " (ICR page iv). However, no alternative evidence is provided to demonstrate whether positive practices by teachers increased to reflect improved school management . Overall, all triggers for moving to the next phase were met and Phase II was approved by the Bank in June 2010. 5. Efficiency (not applicable to DPLs): Efficiency of the project is rated Modest , due to lack of evidence confirming estimated efficiency gains . The ICR notes that as this project is part of a three -phase APL program, a full economic analysis will most likely be prepared at the final phase. However, although complete information on efficiency gains in the education system was not yet available, the ICR does not present data on other aspects of project efficiency . The PAD presents estimates of the yearly savings for the education system, based on decreases in the school drop-out rate (by 0.05%) and repetition rate (by 3.7%). The latter rate reflects the number of students that take longer than normal to complete a cycle of education (i.e. higher recurrent costs for the education system ). Based on the project's support for 20,000 schools (with 4.9 million students) per year during the project period, the estimated savings for the education system was US$ 12.6 million. The actual number of schools supported met the targeted amount; however, there were no data confirming the actual decreases in drop -out or repetition rates due to the long-term nature of the program. (The ICR cites a 2005 study that found that participation in PEC reduced dropout rates by 0.24%, but repetition rates by only 0.31%.) In addition, private financial contributions to schools, which were expected to increase alongside increased community involvement and thus reduce public cost to the education system, in fact decreased significantly by the end of the project. However, this not so much a failing as a reflection of inclusion of many smaller schools in marginal areas compared with the baseline . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The overall outcome of the project is rated Moderately Satisfactory . The project objective and design were substantially relevant as they were likely to lead to increased community participation in schools in the short -term, and to improved learning outcomes in the longer -term. The objective to expand and strengthen school management was substantially achieved, due to a substantial increase in coverage of the program and some evidence of strengthening of the program . However, there was limited information on the efficiency of achieved objectives . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The Government and local communities continue their strong commitment to (and demand for) the PEC program. An APL Phase II loan for US$220 million has been approved by the Bank and incorporates lessons learned from Phase I, including ensuring better understanding of the PETE process by stakeholders and increased coordination of the different federal government programs at the basic education level . a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: Quality at Entry The project design drew upon extensive sector work and a comprehensive social assessment, as well as lessons from similar school -based management programs in Latin America and in other regions . Triggers for moving forward to Phase II were concise and set the way for program expansion as well as program assessment. Risk assessment was extensive, although two key risks were not anticipated but subsequently materialized: grant disbursement outpaced technical assistance in monitoring the project, and some states had difficulty adopting and utilizing SIPEC . The ICR notes that due to these institutional capacity issues, "the project faced issues with timely receipt of financial and other information, " although the project team reports that outcomes were not ultimately affected . Supervision The Bank team demonstrated flexibility and responsiveness during the supervision period . It responded to the varying capacity levels of the states by allowing states to provide monitoring information through their own existing system . At the same time, the team ensured that minimum standards were being met by developing an accreditation system for state -level information systems. The ICR notes (page 10) that initially, "PEC's ability to monitor progress at the school level was weak, but with Bank support and supervision, a concerted effort was made to improve data collection ." Eight supervision missions were carried out, which aided in continual improvements of the project . For example, an Action Plan was developed to strengthen financial management while disbursements against school grants was informally suspended by the Borrower . at-Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government The Government demonstrated strong commitment to expanding and consolidating gains from the initial phase of the PEC program, reflected in part by fulfillment of its financial obligations to the program . Implementing Agency The Secretariat for Public Education (SEP) administered the project at the national level and functioned effectively in disseminating information about the program and disbursing grants . The primary issue that arose - the outpacing of monitoring capacity versus grant disbursements - was effectively addressed when the Secretariat for Finance informally suspended disbursements against school grants to allow PEC to strengthen its financial management capacity . At the state level, the 32 State Education Agencies varied in their institutional capacity, as reflected in the difficulty of some states in adopting the SIPEC . This approach was adjusted accordingly, in which those states with more limited capacity could provide financial and monitoring information through their own existing systems . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design : The M&E framework included indicators for both long -term and intermediate outcomes. Although, as noted previously, the key indicator for measuring commitment of local communities was flawed, it is being adjusted for subsequent phases of the project . Technical assistance ensured that monitoring systems were strengthened to monitor progress and that evaluation activities would be able to measure impact by the end of the three phases of the APL. M&E Implementation : Indicators were refined as needed during project implementation . As discussed previously, State Education Agencies varied in their institutional capacity to utilize SIPEC and requirements were adjusted accordingly. The baseline survey for the impact evaluation was completed, although the Borrower's ICR notes that this activity was delayed due to the failure to take into account the Government's capacity and resources to carry it out. M&E Utilization : The ICR reports that some of the M&E-related outputs were used in the design of the Program's next phase and will also be used for the development of a detailed impact evaluation . M&E information has also been used to measure the impact of PEC's ability to reach poor and indigenous communities, to measure PEC's impact on social communication, pedagogical practices and accountability, and to assess efficiency in monitoring of PEC schools. a. M&E Quality Rating : Substantial 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards : This was an environmental category C project . The Indigenous Peoples' safeguard policy (OP 4.10) was triggered. Therefore, an Indigenous Peoples' Development Plan was prepared for the project, which identified the main issues affecting access to basic education and incorporated expectations identified during the consultation process. The ICR reports that the project surpassed the target for the number of indigenous schools participating in PEC. Fiduciary : As discussed previously, the varying capacity levels of the state education agencies affected the timeliness of financial and project monitoring reports . The ICR reports that there were no other major financial management issues. All Audits were completed and there were no issues raised . Procurement : An in-depth procurement review carried out in 12 states by an independent firm showed that PEC schools were overall complying with procedures and highlighted some specific areas for improvement . The ICR reports that these areas were successfully addressed under the Action Plan, including recommendations for technical assistance to schools on procurement guidelines . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately The project objectives and design were Satisfactory substantially relevant, and the objectives to improve school management by strengthening and expanding the school program were substantially achieved. However, there was limited information on efficient achievement of objectives. Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Moderately There is no disagreement. The ICR Satisfactory also rates Quality at Entry as Moderately Satisfactory, and per the Harmonized Evaluation Criteria it should also have rated Bank Performance as Moderately Satisfactory. Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Flexible information and monitoring system requirements helped to respond to the Program's information needs, while being responsive to the varying capabilities of states and schools . An APL is a useful instrument to support implementation of a program where long -term involvement is necessary to produce expected development outcomes . Indicators for measuring progress of the early phases should be realistic and measurable, with an eye towards measuring long -term outcomes. Project design should consider mechanisms for keeping fast -disbursing components at pace with relatively slow-progressing technical assistance components . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: ICR Quality is rated Satisfactory . The ICR is overall consistent with guidelines; however, there is no information on actual project costs, and the some of the data in Annex 2 do not match the data in the Results Framework Analysis (for example, the percentage of community members participating in the design or adjustment of PETE or the percentage of community members observing teachers encouraging and supporting student performance ). There are no details provided on the survey which provided some of the key data on outcomes . a.Quality of ICR Rating : Satisfactory