Document of The World Bank Report No.: 21545 FOR OFFICIAL USE ONLY PERFORMANCE AUDIT REPORT CHILE PRIMARY EDUCATION IMPROVEMENT PROJECT (LOAN 3410-CH) March 21, 2001 Sector and Thematic Evaluation Group Operations Evaluation Department This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Currency Equivalents (annual averages) Currency Unit = Chilean Peso $1.00 = Ch. 525 (August 2000) $1.00 = Ch. 339 (March 1991) Abbreviations and Acronyms GDP Gross domestic product ICR Implementation Completion Report MINEDUC Ministry of Education MIS Management information system NGO Nongovernmental organization OED Operations Evaluation Department PCR Project Completion Report PIU Project implementation unit SAR Staff Appraisal Report SIMCE Sistema de Informaci6n sobre la Calidad de la Educaci6n Fiscal Year Government of Chile January 1 - December 31 Director-General, Operations Evaluation Mr. Robert Picciotto Director, Operations Evaluation Department Mr. Gregory K. Ingram Manager, Sector and Thematic Evaluation Mr. Alain Barbu Task Manager Ms. Helen Abadzi FOR OFFICIAL USE ONLY The World Bank Washington, D.C. 20433 U.S.A. Office of the Director-General Operations Evaluation March 21, 2001 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Chile-Performance Audit Report Primary Education Improvement Project (Loan 3410-CH) Objectives. The project was to: (a) expand the coverage and enhance the quality of preschool education and reduce late entry, repetition, and dropout in primary schools, especially for 5-year-olds living in poverty conditions; (b) strengthen the institutional capacity of the Ministry of Education offices, as well as that of the education departments of the municipalities to direct and manage a decentralized education system, and improve the managerial skills of the municipal and private-subsidized preschool and primary school administrators; and (c) assess alternative cost-effective approaches to meet the secondary education needs of primary education graduates. The project financed institutional development as well as preschool, primary, and secondary education through civil works, technical assistance, and salaries. Total project cost was $216.2 million; it was completed as estimated, in six years on June 30, 1998. Outcomes. The project attained most of its targets and in many instances exceeded them. Though it did not profoundly change the operation of the Ministry of Education, it helped significantly to develop its institutional, managerial, and financial capacity. It produced a significant impact on the quality of education at the classroom level of municipal and rural schools, as reflected in achievement test results. The school-based quality improvement program gave grants to schools interested in stimulating the energy and creativity of teachers. A special education component improved the quality of education for children with various special needs. Preschool education was expanded to cover 29.6% of the eligible population (from 21%). All children were screened for health risks. A computer network was extended to half of the public primary schools. Relevance and Efficacy. The project was consistent with a strategy of developing human resources in Chile and highly relevant to its economic development needs. It used World Bank resources efficiently, both in terms of lending as well as in terms of knowledge acquisition. The Operations Evaluation Department (OED) rates the project as follows. Audit ICR Outcome Highly Satisfactory Highly Satisfactory Sustainability Likely Likely Institutional Development Impact Modest Modest Borrower Performance Highly Satisfactory Highly Satisfactory Bank Performance Satisfactory Highly Satisfactory Lessons The organization of Chilean education is a model that countries may aspire to reach. The successes of the country are to some extent due to income, political stability, low corruption, social conscience, and a This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. 2 culture of affectionate relationships. Nevertheless, other countries can learn from the Chilean successes. The following issues would be of utility: Development of the affective domain. Developing students' interests and value systems is likely to increase the probability that they will retain information and go on to higher levels of schooling. The psychomotor activity of children moving rather than sitting and listening passively may influence the neurotransmitters that improve the mood and create multiple bridges to the information in the brain. Overall, it is probable that unhappy and unattractive schools are one reason behind student absenteeism and dropout. Teachers in other countries might be specifically trained to show affection towards students, petting their heads or hugging them, rather than beating them, as sometimes happens. School appearance. After consultations with the local population regarding attractiveness, specific low- cost items that beautify the school might be considered. Controlled research (questionnaires and class observations) may measure the importance of school appearance on dropout and attendance and the cost- benefit implications. Importance and selective utility of group work students. More research is needed in understanding where group work is beneficial and where it is not. Classroom furniture must consist of square tables that make it possible to rearrange. Searching for answers. When materials are available, searching for answers individually or in groups may help develop the all-important research skills needed in the 21st century. Conversely, in countries where enrichment materials are not woven into the curriculum, they may lie unused or be stolen. Empowering group activities for teachers. Creating competitions among teachers of various areas that require group diagnosis and problem solving may be a way to stimulate interest in school work, which many teachers find unappealing. School vouchers targeted to the very poor. In many countries, as in South Asia, governments pay teacher salaries for NGO or other private schools. But these subsidies are often not targeted to the poor and are independent of achievement levels. These governments might use their funds better (assuming special interests permit this) by giving subsidies based on numbers of poor parents willing to send children to schools and on children's achievement. Lessons pertinent to the Bank's operations from this audit are: The same procurement rules must apply equally to all Bank borrowers. Latin America and South Asia should not use interpretations that differ to such an extent that some countries are forced to spend years preparing for international competitive bidding and others exempted. Burdensome procurement rules exhaust the time and resources that countries may devote to implementation and detract from project substance. Bank appraisals and supervisions should emphasize classroom-level instructional delivery issues at least as much as they emphasize organizational and economic issues. Attachment Contents Principal R atings...........................................................................................................................iii K ey Staff Responsible...................................................................................................................iii P reface............................................................................................................................................. V 1. Background: School Choice and Social Consequences........................................................1 Bank and Other Donor Involvement ........................................ 2 Bank and Government Sector Policies ......................... ............... 3 2. Project Implementation Experience .................................................................................. 4 Project Outcomes in Classrooms - Mission Visits............................... 4 Many Schools Can Be Considered "Effective ......................... 5 Affective Domain of Student Learning: Attention to this Usually Neglected Area ............................... .............. 6 Cognitive Domain of Student Learning: A Need to Focus More on Objectives and Skills.... ................................. 7 Teacher In-service Training ................................................ 9 School Nutrition and Health ............................................ 10 Best Practice Health Care Provision ......................................... 10 Relevance and Efficiency .................11.... .................. . Institutional Development Impact..............1.........................1 Sustainability. ............................ ......................... 12 Bank Performance.......................................... ........ 12 Borrower Performance ................................... ........... 13 3. Issues for Future Consideration....................................................................................... 13 Need to Improve Basic Skills and Verify Effects .................. .......... 13 The Inequity of School Choice Continues .................................... 14 4. L essons .................................................................................................................................... 16 R eferences ..................................................................................................................................... 19 Annex A. Basic Data sheet........................................................................................................... 21 Annex B. Project Activities ...................................................................................................... 23 Annex C. Comments from the Government.......................................................................... 25 This report was prepared by Helen Abadzi, who audited the project in August 2000. William Hurlbut edited, and Pilar Barquero provided administrative support.  Ill Principal Ratings Audit ICR Outcome Highly Satisfactory Highly Satisfactory Sustainability Highly Likely Likely Institutional Development Substantial Modest Borrower Performance Highly Satisfactory Highly Satisfactory Bank Performance Highly Satisfactory Satisfactory Key Staff Responsible Project Task Manager/Leader Division Chief/ Country Director Sector Director Appraisal Luis Pisani Jim Adams Ping Loh Completion William Experton Xavier Coll Myrna Alexander  V Preface This Performance Audit Report (PAR) covers the Primary Education Improvement Project (LN. 341 0-CH), the most recent completed education project in Chile. It also discusses the development of the borrower and Bank strategy for this and for subsequent projects as well as lessons for other countries from the Chilean experience. The audit was conducted to study the effectiveness of the Bank's assistance to the education sector of a country that was emerging from a long period of political and social oppression. The PAR is based on the following sources: Implementation Completion Report (ICR), Staff Appraisal Report (SAR), Loan Agreement for the projects, and project files, particularly the supervision reports. An Operations Evaluation Department (OED) mission visited Chile in August 2000 to collect other pertinent information. The author thanks the many government officials and researchers for their extensive cooperation. Following standard OED procedures, copies of the draft PAR were sent to the relevant government officials and agencies for their review and comments. A number of observations were made, which have been incorporated into the PAR as Annex C.  1 1. Background: School Choice and Social Consequences 1.1 While visiting Chilean primary schools that cater mostly to the poor, one is often struck by children's joy. Multiple activities go on in classrooms, young students carry out work in groups with limited supervision, and they ask questions to strangers. There is much artwork, singing, and dancing, and teachers are affectionate with the children. This atmosphere is in rather striking contrast with the somber and rather orderly classes of many other countries. How did this instructional environment come about and what are its outcomes? 1.2 Chile (per capita income of US$4,820 in 1997) achieved universal coverage in the 1960s; it has about 30.3% enrollment in preschool, 87% in secondary education, and 26% in tertiary education. Since 1978, it has been able to monitor school-level quality through achievement tests in language, math, social studies, and science administered nationally to grades 4 and 8. The Sistema de Informaci6n sobre la Calidad de la Educaci6n (SIMCE) provides class scores for guidance in achievement of objectives and planning, a system few countries have been able to put in place. Though it still needs some technical improvements in development methodology and score presentation, SIMCE serves as quality control. Schools' scores are widely disseminated to the public, and decreases or increases are widely discussed. 1.3 Chilean education has attracted worldwide attention due to its 1980s reform, which centered on school choice. The military regime of Augusto Pinochet decentralized school management to provinces and municipalities, which assumed many important decisions, including the provision of inputs and employment of teachers.1 All parents could obtain vouchers (worth 26,000 pesos per month in 2000 to recipient schools) and send children to certain eligible private schools if they could obtain admission and afford the additional tuition. (Eligible schools are those which agree to meet certain criteria and receive public financing.) Many documents were written regarding these controversial policies, whose application is also seen as very desirable by conservative politicians in the United States.2 1.4 However, the outside world has paid relatively little attention to the social segmentation and "creaming" that this experiment has produced. About 5-9% of students with the highest socioeconomic status and "social capital" go to private schools that receive no public financing and therefore cannot accept vouchers.? About 45% of the students take advantage of the vouchers if they can be admitted and pay. When additional financing is minimal, many low-income children attend religious and other nongovernmental organization (NGO) schools that would otherwise be unaffordable. The remaining 55% of the students must go to the fully financed 1. Repfblica de Chile. 1988. 2. A laudatory reference is West, Edwin. 1996. Perry, G, and D. Leipziger (Eds). 1999. Chile: Recent Policy Lessons and Emerging Challenges. World Bank Institute, Washington, DC. Critical references are included, such as Mizala and Romaguera (1998) for Chile, Carnoy and McEwan (1997) for the US. 3. Ninety two percent of the students in Chile are in schools that receive some level of state financing, either complete or partial. 2 public schools found in municipal and rural areae. These children tend to be from poor families and/or have behavioral or learning problems. (In small towns with few private schools, municipal schools may also serve middle-class children, particularly in higher grades.) Ironically, decentralization creates more inequity because teachers are also "creamed." Wealthier municipalities may add bonuses to salaries and attract the best. So in areas like periurban Santiago, many teachers hold two jobs to make ends meet while they deal with problem children that require special attention. 1.5 The democratic government that took over in 1989 had multiple problems to deal with, and it was more practicable to leave the voucher system standing. However, the social conscience of the country dictated that strong efforts be made to mitigate the inequity towards the poor. An important part of the reform of the 1990s, announced in 1996, was a poverty alleviation measure to focus resources, basic skills, innovative instructional methods, school autonomy, teacher training, curricular relevance to daily life, and humane care on the students of the municipal and rural schools. 1.6 To carry out this task, the government focused on improving the implementation capacity of the Ministry of Education (MINEDUC). It approached the World Bank for assistance because, more than loan proceeds, it was interested in the expertise that this institution could provide. Bank and Other Donor Involvement 1.7 The Bank had financed two vocational education project in 1968 and 1970 (Table 1), which were completed successftilly. The third project (Ln. 668-CH for US$7 million) was to strengthen primary teacher training colleges and agricultural secondary schools. However, its implementation coincided with a period of political turmoil, and the outcome was unsatisfactory. Though the Bank financed projects in other sectors, it did not finance new education projects for almost 20 years. Table 1. World Bank Education Projects in Chile Fina clsin Loa amunt Cancelled or Project name Approval Year Loan no. Final sing Loan ount undisbursed Completed Projects Education I 1966 431 12/31/69 2,750,000 47,369 Education Il-Vocational 1970 666 12/31/75 1,500,000 164,945 Education i1 1970 668 12/30177 7,000,000 1,550,000 Primary Education Improvement 1992 3410 6/30/98 170,000,000 49,213,412 Ongoing Projects Secondary Ed 1995 3883 6/30/01 35,000,000 11,083,000 Higher Education 1999 4404 6/30/04 145,450,00 0 Millennial Institutions(Learning 1999 4466 3/31/02 5,000,000 0 innovations loan) 1.8 Other donors in education included the Inter-American Development Bank, which financed a project for unemployed youth with the Ministry of Labor (Chilejoven program). In the early 1990s, the Nordic countries financed a MINEDUC program (called P900) to improve performance in the 900 schools with the lowest SIMCE scores. It consisted of teaching materials, school repairs, inservice training for teachers and supervisors, remedial workshops for low-achieving students, young people chosen from the community to provide social stimulation 4. Fully publicly financed schools are called municipal or rural, according to their location. 3 for students, and preparation time for teachers. The program continued in subsequent years, and the rural schools program financed by the Bank used the same model. Bank and Government Sector Policies 1.9 The Bank and the government initially had some ideological differences. The post- dictatorship government was more interested in equity for the social distribution of results, increasing the role of the state in controlling the quality of education, setting national standards for educational attainment, teacher salary increases, and participation of teachers as a professional body in school improvement.5 The Bank was very much interested in expanding coverage (particularly of preschool education since primary education had almost become universal), improving efficiency and learning outcomes, and keeping the educational expenditures affordable.6 In 1989 the institution was focusing strongly on primary education expansion and borrower staff perceived a limited interest in financing curricular changes, bilingual education, or secondary education (which Chile clearly needed due to the high average level of education of its citizens). The Bank encouraged Chile to maintain the voucher system and expand the application of private solutions to public problems, as the Pinochet government had promulgated. Loan conditions focused on maintaining these policies, obtaining targeted cost recovery of textbooks, maintaining budget proportions of the 1980s, which favored pre-primary and primary education (total 60%), paying teachers on the basis of merit, and ensuring sustainability of the MINEDUC actions once the project finished. 1.10 After considerable dialogue, the project objectives were set: a) Assist the government in enhancing the efficiency, quality, and equity of primary education in selected schools in urban and rural areas; b) Expand the coverage and enhance the quality of preschool education to increase primary school preparedness and reduce late entry, repetition, and dropout in primary schools, especially for 5-year-olds living in poverty conditions (children with psychomotor and social deficits are more likely to repeat grades and to drop out of school); c) Strengthen the institutional capacity of the Ministry of Education (MINEDUC) central, regional, and provincial offices of MINEDUC, as well as the capacity of the education departments of the municipalities to direct and manage a decentralized education system, and improve the managerial skills of the municipal and private-subsidized preschools and primary school administrators; d) Assess alternative cost-effective approaches to meet the secondary education needs of primary education graduates through studies (results implemented during a follow-on project). 1.11 Although the project started as a focused effort to improve the conditions of the municipal and rural schools, subsidized private schools also became eligible for most benefits targeted to the poor. Textbooks were given free, and the targeted cost-recovery policy was not implemented. To some extent this compromised the poverty focus of the project. More specific components and targets are outlined in Annex B. 5. Cox, Christian and Beatrice Avalos. 1999. 6. World Bank. 1991. 4 2. Project Implementation Experience 2.1 The government was highly motivated to succeed; implementation proceeded faster than expected and in general terms was highly successful. Thus began in 1990 a collaboration that became noted for its closeness, mutual understanding, appreciation, and trust. As a result, the country requested Bank help for projects in secondary and higher education. The government also requested a resident Bank staff member to advise MINEDUC on organizational issues (see section on Bank performance). An important factor in the collaboration was the relatively low level of corruption in the country's education sector, which assured government and Bank staff that the money would probably be spent for the purposes intended. As a result, almost all project input and outcome targets were met or surpassed (Annex B). The government also obtained better prices than expected for some items, and US$40 million was cancelled from the loan. 2.2 The main implementation problem concerned Bank procurement rules, which at that time were not well adapted to educational materials or to school-based expenditures, such as the quality improvement projects. The Latin America and Caribbean (LAC) procurement specialists determined that the budget levels for various subcomponents, such as the evaluation of the preschool program and the construction of a computer network needed international competitive bidding. The process cost the government one year of implementation with much staff-intensive paperwork, such as letters to embassies, publication in foreign journals, and waiting. Delays also meant that the budget allocated for various activities was sometimes lost and had to be reacquired. The benefits of the bidding process were unclear, since the interested parties were mainly Chilean universities, which ultimately obtained the contracts. (See section on Bank performance.) Project Outcomes in Classrooms - Mission Visits 2.3 Much has been written about this project's impressive implementation record, including evaluations by the government and documents on various aspects and components by Bank staff. The operation reportedly had a significant impact on classroom-level instructional delivery through the provision of lots of enrichment materials, teacher and supervisor training in participatory methods, the quality improvement projects, teacher and supervisor employment, investments in the assessment system, and an expansion of the preschool program that prepares poor children to enter primary school.8 The SIMCE scores of rural schools increased significantly. Teacher questionnaires and classroom observations showed that 71% of the teachers trained in the new instructional methods used them at least some of the time. 2.4 The audit mission visited (unannounced) a total of 15 municipal, private-subsidized, targeted low-performing (called P900), and rural schools near the cities of Santiago, Concepci6n, Valdivia, and Castro. Observations are outlined below. 7. E.g., Michael Potashnik: Computers in the Schools; Francoise Delannoy: Education Reforms in Chile; Christian Cox: International cooperation and donor assistance in the development and implementation of educational programmes. 8. Ministerio de Educaci6n. 1999. 5 Many Schools Can Be Considered "Effective" 2.5 Physical condition. Municipal schools, particularly in the larger cities, are often old but are well maintained by the municipalities (an advantage of decentralization). Efforts are made in rural areas to hire one teacher who can perform emergency maintenance. Staff try to make the schools attractive to children. Besides decorations and signs on the walls, the project financed bells, symbols, flags, and upgrading of facades. Four principals reported that when large-scale repairs and extensions were too expensive for their municipalities, they had raised funds through parents' activities. 2.6 Parental involvement. The project intensified the previously significant rate of parental involvement. In almost every school visited by the mission, parent activities were either scheduled or taking place: helping teachers in class, teaching dance, painting the school, meeting with the principal on the latest SIMCE score results. Discussions with parents and staff revealed that that involvement had been planned in some detail and was ongoing rather than sporadic. Most parents were women without a regular job, but others were men or farmers of both genders. Some mentioned that they spent time in schools because they were appreciated and found it a pleasant pastime. The project also improved communication with parents in indigenous areas, who were disengaged from schools. 2.7 Teacher initiatives. The essence of school reform was to increase teachers' initiative, make them responsible for improving the school environment rather than merely following instructions. The government doubled the extremely low salaries of teachers, a move that substantially raised morale and made teachers more likely to cooperate. The instrument chosen to increase teachers' control of the environment was the Proyecto de Mejoramiento Educativo (PME), a means to fund teacher-made proposals for inputs that would improve instructional quality. Teachers were to diagnose school deficiencies and propose small projects that would be considered for funding. Through supervisors, the government appealed to the teachers' professionalism, team spirit, and competitiveness to do the work. Response initially was slow, but the incentives were effective. Almost all schools visited by the mission had received a PME or were in the process of preparing one. Examples were: * Purchase of a small radio transmitter for a rural school, training of students in journalism and handling of audiovisual equipment. Sixth and seventh graders interviewed the OED mission and transmitted the interviews to all classes during the recreation break, following up with riddles and other entertainment. (Only boys volunteered to operate the equipment, while girls were the journalists and announcers.) * Theater training and equipment for a subsidized private school in Valdivia. The students enacted sketches about Chilean history, learned about costumes and make-up, and were videotaped. * Parents paid to paint and repair a school in a periurban area of Santiago.'o 9. Originally, U.S. schools were defined as effective if the same percentage of low- and high-income children passed criterion-referenced tests. They are characterized by school maintenance, focus on instruction, principal's leadership, and a sense of common goal. Chilean schools are essentially segregated by social class, but principles can still be applied. 10. The government commented that this observed component must have been only part of the project, because painting schools would be considered an inadequate project. 6 2.8 Though some differences in SIMCE scores might be attributable to PMEs, their effect on student skills is uncertain and probably diffuse; they were not linked to basic skills, about 75% of them were used to develop language, and very few focused on math or science. However, their effect in instigating teacher involvement was considerable, and teachers spent some personal time in the development and implementation of these projects. Nevertheless, funding for this program was reduced once the Bank-financed project ended, though efforts are made to fund as many PMEs as possible. Also, the evaluation and processing of the proposals takes up considerable MINEDUC resources, and this activity may not continue in its current form for long. However, it has served the purpose of making schools focus on a common goal, a hallmark of effective schools. 2.9 School management. Most principals are senior teachers with limited authority over finances or personnel. However, they received considerable training, notably in computer education, and most were proudly using it to enter student data or write reports. Within the limitations of their function, they were rather knowledgeable and seemed motivated. Some had proven records in involving parents and obtaining donations for school activities. Affective Domain of Student Learning: Attention to this Usually Neglected Area 2.10 The government is taking students' mental health and development of the affective domain" quite seriously. To reduce dropout and behavioral problems and to improve students' future opportunities, reformers tried to make school environments attractive, stimulate interest in the academic material, and develop students' social skills and self-esteem. Future plans include assessing student motivation, self-perception, self-control, and self-efficacy in various areas in order to chart secondary school students' personality profiles and to channel their energies according to their strengths, in hopes of lowering the probability of problem behaviors. Some notable activities are: 2.11 Enlaces - computer literacy. This very innovative program, developed by a dedicated group of computer scientists in a provincial university, provided computer networks with Internet connection to about half of the municipal and rural schools. (The other half are in the process of receiving it.) Software was developed to enable students to explore science, communications, and other areas. (Curricular-base software is constantly being developed.) Depending on size, schools received 3-9 computers, and teachers were trained to facilitate student explorations. Classes take turns in the computer room, where groups of children explore the evolving software available in Enlaces, do educational puzzles, or learn word processing. A notable program includes a plaza of a typical small town, where children open doors and explore subjects such as history, biology, and folklore. With a limited number of computers, the program is still used only for exploration and familiarization, but in the long term software will be produced that corresponds to instructional objectives. Aside from the actual skills of basic computer operation, the effects of this work on students (such as development of spatial perception) may be stimulating creativity, group work, self-esteem, motivation, and academic interests. The Enlaces program has received much publicity worldwide. Key to its success has been the technical assistance network of 17 universities that provide training and help to schools in their areas which receive computers. 11. The original classification of educational objectives in the affective domain is: (i) receiving, (ii) responding - actively attending, (iii) valuing, (iv) organization - conceptualization of values and formation of judgments; (v) characterization - organization of values, beliefs, attitudes into an internally consistent system. (Bloom, B. T. Hastings and G. Madaus. 1971 "Handbook on Formative and Summative Evaluation of Student Learning" McGraw-Hill.) 7 2.12 Valuing student environments. In various segments of the curricula activities are included to make students value their homes and culture. For example, they are asked to interview parents and relatives about their professions or show why it is better to live in the country rather than the city. The 900 lowest-scoring schools have funds to hire monitores, local young school graduates with good reputations who work part-time. Monitors receive short-term training in how to develop children's social skills and interaction patterns with games, dances, handicrafts, discussions about learning to value their environment and family, and cultural appreciation. It is thus hoped that self-esteem will improve and that children will practice successful behaviors and feel confident about who they are. The audit mission observed several activities that used monitors. 2.13 Education through art and play (educaci6n Iidica), and reading. Students spend several hours a week in drawing and handicrafts, either during artistic education class or as part of various courses. (Boys are still most likely to do wood carving and other technical tasks.) To develop interests and make students value reading, the project flooded the public schools with class libraries and book collections. These materials are for recreation and not necessarily related to curriculum. Students reportedly do use them, and the books did look worn. 2.14 Attractive instruction and emphasis on the various aspects of the affective domain probably has an affect on the development of the cognitive domain. Teachers report a low level of absenteeism because students like school, come, and prefer to stay. (Longitudinal data on student attendance were not immediately available but could be analyzed in the future.) The joyful environment, personalized care and structured tasks of information discovery may compensate to some extent for the "social capital" that these children lack due to the poverty of their homes and the limited interaction with children of higher socioeconomic levels. Cognitive Domain of Student Learning: A Need to Focus More on Objectives and Skills 2.15 The most important aspect of the reform has been the change in the philosophy and implementation of instructional delivery. Synthesis of knowledge is emphasized rather than mere information recall. Teachers were asked to shift their role from a lecturer to a facilitator of learning, directing groups of children to explore and carry out activities, to a considerable extent on their own. To elicit teachers' creativity, the curricula specify objectives but not the activities used to achieve them. Teachers are encouraged to seek help from their colleagues as well as from supervisors, senior teachers who were trained to act as educational advisors. Teachers in the relatively remote rural areas visited by the audit mission knew about and acted on the reformed instructional principles. 2.16 The various interventions in schools have probably resulted in improved SIMCE scores, which had stagnated in the 1980s.12 The achievement gap between low-income schools and the national average was increasing before the project but it shrank from 30 percentage points to about 10 points in the 1990s.13 Nevertheless, the gap between municipal and private schools is large; adjusted learning outcomes for grade 4 in 1996 show that municipal students get 79% of private schools' scores, while private-subsidized schools get 86% of private schools' scores. Yet, achievement on an international level is not what could be expected. In the Third International Math and Science study (grade 8), Chile scored in mathematics at the levels of Colombia and 12. Staff Appraisal Report, p. 10-13. 13. Delannoy. 2000. 8 Brazil, a little below Argentina and Cuba. Class observations may provide some insight into the reasons and the interventions most likely to have influenced SIMCE scores. 2.17 Limited time on task. Group work is a prominent feature of the reform. The tasks observed in primary school consisted of much student interaction. However, time actually spent on tasks seemed to be less than 50%, the rest was spent on unfocused activity accompanied by considerable noise. Since many activities are not well-structured, students tend to misbehave; some university observations indicated that in low-income classes, 10 of the 45 class minutes may be spent on discipline. By contrast, classes at a Catholic subsidized school, though larger, were much more orderly. Yet, the mission often observed the qualities that make group work a desirable instructional method. When tasks were complex, students shared information, and the better students explained procedures to the others without interrupting the teacher. But when tasks were simple or actually individual, students completed them and then chatted. Possibly to maximize time on task, group activities are practiced much less in private schools and higher grades. 2.18 Unclear linkages between objectives and activities. Given the need to match the achievement of private schools, one would expect much greater emphasis on basic skills. Instead, the mission observed several classes involved in cooperative and individual handicrafts (e.g., students gluing pieces of clothespins over tin cans). Even after teacher explanations, the instructional objectives for some activities were unclear. Activities may focus student attention to various mathematical properties or mnemonics for grammatical rules, but instead these handicrafts seemed an end in themselves, without such cognitive linkages. Furthermore, these activities were coupled with a limited emphasis on math and science. Classes and activities emphasizing communication and language were much more frequent. 2.19 Relatively unstructured class time. To store information and later retrieve it, students must clearly understand where to categorize it. This is why objectives, their rationale, applications of knowledge, evaluation, and conclusion are important in all classes of all areas and levels. The classes observed in Chile often had no clear beginning and did not sum up the material learned. Children continued activities started earlier, and when the bell rang, they left. To some extent, this is attributable to the Chilean philosophy that the teacher is merely a facilitator helping children to find answers and explore. The skill of working with little guidance is surely valuable, but the amount of time taken up may not compensate for the added value. 2.20 The advantages and disadvantages of the reformed instructional system are exacerbated in rural multigrade classrooms. Children indeed were found to follow written instructions alone or in groups and do rather complex assignments with little guidance. Without structure, however, they did not know the purpose for carrying out these tasks and could not prioritize what to learn. The super-busy teacher often concentrated on older children going to secondary schools and gave art tasks to younger students. The average children in municipal classes had acquired basic word decoding skills by the middle of the first grade and could read, albeit with limited comprehension. However, children in multigrade schools had received very little reading instruction in the same period. Teachers justified this with the view that the curriculum allows two years to learn reading and that it was pointless for children to read unless they understood. Without attempting to read and comprehend, however, the multigrade first graders were limited to drawing tasks and could not follow written instructions like their slightly older peers. Thus, children started falling behind in basic skills early on. It might be possible to teach children basic literacy through computer 9 programs of the Enlaces system, but programs were not available in classes visited by the OED mission.14 2.21 The relative lack of structure and low SIMCE scores may be one reason why parents who can afford to, vote with their feet. Though some municipal schools in populous periurban areas have classes of 40 students, most of those visited by the mission had 15-20 students each. By comparison, nearby subsidized private schools that required no extra tuition were overflowing with 45 students per class and multiple sections. 2.22 Typically, school principals and teachers expressed satisfaction with the SIMCE scores, frequently stating that they had risen since the last measurement. However, this satisfaction may be premature. School staff compared their school to others in the same area and socioeconomic condition (a feature that sometimes makes school authorities overstate students' poverty)15 and were oblivious to the national averages or large differences with purely private schools. They also did not have a clear idea which instructional aspects to remedy given their SIMCE scores. There is a concern that while the satisfaction level is high, students from higher-scoring schools will be the ones passing the university entrance examinations. 2.23 MINEDUC is cognizant of the limited instructional time and has already revised classroom activity plans to provide more structure and focus on basic competencies. A draft was completed in August 2000. Establishing full-day schools has also been an important policy, agreed to by the government and the opposition. So, MINEDUC is in the process of lengthening the number of hours students remain in school, at considerable expense, in order to care for them better. Several schools in rural areas already retain students in schools for a full day and obtain many opportunities for cultural and social stimulation. Given the wastage of time at the class level, however, and need to focus on basic skills, the benefits of this plan should be evaluated carefully. Teacher In-service Training 2.24 The school reform was communicated through extensive in-service training, cascading from MINEDUC to municipalities and local supervisors through numerous meetings and conferences. Emphasis was placed on using the new instructional materials and changing instructional delivery modes. Nevertheless, in-service training was the least successful project component. The MINEDUC staff in charge lacked the capacity to train the 130,000 teachers of the country systematically, and initially the teachers' union did not cooperate. The proposal presented to the ministry was weak, and the funds allocated from the project were partly cancelled. (The project did not deal with pre-service training, which now takes place in universities.) 2.25 Training was much more successful for rural teachers, who had been isolated in earlier years. Supervisors have worked out systems of getting to the more remote schools on a regular basis; vehicles drop several people in various localities and later pick them up. The project established microcenters, where area teachers meet for one day a month and exchange experiences.16 The mission visited an annual five-day meeting of microcenters in Valdivia and 14. The government commented that in 1998, the program Abrapalabras was distributed to half the schools. 15. Greaney. 1996. Also: Horn, Robin and Eduardo Velez. 1991. 16. Chile is a country where getting teachers to rural areas is not very hard. Particularly after the large salary increases, teachers are able to make ends meet. Even if they do not want to live in the teacher quarters available in many schools, several may own cars and may carpool to schools from nearby towns. 10 observed detailed activities and discussions. Many teachers were interested in and had clear opinions about the strategies and philosophy of the reform. 2.26 Training on site, in groups of 20 teachers, has also been quite successful. 2.27 An evaluation of the rural basic education program showed that about 71% of the teachers put into practice at least some of the techniques they had learned. Given the expenditures and effort, this percentage seems low. One problem is that training is conducted through discussions, which are usually not effective in modifying teaching behaviors. Role modeling is much more effective, and teachers can improve faster through studying successful and unsuccessful teaching events, videotaping, and feedback. However, these methods are not used in Chile. It may be useful to follow up after training and see what techniques teachers actually put into practice. School Nutrition and Health 2.28 School nutrition works very well in this country. Depending on height and weight for their age, low-income children get nutritionally balanced meals. In all schools visited, the kitchens were clean and gas - provided by private companies - was used for cooking. Local residents worked as cooks, and no problems were reported with contractors. 2.29 The project financed diagnostic examinations and treatment for low-income students, including provision of eyeglasses, hearing aids, posture aids, and dental care. Children with emotional problems are treated by psychologists and special education teachers. The mission witnessed children going to and returning from doctors and special education specialists. Best Practice Health Care Provision In Cerro Navia, a low-income suburb of Santiago, there is a German clinic where many primary schools send entire classes for medical checkups and treatments. The clinic has a classroom attached to it, where class is conducted while students are examined one by one. This way no class time is wasted, and students are kept busy with their own peers rather than wait and worry about painful procedures. During alternate visits, students are systematically diagnosed and treated. This is a best practice example in terms of health provision with minimal school disruption. 2.30 Special education. Chile integrates many children with learning difficulties in regular classes and provides special assistance to them as needed. Teachers expressed willingness to work with these children. However, a number of them also discussed their problems in their presence. The negative effect was sometimes immediately visible on student behaviors. The mission observed reading specialists work with dyslexic children. Though willing and interested, many of them did not have adequate materials or training. The computers of the Enlaces program could be used with software developed to help students who are behind in reading or math to catch up, but this use does not yet exist. 2.31 Another project component, preschool education, was implemented extensively, and an evaluation study showed significant SIMCE gains after one or two years of students' attendance. The detailed objectives of preschool programs for poor children still seem unclear; some classes visited by the mission seemed to provide mainly babysitting. Some Chilean research, however, indicates that stimulation may have to start much earlier, at 18 months, in order for children to catch up with the development of children with more economic means. 11 Relevance and Efficiency 2.32 The project has been consistent with a strategy of developing human resources in Chile and particularly relevant to its economic and social development needs. Its design and implementation used Bank resources very efficiently. Nevertheless, there were other relevant objectives that the Bank did not pursue, such as the improvement in access and quality of secondary education." Although studies were carried out for a subsequent project, some MINEDUC staff interviewed were of the opinion that the length of time between studies and intervention was too long. Also, the Bank did not show much interest in policies to reduce the inequity created by school choice, an issue important to the government. Institutional Development Impact 2.33 The institutional development impact of the project per se was modest. The pertinent component of the project focused on the provision of training and materials, such as computers and seminars for various levels of staff. Furthermore, MINEDUC staff training had a limited effect, partly because some staff did not have the requisite education level to benefit from it. However, the impact of the collaboration with the Bank through the project turned out to be substantial. The project implementation unit (PIU) team was inspired by the Bank's operational organization and team approach. MINEDUC staff report that they saw lawyers collaborate with engineers early on and realized that doing so could save time and prevent conflicts during implementation. For this reason, they decided to structure teams for project appraisal and implementation in modes similar to the Bank's. They also found the operations manual the Bank requested prior to effectiveness an important tool to clarify their own methods and used this tool as well as the Bank's bidding documents in government-financed operations. Rather than disband the PIU and integrate it completely into the ministry, as has been done in other countries, the government kept it as a distinct unit for subsequent projects. Some of the best MINEDUC professionals are known to have worked in the PIU, which often employs younger people with advanced degrees.8 2.34 Considerable organizational help was also given by a Bank staff who functioned as a resident advisor to the Minister of Education in 1997/98. In collaboration with a retired Bank staff member living in Chile, she helped MINEDUC analyze the goals of its various departments, the organizational structure, and how each department could be restructured to meet its goals. She also helped establish an educational advisory service. Best practices and Bank research findings were downloaded on a regular basis and disseminated to managers, afunction that was maintained after the end of the advisory term. The deliberations and ideas left behind were still being processed when the audit mission came. 2.35 Overall, institutional development impact is rated substantial. The magnitude of inspiration and benefit from the Bank's organizational example constitute a best practice. 17. The 1992 Wappenhans report recommended attention of a few selective variables and may have acted as a deterrent to financing more secondary education inputs. 18. Bank experience with PIUs indicates that, when well staffed with technical skills, PIUs have proved useful or even indispensable for the achievement of rapid and efficient implementation of projects. ("Utilization of Project Implementation Units," OED Evaluation Lessons, Knowledge Management Group, April 2000). 12 Sustainability 2.36 The project outcomes are clearly sustainable in terms of resilience to risk of net benefits over time. After the project ended, financing for the operation of various components, though diminished somewhat, continued at levels appropriate to maintain the operation of the various initiatives. Sustainability is rated highly likely. Bank Performance 2.37 Overall, Bank performance is rated satisfactory. However, it may be considered in three different aspects: 2.38 Task manager performance. To some extent at least, cultures and personalities were compatible. Government staff expressed high appreciation for the Bank's support at the personal level. Task managers and division chiefs alike showed sensitivity to the country's needs and did everything possible to facilitate implementation. Government staff were particularly appreciative of the support given by operations staff and managers in order to negotiate the onerous interpretations of procurement rules imposed by the Bank. In this respect, Bank performance is rated highly satisfactory. 2.39 Procurement. The Bank's procurement advisors insisted that the country conduct international competitive bidding for various evaluations and technical assistance components. They also insisted that municipalities and schools that received project funds abide by the same procurement rules. However, the Bank rules that resulted in considerable wastage of time and resources in Chile were applied quite differently at about the same time in other operations, such as India's District Primary Education Project. There, the government was allowed to contract large amounts of technical assistance and studies money to their own chosen providers without any competitive bidding." The Bank staff who worked on the Indian projects structured procurement so that activities were small and many (at the district and village level), below the levels triggering varicas competitive bidding rules. Also, IDA rules allow some preference to local contractors. Nevertheless, two borrowers with substantially similar project components had to abide by very different procurement conditions. 2.40 Instructional expertise. The government and the Bank agreed that the projects implemented would reflect best practice experiences. However, the Bank's supervision-related documents make practically no mention of instructional issues. Though Bank staff with instructional expertise participated in the project appraisal, those who supervised the project were mainly generalists and economists. It was widely believed that the country possessed instructional expertise and that intervention at that level was unnecessary. Therefore, the project content focused on infrastructure, materials provision, studies, and training that could have taken place in any other ministry. Some of the MINEDUC staff interviewed wrongly concluded that the Bank does not provide instructional expertise anywhere in the world. 2.41 Optimal information processing is the reason for providing material inputs, and inattention to it is unfortunate. The reform experimented with the role of the affective domain in school effectiveness, but the Bank did not advise on its evaluation or on related research. (See next section.) The instructional principles of the reform could have been honed on the basis of the state-of-the-art educational research, possibly through twinning with a U.S. or Canadian 19. Comments from the region state that the two projects were too different to be compared in procurement issues. 13 university. Earlier understanding of how to optimize information-processing by poorer children in classes might have resulted in higher times on task, greater instructional structure, and even better SIMCE scores. Borrower Performance 2.42 Borrower performance is rated highly satisfactory. Important factors were the staff continuity and existence of inspirational managers in the PIU. These were highly respected and drove staff to achieve more and spend their own time in implementing the project, a feeling which staff describe as "mistica del trabajo." Such charismatic managers are known to bring about satisfactory implementation even in lower-income countries with problems much worse than those of Chile. The project probably also benefited from the atmosphere of hope and optimism that prevailed after the restoration of democracy and the relatively low perceived corruption.20 3. Issues for Future Consideration Need to Improve Basic Skills and Verify Effects 3.1 The philosophy and techniques exhibited in Chilean public schools constitute a series of hypotheses that have not been tested. They may or may not have the desired effects, but the country is spending considerable resources on them. Knowing the answers to some key questions would help focus resources and strengthen basic skills. Among the questions that should be answered are: * Does emphasis on affective domain activities strengthen basic competencies for children of poor homes? How much is optimal? Is there perhaps a tradeoff between student happiness in school and basic skills acquisition? Perhaps affective activities make poorer students more likely to stay in school and study, perhaps they just take up precious time. * What are the effects of art education and computer use? Do they help develop specific transferable skills (such as spatial perception) or are they mainly recreation? Are children who are successful in these areas perhaps more likely to attempt math and science? * How significant are the effects of the cultural enrichment activities on students' self-esteem, self-control, and academic self-efficacy vis-i-vis other school and home variables? Do they succeed in making children less vulnerable to the effects of abuse, for example? * Do the 900 lowest-scoring schools that receive temporary support improve permanently or do their scores drop again after support is withdrawn? 3.2 Statistical treatment of data can separate the effects of various confounding variables and demonstrate (where possible) how much improvement in various cognitive and affective measures students show for $1 of resources spent in specific programs. However, quantitative evaluation expertise is limited in Chile, and studies do not produce such information. The studies financed by the project and by other government projects often use qualitative methods and 20. The Transparency International Corruption Perceptions Index for Chile was unavailable at the time of the audit. 14 provide common-sense answers that do not help reorient activities. For example, the evaluation of the rural basic education program is limited to univariate tables, has no inferential statistics, and makes no attempt to disentangle the effects of various variables. Thus, nothing is known about interactions and suppressor or mediator variables. Similarly, there are no long-term multivariate studies of SIMCE information, though data exist back to 1978. 3.3 The country has reached the point where robust statistical skills are needed among core staff in MINEDUC and various universities. The various social science faculties often offer limited training in statistics and experimental design, partly because of traditional linkages with European schools of thought, where education and psychology are theoretical rather than experimental disciplines. Training in sophisticated evaluation designs and data interpretation may be achieved though obtaining advanced degrees in North American universities and/or through systematic courses in Chile tailored to MINEDUC evaluation needs. 3.4 Computer use for remediation. After the very successful implementation of the Enlaces system, a second stage is under preparation to prepare activities for all curricular objectives. This very laudable objective will take several years to complete and will require many more computers in each school than are available. It may be feasible, in the meantime, to provide computer-based remedial work for special education children (e.g., dyslexics), literacy activities for first graders in multigrade schools, and children who fall behind in math classes that require a serial treatment of objectives. Software can be developed and used within the capacity of the current system in each school. The Inequity of School Choice Continues 3.5 The heroic efforts of the reform in improving the quality of public schools do not mitigate the inequities of school choice, according to a Bank-sponsored study that attempted to evaluate the effects of the system.1 Despite many methodological limitations, the study found that households of higher socioeconomic status are substantially more likely to enroll their children in schools exhibiting higher levels of achievement than households of lesser means. This happens largely because schools are free to choose their students. Since they want well-behaved and cooperative children, they choose on the basis of social rather than academic criteria. The study also found that if parents are really persistent about school choice, they may break down the social barriers, a hopeful issue which indicates that school choice information should be more explicitly available. But for the vast majority of the low-income families, prospects are limited; even if parents persist, schools are free to reject their children. 3.6 Is school choice saving money? The question has been hard to answer. Municipal and rural schools are still less efficiently run than private subsidized schools, and the municipalities are still unable to realize the administration savings that characterize private schools. For example, teachers may work fewer hours than expected or be on long leaves of absence22 On the other hand, they serve the students other schools reject. More than 70% of the students belong to the lowest socioeconomic strata, as compared to 45% in the subsidized schools. The compensatory funds given to the rural and lowest-scoring schools amount to only about Ch.800 per month per child compared to a subsidy of Ch.26,000 for subsidized schools. Assessing long- 21. Gauri, Varun. 1998. School Choice in Chile: Two Decades of Educational Reform. The study was difficult, mainly because there are no student-level records in Chile, whereby students' achievement test scores can be traced through different years, particularly if they change types of schools. 22. Espinola. 1997. 15 term benefits is hard also because the military government's school choice policy was accompanied by stiff budgetary cuts, and the value of the subvention declined by 32% in real terms from 1981 to 1991, while public expenditure as a percentage of GDP fell from 4.5% in 1981 to 2.9% in 1994. It is possible that education costs in the country dropped when much school construction and maintenance were left up to private entities in exchange for a subsidy. During that period dropouts drastically declined, and students stayed in school longer, but that is often the case when unemployment is high. Private schools have earned money by running morning and afternoon shifts, where teachers work long hours to make ends meet. There is a concern that some private entities are making money by spending less on the students than the subsidy they receive. 3.7 The country is thinking intensely about how to deal with the school choice dilemma. Several educators interviewed believe that letting schools decide what to teach and whom to admit creates conditions that are incompatible with the democratic institutions of Chile. For example, many Catholic schools demand the parents' marriage certificate to admit children, thus excluding children of single mothers.23 One school that accepted all children, nevertheless gave preference to siblings of students and children of alumni, thus creating situations of privilege with state funds. Some options to deal with the social problem might include: * Limiting subsidies to low-income students and schools that require zero or minimal extra tuition. Giving subsidies to schools that then require large additional tuitions may defeat the purpose of the vouchers and divert money from those who cannot pay to those who can. (Some poor high-scoring students receive scholarships to the schools that require additional financing, but those are a small proportion of the population.) * Forbidding admission based on certain social criteria, such as legitimacy, parents' marital status, or religion, and using judicial means to break down the social barriers. (For example, Concepci6n municipality fights against this and succeeded in reinstating a pregnant girl to a Catholic school.) * Admitting subsidized students who meet minimal requirements on the basis of a lottery. * Creating differential subsidies based on student income. * Spending even more on the poor. Given the educational privileges of students in subsidized schools, it seems that the country needs to spend even more on the deprived population as a compensation. One feature might be to offer a higher pay to the municipal teachers who work in low-income schools. 23. Bernardo O'Higgins, the father of the country, would not have been admitted to the subsidized schools because he was illegitimate. 16 4. Lessons 4.1 The "Chile recipe" for the education of lower-income students consists of: * A middle income economy, political stability, few natural and other physical disasters, low corruption in the education system * Parents with basic education, many non-working mothers who have time for involvement * Municipalities that have the authority and means to supervise teachers and maintain schools * Adequately paid teachers, who spend time as groups in school activities * A happy environment that may attract children to school * A surfeit of instructional and enrichment materials that are actually used * Students trained to search for answers, often through group work * Learning assessments for achievement monitoring * Attention to children's health and nutrition. 4.2 The organization of Chilean education is a model that other countries may aspire to reach, but what can they learn from Chile's experience? The above items are not easily disseminable in lower-income countries. Chile's relatively high per capita income makes it possible to obtain physical and human inputs that other countries cannot afford. For example, it has been difficult, despite Bank interest and help, to sustain learning assessments in many countries. Not only are these systems expensive, their contents overlap with public examinations, and countries with limited implementation capacity find it difficult to make changes once they know the diagnoses for each school. Other features are particular to the cultural traditions of Chile. For example; in a country where people usually greet each other with a kiss, it is easy for teachers to act affectionately towards children. The high social sensitivity in Chile towards the poorer people may make teaches more willing to spend extra hours doing school-related work; the teachers' union to a considerable extent supports educational efforts directed towards the poor. With corruption in the country at relatively low levels, it is also easier to implement components like school nutrition that have proved difficult elsewhere. 4.3 Yet, some features of effective Chilean schools are related to human relationships and resonate with the experiences of other countries. When students in areas as diverse as the United Kingdom, Uzbekistan, Thailand, and South Africa were asked their views of what makes a good school, they all agreed on three key elements: (a) good teacher-pupil relationships; (b) support for overcoming learning difficulties; and (c) good communications with parents. Therefore, the following ingredients of the Chilean recipe may be useful to other countries: 4.4 Development of the affective domain. This area has been long neglected in favor of cognitive issues. Yet, developing students' interests and value systems is likely to increase the 17 probability that they will retain information and go on to higher levels of schooling. The psychomotor activity of children moving rather than sitting and listening passively may influence the neurotransmitters that improve the mood and create multiple bridges to the information in the brain. Overall, it is probable that unhappy and unattractive schools are one reason behind student absenteeism and dropout. 4.5 Affection and acceptance. Mistreatment, beatings, and "pushout" are a frequent problem in lower-income countries, particularly when teachers belong to higher socioeconomic strata than the students. Teachers in other countries might be specifically trained to show culturally appropriate expressions of affection towards students, such as petting their heads or hugging them, rather than beating them, as sometimes happens. 4.6 School appearance. Bank projects try to keep school construction costs as low as possible, which often results in unattractive schools. After consultations with the local population regarding attractiveness, specific low-cost items that beautify the school might be considered. Controlled research (questionnaires and class observations) may measure the importance of school appearance on dropout and attendance and the cost-benefit implications. 4.7 Importance and selective utility of group work students. Group work and peer tutoring is often discussed but rarely put in practice, partly because teachers find it easier to control classes that face them. But those techniques may save teachers work. As discussed earlier, more research is needed in understanding where group work is beneficial and where it is not. Classroom furniture must consist of square tables that make it possible to rearrange. 4.8 Searching for answers. When extra instructional materials are available (e.g., dictionaries and encyclopedias) and textbooks include search questions, students can be directed to use the materials. Searching for answers individually or in groups may help develop the all- important research skills needed in the 21st century. Conversely, in countries where enrichment materials are not woven into the curriculum, they may lie unused or be stolen. 4.9 Empowering group activities for teachers. Creating competitions among teachers of various areas that require group diagnosis and problem solving may be a way to stimulate interest in school work, which many teachers find unappealing. Quality improvement projects, which might be burdensome for central authorities, might be awarded by local authorities. 4.10 Keeping students in school longer. Short school days are a perennial problem in many countries. Moreover, the Bank has encouraged the full use of the buildings in two or even three shifts. Means must be found to make students stay in school longer, if possible through the help of school "monitors" that provide academic and enrichment support rather than higher-paid teachers. 4.11 The value of early and continuing cognitive stimulation. Class and school enrichment materials are essential learning tools. In poorer countries these may be provided through NGOs and civil society. 4.12 School nutrition. This component has been controversial because food may be abused by teachers who administer it. Also, the programs are expensive and may result in small and insignificant rations, such as the nutrition program of the Dominican Republic. Perhaps NGO participation may bring these about satisfactorily in countries with a high level of commitment to school nutrition. 18 4.13 Some special education, at least for children with mild disabilities such as dyslexia, may be feasible in class, particularly if low-paid aides can be trained in basic techniques to work with children who fall behind. 4.14 School vouchers targeted to the very poor. In many countries, as in South Asia, governments pay teacher salaries for NGO or other private schools. But these subsidies are often not targeted to the poor and are independent of achievement levels. These governments might use their funds better (assuming special interests permit this) by giving subsidies based on numbers of poor parents willing to send children to schools and on children's achievement. 4.15 Interventions must be accompanied by robust educational research that will separate the effects of various confounding variables, as discussed in paras. 3.43-3.44. 4.16 Lessons pertinent to the Bank's operations from this audit are: * The same procurement rules must apply equally to all Bank borrowers. Latin America and South Asia should not use interpretations that differ to such an extent that some countries are forced to spend years preparing for international competitive bidding and others exempted. Burdensome procurement rules exhaust the time and resources that countries may devote to implementation and detract from project substance. The procurement management board might want to study this inequality. * Bank appraisals and supervisions should emphasize classroom-level instructional delivery issues at least as much as they emphasize organizational and economic issues. * Under some economic and social conditions, some countries find the Bank's educational advice very useful and invite it. More research must be undertaken to find out under what circumstances the advice in specific issues is more likely to resonate with the policies of governments. 19 References Cox, Christian and Beatrice Avalos. 1999. Educational policies, change programmes, and international cooperation: "The case of Chile." In Kenneth King and Lene Buchart (Eds.) Changing International Aid to Education. Paris: Unesco Publishing/Norrag. Delannoy, Francoise. 2000. Education Reforms in Chile, 1980-98: A Lesson in Pragmatism. World Bank: Country Studies, Education Reform and Management Series, Vol. 1 No. 1. Pp. 40-41. Wasghinton D.C. Espinola, Viola. 1997. Descentralizaci6n del Sistema Educativo en Chile: Impacto en la Gesti6n de las Escuelas. World Bank: LAC- HD Group. Report no. 10. (Internal documents no. 17123). Washington, D.C. Gauri, Varun. 1998. School Choice in Chile: Two Decades of Educational Reform. University of Pittsburgh Press. Greaney, Vincent and Thomas Kellaghan. 1996. Monitoring the Learning Outcomes of Education Systems. World Bank: Directions in Development series. Washington, D.C. Horn, Robin and Eduardo Velez. 1991. Developing Educational Assessment Systems in Latin America. A Review ofissues and Recent Experience. World Bank. Latin America and the Caribbean Technical Department. Regional Studies Program Report No. 9. (Internal Documents report No. 10006). Washington, D.C. Ministerio de Educaci6n. 1999. Evaluaci6n del Programa Educaci6n Basica Rural 1992/1998. Santiago: Ministerio de Educaci6n. June. Potashnik, Michael. 1996. Computers in the Schools: Chile's Learning Network. World Bank. Human and Social Development Group. Latin America and the Caribbean Region, LASCH Paper Series No. 4. (Internal Documents Report no. 15929.) Washington, D.C. Repdblica de Chile. 1988. Desarrollo de la Educaci6n, 1986-88. Santiago, Ministerio de Educaci6n. (Internal Documents Report No. 16170). Washington, D.C. West, Edwin. 1996. "Education Vouchers in Practice and Principle: A World Survey." World Bank. The full report corresponding to Education Vouchers in Practice and Principle: A World Survey, Human Capital Development Working Paper No. 16. (Internal Documents Report No. 19275.) Washington, D.C. World Bank. 1991. StaffAppraisal Report No. 9769-CH, September 11. Washington, D.C.  21 Annex A. Basic Data sheet CHILE-PRIMARY EDUCATION IMPROVEMENT PROJECT (LOAN 3410--CH) Key Project Data Actual or Actual as % of current appraisal estimate estimate Total project costs (US$M) 216.2 89 Loan amount (US$M) 170 Cancellation (US$M) 53.4 Date physical components completed: June 30, 1988 Cumulative Estimated and Actual Disbursements (US$ million) FY92 FY93 FY94 FY95 FY96 FY97 FY98 Appraisal estimate 12.4 46.6 80.2 109.9 135.5 158.4 170.0 Actual 8.3 12.1 44.1 67.2 81.6 98.8 116,7 Actual as % of estimate 67 26 55 61 60 62 69 Date of final disbursement: December 17, 1997 Project Dates Steps in project cycle Original Actual Identification May 1990 April-May 1990 Preparation September 1990 December 1990 Pre-appraisal March 1991 March 1991 Appraisal May 1991 June 1991 Negotiations August 1991 August 1991 Development Policy Letter August 1991 August 1991 Board presentation September 1991 November 1991 Signing October 1991 November 1991 Effectiveness January 1992 March 1992 Midterm review May 1996 May 1996 Loan closing June 1998 June 1998 Final disbursement March 1998 December 1997 Staff Inputs (staff weeks) Stage of project cycle Actual Weeks US$ Preparation to Appraisal 7.3 19,255 Appraisal-Board 11.3 34,324 Negotiations through Board Approval 113.6 293,470 Supervision 71.1 258,385 Completion 4 20,000 Total 207.3 625,434 22 Annex A Mission Data Duration of Performance ratings Stage of project Date No. of staff in mission Specializations implement. Develop. Types of cycle (month/year) field (# of days) represented' Statusb Objectivesb problems Through Appraisal April 90 1 6 15 SES, E, C --- -- -- July 90 1 3 EA August 90 5 15 SES, E, OA Dec. 90 10 March 91 9 15 SES, DS, E, I, TA Appraisal through June 91 3 4 DC, SES Board Approval Aug. 91 1 10 FMA --- --- -- Supervision January 92 1 4 SES S S March 92 4 4 SES,ES,P,DS S S June92 1 4 SES S S October 92 4 9 SES, C, EC HS HS October 93 4 7 SES, POO, CTT HS HS April 94 1 4 SES HS HS October 94 3 3 SES, POO, COS HS HS May95 1 5 SES HS HS April 96 5 10 HDE, SES, A, OO HS HS Aug. 96 1 5 HDE HS HS May 97 4 5 HDE,SES,C HS HS Completion Sept. 98 2 2 POO,C --- --- a. A = Architect; C = Consultor; COS = Consultant on Organizational Services; CTT = Consultant on Teacher Training; DC = Division Chief; DS = Disbrusement Specialist; E = Economist; EA = Economic Analyst; FMA = Financial Management and Audit; HDE = Human Development Economist; L = Legal Specialist; OA = Operations Analyst; P = Procurement Expert; POO = Principal Operations Officer; TA = Task Assistant. b. HS = Highly satisfactory; S = Satisfactory; U = Unsatisfactory Related Bank Loans Loan Title Loan Purpose Year of Lon iteapproval Status Preceding operations Vocational Training project 431-CH Expansion of the training capacity of 1965 Completed the National Institute for Vocational Training Second Vocational Training project 666-CH 1970 Completed Third Education Project 668 Support the MINEDUC to expand 1970 Completed and improve primary teacher training colleges and agriculture secondary schools. Following operations Secondary Education Quality 3883-CH Improve the internal and external 1995 Under improvement Project efficiency, quality and equity of all implementation the municipal and government subsidized private secondary schools, as well as strengthen sectoral managerial capacity. Higher Education Improvement 4404-CH Improve the performance of the 1998 To become Project Chilean higher education system in effective. terms of coherence and efficiency, quality relevance and equity. 23 Annex B. Project Activities Componentsl Activities Targets to be Outputs Outcomes Subcomponents Achieved (a) Primary All multigrade rural education schools reached School-based quality 5000 selected rural All schools eligible; Developed teacher improvement projects and private- Almost 100% rural collaboration, subsidized schools schools developed & execution skills; executed them student performance changes not significant Assessing student 240,000 students in Changes in student achievement grades 4 and 8 performance can be measured Inservice training 78,000 teachers 510 microcenters for 8,000 principals 625 rural teachers supervisors Textbook distribution 24.5 million 30 million textbooks; Materials are used on targeted cost basis 870000 sets of 2 million teaching heavily, may account cmlmnay materials for student score materialsinrae Hiring new teachers to 500 0 complete 8 grades in rural schools Classroom 500 constructions 190 built Classes used and well construction and 2500 upgrades; (overestimated) maintained; upgrade in rural areas facades improved 2232 upgrades Pilot computer 66 2242 schools in 1998 50% of primary ed network in rural Children explore schools computer literacy Health screening 250,000 students 1,862,000 screened Children received teachers trained needed treatment Incentives for training in rural areas (b) Preschool education Expanding coverage To 30,000 urban and Coverage from 21% Low-income children 16,000 rural children to 29% of children better prepared for primary school Hiring staff in existing 23 supervisors Targets met Kindergartens better services 140 teachers staffed 1255 teacher aides Training rural parents 3000 workshops for Targets met Parents understand in early stimulation 23,400 parents child development better Classroom Construct 100 110 built Preschool buildings in construction and classes, refurbish 75 better condition upgrade in rural areas Provide food and 18,000 rations daily Targets met Children better ted instructional materials 40,300 sets of materials Hiring staff in new 50 supervisors Targets met Kindergartens better services 163 teachers staffed 164 teacher aides Inservice training 2000 supervisors, Targets met Staff better prepared 3300 principals and teachers, 2700 teacher aides 24 Annex B Componentsl Activities Targets to be Outputs Outcomes Subcomponents Achieved Designing, producing, 102,000 sets Targets met Enrichment materials distributing materials available for children Strengthening parent Workshops, media Parents of 14000 participation campaign children reached Evaluating results (c) Institutional development Train staff in planning 500 Ministry staff Training targets met Management capacity 670 municipal staff partly upgraded Train in management 1200 administrators Training targets met Management capacity partly upgraded Develop practical Planning, budgeting Manuals developed Changes still under manuals way Educational Partially Rapid acquisition of information system implemented data now possible (d) Secondary education Finance studies in 12 12 carried out; Results piloted in 124 curriculum, teacher dialogue with schools training, efficiency, parents proposal cost-effectiveness prepared 25 Annex C. Comments from the Government Miistepir de Educación UNIDAD l CURRICIlIUMH UYLUAClON Santiago, 26 de enero, 2001. Alain Barbu Jefe Grupo de Evaluación Sectorial y Temática Departamento de Evaluación de Operaciones The World Bank 1818 H Street Washington. D.C. 20433 U.S.A Estimado Sr. Barbu: Ref: Chile : proyecto de Mejoramiento de la Calidad y Equidad de la Educación Básica (MECEA. Préstamo 3410-CH, Informe de evaluación Ex- Post. Con un atraso que lamentamos, respondo a su carta del pasado mes de noviembre, relativa a la versión preliminar del Informe de Evaluación Ex-Post del proyecto mencionado en referencia. Quiero manifestarle que el Ministerio de Educación de Chile valora como muy positiva y aportadora la experiencia de evaluación ex-post realizada y que el Informe del caso aporta una visión que es nueva, profunda y muy convergente con las preocupaciones actuales del liderazgo del Ministerio. Aporta además un conjunto de proposiciones específicas que consideramos útiles e interesantes para la permanente tarea de robustecimiento y mejora de nuestras estrategias e instrumentos de intervención directa o indirecta para el mejoramiento sustantivo de la calidad y equidad de la educación escolar chilena. Asimismo, la versión preliminar del Informe contiene algunos errores e imprecisiones de hecho y otros de interpretación, que se detallan en el Informe adjunto a ésta, y que, por cierto, nos gustaría ver corregidos . Le saluda, atentamente. g Cristián Cox Coordinador Nacional Programa MECE Ministerio de Educación de Chile  27 Annex C COMENTARIOS AL PERFORMANCE AUDIT REPORT CHILE PRIMARY EDUCATION PROJECT (LOAN 3410-CH) MINISTERIO DE EDUCACION DE CHILE ENERO 2001. El Informe de Evaluación Ex-post del proyecto MECE-Básica (Proyecto 3410-CH), aporta al Ministerio de Educación una visión externa de mucho valor sobre lo que éste considera hoy día como el núcleo crítico de sus políticas. En efecto, luego de una década de inversión e innovaciones consistentes para levantar las condiciones de base en que opera el sistema escolar del país, la pregunta que ordena toda la agenda de política educacional en el presente es: ¿cómo afectar más decisivamente los aprendizajes alcanzados por los alumnos? Desde esta perspectiva el Informe aporta un análisis que es de mucho valor y vigencia para la conducción ministerial, responsable de definir los nuevos énfasis de las políticas de mejoramiento de la calidad y equidad, así como la corrección de los caminos e instrumentos que no han resultado efectivos. El Ministerio comparte la tésis general del análisis ex-post: gracias al programa MECE y la continuidad de sus componentes y líneas de acción post- 1997, así como otras medidas de política (extensión de jornada, reforma curricular) se han establecido contextos y oportunidades de aprendizaje indudablemente más ricos que los existentes en 1990. En estos además impera una base motivacional y moral cálida y benevolente, que promueve la participación de los alumnos. Sin embargo, esta buena base para el aprendizaje no se está aprovechando en todo su potencial, y hay un cierto desdibujamiento o falta de priorización clara, en las escuelas, de los logros más convencionalmente entendidos como 'cognitivos'. Esta interpretación es consistente con los resultados obtenidos por Chile en la última aplicación del TIMSS (1999): teniendo indicadores en o cerca del promedio internacional en un conjunto de factores definitorios de la calidad de las oportunidades de aprendizaje que un sistema escolar ofrece (gasto por alumno, tiempo isntruccional anual, disponibilidad de textos y computadores, calidad de infraestructura, etc), los resultados de aprendizaje en matemática y ciencias (cohorte de 14 años de edad), ubican al país en el lugar 35 de los 38 medidos (ninguno de América Latina). La recomendación genérica del Informe, Cognitive Domain of student learning: a need to focus more on objectives and skills, la compartimos plenamente. 28 Annex C A continuación señalamos un conjunto de puntos factuales que necesitan corrección en el Informe; así como otros, materia de interpretación, donde exponemos la perspectiva del Ministerio de Educación. Ambos son ordenados de acuerdo al orden de párrafos del propio Informe de Evaluación Ex-Post. ALCANCES Y CORRECCIONES. 1.2 El juicio "though technically imperfect" sobre el SIMCE debe ser especificado y puesto en perspectiva. ¿En qué consistirían sus mayores imperfecciones y en qué etapa? 1.3. El juicio "all parents could obtain vouchers (....) and send children to certain eligible private schools if they could obtain admission and afford the additional tuition" , revela una comprensión errada del sistema de financiamiento vía subvención. En efecto: a) el sistema de subvenciones no incluye transferencia alguna a padres; b) el sistema de subvenciones no se aplica en algunos establecimientos ("certain eligible private schools"), sino en la totalidad del sistema que recibe financiamiento público (y que atiende al 92% de la matrícula), que incluye: establecimientos municipales y establecimientos privados-subvencionados. Sólo los establecimientos privados-pagados (8-9% de la matrícula), no reciben subvención. En el mismo párrafo 1.3., sugerimos que se mencione que junto a los "many laudatory documents regarding these innovative policies", se mencione que hay una literatura completa que tiene la visión opuesta, tanto en Chile (Mizala y Romaguera, 1998), como en Estados Unidos, (Carnoy y Mc. Ewan, 1997).1 1.4. Este párrafo debe ser reescrito en su totalidad, ya que confunde muy seriamente las categorías básicas de financiamiento y organización del sistema escolar de Chile. Así: a) no se trata de que los establecimientos privados "do not accept vouchers": la ley de subvenciones define que no pueden recibir financiamiento público; b) La afirmación "45% of the students take advantage of the vouchers if they can be admitted and pay" no hace sentido en Chile. La subvención cubre el 92% de la matrícula; en el nivel básico municipal no supone pago adicional alguno; en los establecimientos privados-subvencionados, puede significar un pago adicional ("financiamiento compartido"), sujeto a reglas y controles públicos sobre sus montos . c) "the remaining 55% of the students must go to public schools, municipal and rural". La inmensa mayoría de los establecimientos rurales son municipales. 1 Ver referencias y discusión en artículo sobre educación del libro, G.Perry, D.Leipziger (editores) (1999) Chile. Recent policy lessons and emerging challenges, World Bank Institute, Washington. 29 Annex C 1.5. La afirmación "The heart of the reform of the 1990s. (...) was a poverty alleviation measure to focus (....) on the students of the municipal and rural schools..", es imprecisa y sub-representa el esfuerzo de reforma: a) el corazón del esfuerzo de reforma es el mejoramiento de la calidad de los aprendizajes y la equidad de su distribución social, en el conjunto del sistema escolar (esto es , incluyendo la matrícula privada-subvencionada); b) la reforma incluye dimensiones sistémicas mayores, que van mucho más allá de un programa de acción contra la pobreza, como: cambio profundo del currículum, extensión de la jornada escolar. 1.11. La afirmación "Although the project started as a focused effort to improve the conditions of the municipal and rural schools, subsidized private schools protested, and they also became eligible for most benefits", es una completa equivocación. Las políticas públicas en educación en los 90 están referidas al conjunto de la matrícula subvencionada, es decir, tanto la municipal como la privada subvencionada. El Ministerio de Educación no distingue, en ninguno de sus programas, entre matrícula municipal y matricula privada subvencionada. El proyecto MECE no fue una excepción a este criterio de base. El episodio de la "protesta" de los establecimientos subvencionados nunca existió.2 2.1., 2a línea: Dice "thus began in 1989"; debe decir, 1990. 2.7. El ejemplo de PME "parents paid to paint and repair a school in a periurban area of Santiago" , es inadecuado. Los PME fueron normativamente definidos como obligados a tener un foco en aprendizaje; un foco pedagógico. Ello está en las reglas del concurso de proyectos y en la base conceptual misma de este importante componente del MECE. Es probable encontrarse que una de las líneas de acción de un PME puede corresponder a una acción como la descrita, pero no un Proyecto en su conjunto: simplemente no calificaría como postulable a financiamiento público. Si efectivamente la Misión encontró un PME con las características descritas, ello revela una falla del correpondiente sistema (provincial) de evaluación de proyectos generados por las escuelas, y comentada-destacada como tal. 2.8. La afirmación "many fewer PMEs are funded", es inexacta. El número de PMEs financiados despues de la finalización del MECE-Básica (período 1998- 2000) es de 2.090, con un promedio de 697 al año; el mismo promedio para el periodo 1992-1997, es de 762 PMEs al año. 2 Lo único asimilable a lo afirmado, es que las escuelas que atienden a matrícula con handicaps ("escuelas especiales") no habían sido consideradas en el proyecto MECE inicialmente, y luego de una presión de los educadores y padres de las mismas, fueron incluidos. 30 Annex C 2.11. El párrafo sobre Enlaces está bajo un título erróneo: Affective domain of student learning. Enlaces no tiene tal especificidad o restricción formativa. Las afirmaciones "classes take turns in the computer room, where groups of children explore what is available, do puzzles, or learn word processing", junto a la de que "in the long term software will be produced that corresponds to instructional objectives", transmite una visión "laissez faire" de Enlaces, no muy enfocado en el currículum y en los aprendizajes, que, debe ser matizada, porque ; a) el diseño del proyecto, sus acciones e insumos y lo que se puede observar en cientos de escuelas es un trabajo con una clara vinculación con los aprendizajes. b) la entrega de software que es curriculum -focused, no es un tema del futuro, sino uno ya implementado hace años. En particular, Enlaces ha entregado software educativo de las diferentes disciplinas y capacitado a los docentes para trabajarlos con los alumnos, desde 1997. Por último, refutando la penúltima frase del párrado 2.11 ("Aside from the actual skills of basic computer operation, the effecto of this work on students (...) are yat unknown"), Enlaces sí tiene estudios evaluativos externos respecto al efecto en los alumnos, profesores y escuela en general de su intervención, los que particularmente dan cuanta de los impactos del programa en creatividad, trabajo en equipo, autoestima, motivación, de los estudiantes, etc. 2.16. La referencia a una medición internacional (Third International Math and Science study (grades 3 and 4), ie. TIMSS, y las comparaciones con otros países, son erróneas: a) Chile participó en el TIMSS-R de 1999, ocasión en que ningún otro país latinoamericano participó; además lo hizo con su población de 8° año; b) las comparaciones aparentemente se basan en los resultados del Laboratorio Latinoamericano de Unesco, medición que efectivamente consideró el 30-40 año básico, pero en Lenguaje y Matemática (no ciencias). En tal medición, Chile obtuvo el segundo puntaje de la región en lenguaje, luego de Cuba, y el tercero en Matemática , luego de Cuba y Argentina. 2.20. La afirmación en las últimas dos lineas , "It might be possible to teach children basic literacy through computer programs of the Enlaces system, but programs are not yet available" , es errónea. En 1998 se distribuyó el software Abrapalabras (para aprender a leer y escribir) a más de la mitad de las escuelas incorporadas a Enlaces a esa fecha, y todos los años se entrega software de apoyo, sea en lenguaje o matemática, a las escuelas entrantes. 2.21. La frase inicial está incompleta. ¿Qué pasa con los parents who can vote with their feet? 31 Annex C 2.23. Es nuestra opinión que la forma del párrafo no le hace justicia a uno de los componentes mayores de la reforma iniciada el 96, ciertamente el mayor en términos de recursos: el cambio de la jornada escolar -de tener un sistema de dos turnos de alumnos por establecimiento, medio día cada uno, a uno de un turno que asiste a clases todo el día-. Este componente de la reforma implicó dos cambios legales importantes, tras los cuales hay acuerdos políticos Gobierno-Oposición que hablan de la importancia nacional de este componente de la reforma: a) uno para permitir invertir fondos públicos en infraestructura educacional de propiedad privada (en los establecimienttos privados subvencionados); b) un cambio mayor en la ley sobre el impuesto al valor agregado, en que Gobierno y Oposición acordaron una tasa más alta para financiar la extensión de la jornada escolar. 2.24. Linea 5: no es 180.000 profesores, sino 130.000. 2.24 a 2.26. En el tratamiento del tema teacher training' , no se hace mención a las estrategias de capacitación -in situ, a 20 profesores por establecimiento, prolongadas por dos años- a través de una red de universidades implementada por el proyecto Enlaces. Consideramos que en este contexto debe destacarse la existencia y naturaleza de la Red de Asistencia Técnica Enlaces (RATE). Esta es una dimensión clave de este importante componente del programa evaluado: diecisiete instituciones universitarias que mediante un convenio con el Ministerio de Educación, capacitan en forma permanente y apoyan, tanto educativa como tecnológicamente, a cada escuela que recibe computadores. Se trata de una dimensión de construcción institucional y sustentabilidad del proyecto de primera importancia: luego de 8 años de funcionamiento, esta red de universidades y escuelas trabajando con propósitos comunes, es una fortaleza institucional del sector educativo de Chile. 3.2. La afirmación "quantitative evaluation expertise is limited in Chile", debe ser acotada al ámbito educacional. Hay abundante y de muy buen nivel expertise en Chile en evaluaciones cuantitativas de tipo econométrico, muchas veces aplicadas con éxito a temas de política educacional. 3.6. La frase "municipal and rural schools are still less efficiently run than subsidized schools...", repite los errores categoriales ya señalados al respecto. Prácticamente todas las escuelas rurales son municipales; y ambas categorías son "subsidiadas", o financiadas públicamente vía subvención. La frase, discutible porque ambos tipos de administración trabajan con alumnos y familias de distinto perfil socio-económico y cultural, lo cual define muchos factores de gestión, debe decir: "municipal schools are still less efficiently run than private-subs idized schools.."