ICRR 12118 Report Number : ICRR12118 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 07/18/2005 PROJ ID : P006181 Appraisal Actual Project Name : Bo- Education Reform Project Costs 178.9 US$M ) (US$M) Country : Bolivia Loan/ US$M ) 40 Loan /Credit (US$M) 35.9 Sector (s): Board: ED - Primary Cofinancing 95.4 education (29%), US$M ) (US$M) Secondary education (28%), Tertiary education (23%), Central government administration (20%) L/C Number : C2650 Board Approval 95 FY ) (FY) Partners involved : IDB, SIDA, KfW, Closing Date 01/31/2002 07/31/2003 Netherlands, Other Prepared by : Reviewed by : Group Manager : Group : H. Dean Nielsen Roy Gilbert Alain A. Barbu OEDSG 2. Project Objectives and Components a. Objectives The objective was to support the first phase of Bolivia's Education Reform Program designed to improve the quality and equity of primary education, strengthen the ability of the National Education System (SNE) to set policies and guidelines for the education sector, and strengthen the capacity of the public education system to deliver education services effectively and efficiently . b. Components 1. System Transformation and Strengthening (Estimated: $25.4 million; Actual: Undetermined), covering registration of teaching and non-teaching personnel; rationalization of staffing; provision of training and technical assistance for technical and administrative personnel; the provision of information and technical assistance to community organizations; design and administration of an administrative system; design and implementation of a personnel management system; and repair and reconditioning of office space . 2. Pedagogical Quality Improvement (Estimated: $112.9 million; Actual: Undetermined), covering transition in bilingual and multi-grade education; curricular development, production and distribution of school equipment and learning materials (including teachers' guides); inservice training; and pedagogical advice for teachers; reform of initial teacher training; educational research and pilot programs, and development and implementation of a national learning assessment system . 3. Project Support and Monitoring (Estimated: $12.2 million; Actual: Undetermined), covering establishment and functioning of a technical services division; a management information technical division; a special monitoring and support unit; and public communications . c. Comments on Project Cost, Financing and Dates Financial commitments were for $178.9 million in "core funding" by partners, including $40 million by IDA, $52.5 million by IDB, $14 million by SIDA, $8.5 million by KfW, $7.5 million by the Netherlands, $12.9 million by various other sources, and $43.5 million by the Bolivian Government. IDA allocations (not including contingencies ) consisted of $4.3 million for Component 1, $24.9 million for Component 2, and $7.5 million for Component 3. Due to flawed financial accounting in the Project, records of expenditures were incomplete and thus totals could not be provided . IDA disbursements were 12% below estimates for Component 2 and 18% above estimates for Component 1 and 3, which were in practice eventually merged . At the mid-term evaluation major restructuring was undertaken, which, although not changing Project objectives, changed financial allocations : IDA amounts for books, materials and goods, and for Ministry salaries increased by 62% and 230% respectively; whereas amounts for civil works, training and operating costs fell by 68%, 72%, and 36% respectively. 3. Achievement of Relevant Objectives: 1. Quality and equity of primary education . Partially achieved. There were no outcomes indicators of quality and equity, but output indicators demonstrated that books, equipment and materials (based on a new active learning curriculum) were delivered to schools and teachers were given inservice training (although about 25% fewer than planned). Student learning outcomes were measured both by a UNESCO regional exam of primary language and math achievement (LLECE) and the assessment system created by SIMECAL under the project . On the LLECE exam for language Bolivia ranked 8 of 11 reporting countries and in math it ranked 6th; however, all of the countries that scored above it had higher GNPs per capita . Given the fact that this exam was only given once, it could not show whether there were any changes in outcomes connected to Project interventions . However, analysis of the SIMECAL results (not discussed in the ICR) showed no difference between schools which had been applying the new curriculum and those still teaching the old (Joint Evaluation of External Support to Basic Education in Developing Countries - Bolivia Case Study, 2003). Concerning improved equity, there were no indicators for this objective. The ICR does mention enrollment growth for girls relative to population increase, but it is not clear if this has resulted from the Project since there was a parallel project mounted in 1998 (Bolivia Education Quality and Equity Strengthening Project ) which targeted improvements in female participation . The ICR also mentions the existence of a large number of intercultural, bilingual schools (almost 3000 by 2002) but does not provide a baseline, and does not indicate how many were created under the Project . Most accounts seem to indicate that the large indigenous population is being served better; however, there were no breakdowns for ethnic minorities in the completion rate statistics nor in the learning outcome measures (all the ICR provides is an unsubstantiated statement that educational attainment among indigenous group students is improved ). 2. Strengthening of SNE ability to set policies and guidelines . Fully achieved. The Education Reform Program, backed by the Education Reform Law, has made a large difference in the capacity of the Ministry (SNE) to provide leadership in the sector and to create policies and guidelines . Under the Project SNE created new policies and guidelines for the national curriculum in primary education, pre - and inservice teacher training, assessment of learning, the use of bilingual education, and the community participation in school decision making . Many of the policy changes were contested by vested interests, but the SNE had developed the strength and the political will to push them through. 3. Strengthening the capacity of the public education system to deliver education services effectively and efficiently . Mostly achieved. Most of the targets on performance indicators of effectiveness and efficiency tracked by the project were met by the project, except those for staff and teacher training . Especially notable was the establishment of school "nucleos" and parent/community councils (decentralization), and the increase in completion rates (both urban and rural). Also, under the Project strong technical capacity for curriculum design, information systems development and use, teacher and student assessment, and educational research and planning were established. Delivery of services (e.g., instructional materials and training ) were hampered by weaknesses in procurement capacity and financial reporting was generally below standard . Also, there is little research evidence concerning the use and effectiveness (at the school level) of the new curriculum and related materials and teacher training. 4. Significant Outcomes/Impacts: 1. The transformation of the National Education System from a weak agency beholden to special interests to a dynamic and effective organization capable of making decisions and implementing reform programs; 2. The complete revision of the primary school curriculum (towards a more active learning approach ) and production/distribution (although belated) of related learning materials and teacher training, including those for bilingual schools and teachers; 3. The use of public communications strategies to mobilize parental and community involvement in school level forums (Juntos ) and decision making; 4. Improving the capacity for education system monitoring and evaluation through improved management information and student assessment data; 5. Reforming preservice teacher training (connecting it to universities, linking it to the new curriculum and opening it to university educated candidates ) and establishing the foundation for an ongoing, localized teacher advisory service. 5. Significant Shortcomings (including non-compliance with safeguard policies): 1. Failure of the Project Management (in the SNE) to account for the external funds provided and to report expenditures by project component, undermining the basis for assessing Project effectiveness in relation to expenditures (efficiency); 2. Failure of the IDA Project "at entry" to recognize serious SNE weaknesses in financial management and procurement (which troubled the Project throughout ) and to emphasize capacity building for this during project preparation and the early stages of implementation; 3. Underdevelopment of systems and capacity for educational outcomes assessment (designing good outcomes indicators, use of time series and control groups, diagnostic use of examination data, break down of data by ethnic group, etc.); 4. Failure to establish the roles and functions of the pedagogical advisors, trained by the project to visit schools and provide guidance to teachers on improved teaching . Their use was discontinued during the project due to opposition by the teachers' union and some school directors, which might have been avoided if possible objections by these groups had been taken into account from the beginning ). 6. Ratings : ICR OED Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Satisfactory {The ICR's 4-point scale does not allow a "moderately satisfactory" rating.] There was strong institutional capacity building in the project and delivery of inputs and services, but it was not clear if project interventions have led to better learning outcomes; what little evidence there is (national assessment) suggest that the Project has had little impact on learning (see Joint Evaluation ). Equity outcomes were also poorly documented. Project efficiency could not be determined due to failures in financial accounting . Institutional Dev .: High Modest The ICR rating is confirmed based on strong evidence of the almost complete overhaul and modernization of the education management and service delivery systems. . However, OED considers that Bolivia's ability to use financial resources for education more effectively for these purposes is undermined by a lack of accountability deriving from very weak financial management that was unable to report the project's final costs. Sustainability : Highly Likely Likely The sustainability of many of the innovations that require recurrent funding (a continued stream of books and materials, inservice training and advisory services, management capacity building at the local level) appear to depend on continued development assistance; not many of the excellent and theoretically sustainable project elements have been covered by routine Government funding . Bank Performance : Satisfactory Satisfactory Borrower Perf .: Satisfactory Satisfactory Quality of ICR : Satisfactory NOTE ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness. NOTE: 7. Lessons of Broad Applicability: 1. The use of the Ministry of Education (SNE) offices and officers to implement the programs and reforms covered by the Project (National Education Program), instead of Project Implementation Units and consultants (as originally planned), was a key factor in building institutional capacity and local ownership . 2. The political culture of education and educational decision making can be transformed by mobilizing popular participation in school councils and other forums, such as the Indigenous People's Education Councils (CEPOs), particularly in situations where change has been stalled by clashes between the central government and powerful vested interests (teachers' unions, universities and the Church ). 3. Providing beneficiaries with choices on educational innovations is important . In the Project, bilingual education was successful when communities were included in the decision to adopt it (and failed when they were not); also, the use of pedagogical advisors might have survived if they had initially been used in schools that wanted them . 4. Where teachers unions are strong, effective ways of accommodating them must be planned from the beginning . Sometimes this will mean working with them; sometimes building a base (e.g., public opinion) strong enough to override them. In any case, union concerns will need to be listened to and creative ways developed for bringing them into the changes process . 5. A key to Project success was the presence and continuity of high -quality technical teams which were involved in project planning and remained almost unchanged during the length of the Project, despite several changes in government (president and ministers). 6. From the beginning (including during Project preparation ) key political leaders took care to position the education reform package (including points enshrined in the Education Reform Law ) as the product of a national consensus and not as the province of a single party or interest group . For this reason, the Law and its reforms were supported by a long succession of presidents and ministers . 7. Continued support for reform agendas, like the one in Bolivia, can be bolstered by the results of rigorous outcomes evaluations. Recent growth in evaluation capacity in the country should allow it to build impact evaluation elements into future programs (implemented either inside or outside its projects ), which can help to establish that changes are attributable to program interventions and not to other factors, such as (in the case of Bolivia during the past decade) growth in the national economy or changes in the law . 8. Assessment Recommended? Yes No Why? The adoption of the above reforms is a major turning point in Bolivian education; it would be desirable to see the extent to which they are actually taking root and can be built on for further improvements, including improved learning outcomes . A PPAR would also allow evaluators to look more closely at Project expenditures and at whether the Ministry has become more accountable . 9. Comments on Quality of ICR: The ICR was informative and insightful, but could have focused more clearly on the Project's development objectives (for example, improved equity) and to bringing specific evidence to bear on how effectively they were met . It is also curious that the ICR about a project to improve educational quality would not report the results of national educational assessments . The seriousness of poor financial management , resulting in lack of data on project costs and inadequate accountability, was also underplayed. The absence of cost data calls into question the meaning of the ERR estimates provided by the ICR .