ICRR 12881 Report Number : ICRR12881 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/20/2008 PROJ ID : P075189 Appraisal Actual Project Name : Education US$M ): Project Costs (US$M): 12.17 11.05 Improvement Project (Serbia) Country : Serbia Loan/ US$M): Loan /Credit (US$M): 10 11.05 Sector Board : ED Cofinancing (US$M): US$M ): Sector (s): Primary education (32%) Secondary education (32%) Sub-national government administration (18%) Central government administration (18%) Theme (s): Social analysis and monitoring (25% - P) Participation and civic engagement (25% - P) Decentralization (25% - P) Education for all (25% - P) L/C Number : C3636 Board Approval Date : 05/02/2002 Partners involved : Closing Date : 12/31/2006 06/30/2007 Evaluator : Panel Reviewer : Group Manager : Group : Helen Abadzi Kris Hallberg Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: According to the Project Appraisal Document (PAD), the project development objectives were : (a) Building capacity and producing systematic information to help government promote effective education reform; and (b) Improving teaching and learning in schools and empowering local communities to take responsibility for their schools. The PAD and the Legal agreement differ considerably in their statement of objectives . According to the Legal Agreement, "The objectives of the Project are to assist the Republic of Serbia to further define and begin to implement a process of education reform through strengthening institutional capacity, improving information collection, and decentralizing education responsibilities and financing ." The PAD includes an additional objective of “improving teaching and learning in schools � and does not have an explicit goal (stated in the legal agreement) of “decentralizing education responsibilities and financing .� Rather than aiming for a formal process of decentralization, the PAD emphasizes “empowering local communities to take responsibility for their schools �. The reasons for the discrepancy between the Legal agreement and the PAD are unclear, but the detailed project description in both documents suggests that the project aimed at a broader set of objectives than those formally stated in the Legal agreement. The ICR authors noted the difference and chose to evaluate the project according to the PAD, but references are also made to a combined set of objectives (ICR p. 11). This review uses the objectives as presented in the Project Appraisal Document because the PAD is more outcome -oriented. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): a) School improvement grants (US$5.54m at appraisal, US$5.58m actual) to solve priority school problems identified in school development plans; (b) Standards and assessment development (US$2.06m at appraisal, US$1.67m actual) to (i) improve Serbian education through the use of internationally recognized standards; (ii) establish a modern and internationally-recognized system for assessment and examinations; and (iii) establish and develop a sustainable Education Standards and Assessment Service for Serbia; (c) Education information system (US$2.56m at appraisal, US$1.72m actual) to promote effective management of resources and monitoring of academic performance at central, regional, municipality and school levels; (d) Education decentralization (US$1.03m at appraisal, US$1.72m actual) to strengthen governance and management capacity in primary and secondary education . through (i) improving the capacity of municipalities to support to schools located within their jurisdictions; (ii) to improving the capacity of school directors to manage schools and to work effectively with school boards; (iii) strengthening the role of the Ministry of Education and its school inspectors. During implementation, a model school program was included under this component . Indicators were changed to reflect the focus on model schools, but project objectives were not changed . (e) Project managemen t (US$0.49m at appraisal, US$0.33m actual) to manage the project. Note: The cost data of the ICR show inconsistencies . The figures given above are derived from the ICR Annex 1 and are different from the figures in the ICR Section 1.5. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: After an extension of six months, the project was closed on 6/30/2007 and disbursed fully. Due to changes in exchange rate of SDRs, more than the original US dollar amount was disbursed . 3. Relevance of Objectives & Design: The project development objectives were highly relevant . They were consistent with the Bank's 2001 Transitional Support Strategy (TSS) that aimed to provide analytical and advisory assistance . The objectives remained consistent with the 3-year Country Assistance Strategy (CAS) that was prepared in 2004. Both the TSS and CAS emphasize the importance of improving the social well -being of the most vulnerable, building human capacity, improving governance and building more effective institutions . The project's objectives were well -aligned with both TSS/CAS objectives by investing directly in improving learning opportunities for children . The design was substantially relevant to the country and sectoral priorities and to the achievement of objectives , but there were some shortcomings. School development plans and grants are common components of projects in middle-income countries, but it is unclear how they were to improve teaching and learning in schools or whether they did. The PAD did not establish a clear line of reasoning leading from these activities to learning outcomes . Rather than results, the ICR offered frequency counts as indicators (e.g. number of grants given, participation in international assessments). The ICR points out that the school grant compone nt "was only to a limited extent integrated into the ministry’s and municipalities’ other activities. The funds were designed to flow in parallel to other funds supporting schools, and the activities envisioned to be carried out by schools were not designed to be carried out jointly with the ministry’s education inspectors. This lack of integration placed the component without a clear sponsor and supporter within the ministry." However, the region clarified that the grant program was managed by a public official leading the Ministry’s unit on school improvement and development . 4. Achievement of Objectives (Efficacy): (a) Building capacity and producing systematic information to help government promote effective education reform (substantial): A single Education Information System was developed to cover 88% of schools. A law on Local Self-government Financing was put in effect in 2006, defining clearer roles and responsibilities in financial planning, executing and reporting to direct and indirect users . Monitoring of basic performance indicators became possible by municipality, including share of local funding for education, overall spending per student and student -teacher ratios. The Ministry of Education is committed to use the data for future budget preparations . However, a funding formula to monitor system efficiency and equitable per -student distribution of public funding for education in Serbia could not be established during the life of the project . 90% of the principals were trained in school management and use of the management information system . An institute for the evaluation of education quality was established and equipped, and staff received training . However, the center was closed down in 2004, when a new government with different philosophy took over . When it reopened, some of the trained staff had left and did not rejoin . (b) Improving teaching and learning in schools and empowering local communities to take responsibility for their schools . The objective has two subobjectives : (i) Improving teaching and learning in schools (modestly achieved): The ICR mainly presents evidence on outputs rather than outcomes. There is no evidence that teaching improved or that modern methods indeed improve teaching. The ICR shows that: o A School Board Manual was produced and used during training in 1174 schools. o Out of the 1684, more than 800 schools applied for and received school grants (650 were primary). Intensive training was conducted in schools receiving school grants . But outcomes of the school grants are not clearly outlined in the ICR. o Extensive teacher training took place in information technology; 37% of a sample of teachers who received training stated in a survey that they regularly use computers in teaching . o Teachers were trained in instructional methods; 80% of a sample of teachers trained in the use of modern teaching methods apply the new skills to the majority of lessons they teach . Teachers from schools which received the school grant stated that they use new teaching methods more often (when compared to a control group of teachers in schools that did not get grants ). o By means of a national assessment, achievement standards were set for Serbian language and Mathematics at the end of grades 3 and 4 of primary education. Standards for the end of compulsory education were also developed for 10 subjects. A 5% and 7% sample of students were assessed at the end of grades 3 and 4, respectively, but the ICR does say much about outcomes . Also, Serbia participated in PISA 2006 and TIMSS 2007, but the ICR does not say how it performed . (ii) Empowering local communities to take responsibility for their schools (substantially achieved). All 1684 primary and secondary schools of the country received one -day training and became able to develop school plans. Local communities were involved in identifying problems and finding solutions to address them . Survey evidence showed that in 80% of representative sample of schools, stakeholders expressed satisfaction with the opportunity to participate in developing plans . As a result, the Serbian government decided to introduce a law to make it a requirement for all schools to prepare school development plans . An increased number of extra-curricular activities was reported in 10% percent of the schools which received a school development grant. Since this ICR Review evaluates the project against the objectives as stated in the PAD, the results of the decentralization component are not reported here; see Section 11. 5. Efficiency (not applicable to DPLs): Overall, efficiency is rated modest . The ICR states that evidence regarding efficiency is limited and provides evidence that is mixed. A funding formula to monitor system efficiency and equitable per student distribution of public funding was not finalized or used . On the other hand, textbook production costs have been reduced by 33%, while standards and guidelines of rehabilitation may make it easier to plan continued maintenance and minimize expensive repairs. It is unknown how efficiently school grant funds were spent (US$7000 per school); 40% of the funds were intended for equipment provision and small adaptation works while 60% were intended for inservice teacher training . The ICR notes that component lacked a clear sponsor because it was not well integrated with the ministry ’s and municipalities’ other activities. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Outcome is rated moderately satisfactory on the basis of substantial relevance, substantial efficacy, and modest efficiency. The ICR does not present sufficient evidence on the achievement of teaching and learning outcomes, and there were shortcomings in the achievement of the capacity -building objective. Despite these limitations, the school development planning process and the new assessment regime have made it possible for the project to have a nationwide impact on schools. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to development outcome is rated moderate . Government commitment is not assured, particularly given the reluctance to implement components related to decentralization . There is little long-term experience to indicate how sustainable school grants and school development plans are . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Bank performance during project preparation was moderately satisfactory . The Bank did not carry out formal AAA work prior to appraisal but prepared components satisfactorily, including the school grants component that was extensively prepared in partnership with the government . But overall, project objectives were not easily operationally defineable and their indicators were hard to measure . During implementation Bank performance was satisfactory. At mid-term review, efforts were made to bring the project back on track, and the Bank provided extensive assistance at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government performance was moderately unsatisfactory, while the performance of the implementing agency was moderately satisfactory. Serious delays occurred early on in the preparation of school rehabilitation lists . Subsequently a new government replaced critical staff and essentially stopped the work for about a year . After the mid-term review, the project was restarted and most activities were completed . a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: The monitoring and evaluation design are rated modest . An evaluation center that was set up by the project was closed when the government changed in 2004. As a result of delays caused by government decisions, limited work was carried out in evaluation of learning outcomes . Serbia participated in two international tests (PISA in 2003 & 2006 and TIMSS in 2003 & 2007.) Also, national assessments of student achievement have been performed at 3rd and 4th grade, but the ICR does not report any learning outcomes from these tests . Furthermore, there were no measurements to show how school development plans and grants improved learning outcomes . It is unclear whether the students' test scores were used for decisions and for the development of quality improvement activities . Also, the items listed as indicators in the PAD are really objectives to be achieved rather than variables to be monitored. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The focus of the decentralization component was changed at the mid -term review. Instead of focusing on preparing analytical reports to support more decentralization the component was changed to support “model schools�. The staff of 50 model schools were trained in improving school performance quality in key areas, through self-evaluation. The ICR reports that performance has improved, though it does not give details but states that 45 of the model schools were asked to mentor other "partner" schools, and this is an indication that model schools have improved their quality. A positive but unintended impact of the project was that the government decided to introduce a law making it mandatory for all schools to prepare school development plans . 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Moderately Moderately Satisfactory, since Quality Satisfactory at Entry was rated Moderately Satisfactory and Quality of Supervision was rated Satisfactory. Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Investment projects in education must be focused on clear objectives and measurable indicators . Lack of clarity may result in limited commitment to implement a project . The development of evaluation capacity and use of student assessment data for decision -making is a complex and time-consuming process, especially if limited data are available at the outset . Efforts must be made to demonstrate to governments the benefits of evaluation and monitoring data . Objectives involving decentralizing financing and management of schools require strong and broad government ownership. Government changes may delay the implementation of such activities, that may be considered politically sensitive . 14. Assessment Recommended? Yes No Why? There were many management oriented components whose line of reasoning is unclear, and whose impact could be assessed. Also there is a need to gather more evidence on the achievement of teaching and learning outcomes. 15. Comments on Quality of ICR: Overall, the ICR quality rated satisfactory but with reservations . The ICR did a good job of structuring the output data and limited outcome data to assess the achievement of all of the project objectives (from both the PAD and the DCA). The construction of a matrix linking outputs and outcomes (Annex 10) was useful. Nevertheless, the ICR failed to present student achievement data, which should be available, and did not explain why they were not presen ted. It also offered no information regarding the outcomes of school grants or even the specific needs that they were expected to fulfill in various schools . The report on p. 22 only states that "80 percent of schools that received grants are achieving goals planned by the school project ". Thus it is unclear how loan proceeds were spent on or how efficiently these funds were spent . It is also unclear how the 80% figure was obtained and what the expression "are achieving goals" means. The ICR also had some inconsistencies in cost data . The figures given in section 2b above are derived from the ICR Annex 1 and are different from the figures in the ICR Section 1.5. a.Quality of ICR Rating : Satisfactory