Document of The World Bank Report No: 29923 IMPLEMENTATION COMPLETION REPORT (IDA-32200 TF-25581) ON A CREDIT/GRANT IN THE AMOUNT OF SDR 3.7 MILLION (US$5 MILLION EQUIVALENT) TO THE AZERBAIJAN REPUBLIC FOR AN EDUCATION REFORM PROJECT September 18, 2004 Human Development Sector Unit Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective September 8, 2004) Currency Unit = Azerbaijan Manat (AZM) AZM 4,904 = US$ 1 US$ 1 = AZM 4,904 FISCAL YEAR January 1 - December 31 ABBREVIATIONS AND ACRONYMS CAS Country Assistance Strategy CIS Commonwealth of Independent States ERP Education Reform Project ESDP Education Sector Development Project ICR Implementation Completion Report IDA International Development Association IEP Institute on Educational Problems GDP Gross Domestic Product LIL Learning and Innovation Loan M&E Monitoring and Evaluation MOE Ministry of Education OECD Organization for Economic Cooperation and Development PAD Project Appraisal Document PCD Project Concept Document PHRD Policy and Human Resources Development Fund PISA Programme for International Student Assessment PIU Project Implementation Unit PSR Project Status Report QAG Quality Assurance Group SIL Specific Investment Loan TEI Teacher Education Institute TRC Teacher Resource Centers TTI Teacher Training Institute Vice President: Shigeo Katsu Country Director Donna Dowsett-Coirolo Sector Director Charles C. Griffin Sector Manager Maureen McLaughlin Task Manager: Halil Dundar AZERBAIJAN REPUBLIC EDUCATION REFORM PROJECT CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 6 5. Major Factors Affecting Implementation and Outcome 13 6. Sustainability 14 7. Bank and Borrower Performance 15 8. Lessons Learned 18 9. Partner Comments 21 10. Additional Information 21 Annex 1. Key Performance Indicators/Log Frame Matrix 22 Annex 2. Project Costs and Financing 24 Annex 3. Economic Costs and Benefits 26 Annex 4. Bank Inputs 27 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 29 Annex 6. Ratings of Bank and Borrower Performance 30 Annex 7. List of Supporting Documents 31 Annex 8. Borrower's Completion Report (Summary) 32 Project ID: P057959 Project Name: EDUC REF Team Leader: Halil Dundar TL Unit: ECSHD ICR Type: Core ICR Report Date: September 18, 2004 1. Project Data Name: EDUC REF L/C/TF Number: IDA-32200; TF-25581 Country/Department: AZERBAIJAN Region: Europe and Central Asia Region Sector/subsector: Tertiary education (62%); Central government administration (14%); Primary education (12%); Secondary education (12%) Theme: Education for all (P); Social analysis and monitoring (S) KEY DATES Original Revised/Actual PCD: 11/11/1998 Effective: 09/08/1999 09/08/1999 Appraisal: 03/09/1999 MTR: 03/10/2001 03/10/2001 Approval: 05/24/1999 Closing: 12/31/2002 03/31/2004 Borrower/Implementing Agency: GOVERNMENT OF AZERBAIJAN/Ministry of Education Other Partners: STAFF Current At Appraisal Vice President: Shigeo Katsu Johannes F. Linn Country Director: Donna Dowsett-Coirolo Ishrat Hussain Sector Manager: Maureen McLaughlin James Socknat Team Leader at ICR: Halil Dundar Philip O'Keefe and Halil Dundar ICR Primary Author: Pasi Sahlberg 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome: S Sustainability: L Institutional Development Impact: M Bank Performance: S Borrower Performance: S QAG (if available) ICR Quality at Entry: S S Project at Risk at Any Time: Yes 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: Demographic context: The population of Azerbaijan was over 8 million in 2002. The population growth rate has slowed steadily. Average annual growth rates were 1.8 percent in 1970-1979 and then 1.5 percent in 1979-1989. Between 1989 and 1999, the growth rate slowed to just 1.3 percent, and fell further to 0.8 percent in 2000-2002. Together with some net emigration, the declining population growth rate in the 1990s is attributable to a fall in the birth rate from 26.3 per 1,000 of the population in 1990 to 13.8 per 1,000 in 2001. More than 80 percent of the population is Muslim: this includes the Azeris, Georgians and other groups such as Lezghins and Kurds. Most Azerbaijanis are Shia Muslims, but around 30 percent are Sunni Muslims. Azeri is a Turkic language closely related to Turkish and more distantly to the Turkic languages of Central Asia. Educational context: Azerbaijan, like other countries in the Commonwealth of Independent States (CIS), inherited a relatively well-developed educational system. However, since independence, the educational system has been deprived of resources, and the general standard of education has visibly declined. Public expenditure on education was only 3.5 percent of gross domestic product (GDP) in 2001. The situation has been further exacerbated by the emigration of many highly educated professionals. In addition, the Nagorno Karabakh conflict, which led to the displacement of over one million people, also resulted in the loss of all schools in that territory (excluding Nagorno Karabakh), amounting to about one-fifth of Azerbaijan's total number of schools. The Law on Education of 1992 permits both state and private educational institutions, but the latter are still undeveloped. By 2002, about 60 percent of the population over the age of 15 had received at least the stipulated eight years of schooling, down from over 90 percent during the Soviet period. Azerbaijan has been struggling to retain the positive achievements of the past education system while introducing education sector reforms appropriate to the needs of an independent democratic state and an emerging market economy. During the Soviet era, all curricula were developed, standardized, and issued from Moscow with little or no input from the local levels. The major problems concern the process by which curricula are formulated due to the hiatus caused from the break-up of the Soviet Union, limited adjustments in curricula and syllabi to reflect the needs of a market economy and democratic society, overcrowding of the current curricula for general schools, and restricted funds for curriculum development institutions to perform their functions. Furthermore, the authorities have had limited technical capacity for curriculum development (in terms of the number, qualifications, and experience of staff), with no existing curriculum developers specifically trained in this area. The present teacher education system consists of three elements: pre-service education, in-service training and re-training by gaining knowledge of a different subject field or while changing from another occupation to become a teacher. The main problems in teacher education include: poor integration within and duplication of functions among some of the institutions, narrowly specialized teacher education programs for upper grades, lack of functional relationships with schools, inadequate funds for facilities, equipment, books, and other essential teaching materials, and lack of exposure of staff in teacher education institutions to modern teaching methods and teacher training delivery methods. - 2 - Azerbaijan shares the characteristic low student/teacher ratios of most CIS countries, with a ratio of 10:1 in general education. This compares to an Organization of Economic Cooperation and Development (OECD) ratio of around 15:1 and far higher ratios in middle and low income countries. This is caused mainly by the inherited policy of having single subject teachers in secondary (Grades 5-11) education, and the decrease of a "full" teaching load from eighteen to only twelve hours per week. Despite falling real wages and a drop in the birth rate, the number of education sector workers increased by more than 10 percent between 1992 and 1998 (though there are continuing problems with accurate measurement, due to the frequent confusion between actual teachers and authorized teaching positions). During the Soviet period, teachers were given little discretion as to what to teach and how to teach. One of the main characteristics of teaching was the focus on learning facts rather than acquiring skills that allow knowledge to be applied in various situations. Hence, methods of instruction in Azerbaijan are not based on active and problem-based approaches of teaching which are preferred in many contemporary education systems, but are typically rigid and authoritarian, with insufficient attention paid to the needs of individual students. Even where there is a desire to move away from this model, administrators face lack of new teaching resources and other professional development support to teachers on contemporary teaching methods. Project design and objectives: The Education Reform Project (ERP) was the first Bank-financed support to Azerbaijan education sector development. The project supported the Country Assistance Strategy (CAS) (August 27, 1996) objective of the need for "social policies that preserve Azerbaijan's human capital and protect those vulnerable to changes that are sweeping the economy." In particular, it referred to the need to develop a "more rationalized system for delivery of basic social services". An additional related CAS objective was "improving the allocation efficiency of public resources". As the education sector accounts for around 50 percent of total public sector employment currently, promoting more efficient training and use of teachers as anticipated under the project is critical to realization of this objective. The Learning and Innovation Loan (LIL) was designed to strengthen the policy and institutional framework and experiment with new models within the education sector. The belief was that a LIL would provide the analytical and implementation experience to consider a larger operation in the sector in the future. Moreover, it was expected that the lessons from the LIL and the formal approval of the Government's Education Reform Program together would provide a strong programmatic basis for such an intervention. The overall development objective of the project was to improve the effectiveness of teaching and learning in selected grades of general schools in Azerbaijan through: (i) supporting innovations in curriculum content and the curriculum development process; (ii) piloting new teacher education content (including new teaching methods) and delivery mechanisms; and (iii) evaluating the impact of project interventions through monitoring and evaluation in teacher education institutions and pilot schools in order to provide lessons for broader sectoral reform. The project development objective was consistent with the Government's own education reform program and also closely related to the Country Assistance Strategy of that time. However, the terminology (e.g., effectiveness, innovations) that was used to formulate that objective is not very clear and actually leads to considerable difficulties in evaluating the project. Effectiveness of teaching and learning should be defined through the intended outcomes of schooling, not using any general determinants. Two issues should be noted at this point. First, the intended outcomes, i.e., educational goals of general education should be established prior to intervention in order to be able to conduct effective teaching and learning practices. As the content of teaching and learning was changing in the pilot schools, so were the intended outcomes. It was assumed in the project that students will not only learn better but will also acquire different knowledge - 3 - and skills than their peers in non-pilot schools. Second, both baseline data on teachers' pedagogical behavior and students' learning outcomes, and comparative data from non-pilot schools should be available in order to judge how the effectiveness of teaching and learning has changed. The project documentation is not specific in what exactly was to be improved. The main means that were selected to lead to improved effectiveness of teaching and learning were curriculum reform and development of related educational materials (for Grades 1, 2, 5 and 6), and professional in-service training of teachers on new teaching methods and curriculum. In summary, the formulation of the project development objective indicated the core areas to be improved within the education system, but this formulation was not appropriate for a LIL. Instead of aiming at improved effectiveness, the project development objective should have emphasized the learning and innovation nature of a LIL. For example, the project development objective could have focused on improving the Ministry's (including regional and local education administration) knowledge and skills to design, implement and monitor education sector reforms. Selecting a pilot approach in this LIL was a good choice. Including just twenty schools and seven teacher training institutes from five rayons enabled more intensive and focused implementation that is a requirement for improving effectiveness of teaching and learning in schools. In this sense, the project provided an exceptional opportunity for the Ministry of Education (MOE) and the participating institutions to learn about effective alternative arrangements in school development and educational change. However, piloting could have been based on more than one approach in order to provide multiple sources of information about what solutions seem to work in developing education. The key question in reviewing the overall project is how much beneficiaries were able to learn through this project and what they will be doing differently when the education reform will be scaled up in Azerbaijan. 3.2 Revised Objective: The original objectives of the Education Reform Project were not revised. 3.3 Original Components: The project had three components: (1) Curriculum Development; (2) Teacher Education; and (3) Project Management, and Monitoring and Evaluation. What follows is a brief description of these components. Component 1. Curriculum Reform (US$1.5 million) The project was implemented during the period of 4.5 years (instead of 3 as intended) and it was hence agreed that curricula will not be completed for all grades in general school, though much of the support on curricula for selected grades was relevant to all grades of general school (e.g. on curriculum development organization; active learning methods). The project supported curriculum reform through the following activities: o international and local consultant support on main areas of curriculum development, including syllabus, teaching methods, institutional issues, and assessment (including textbook development); o training workshops with local and international consultants for local curriculum developers and those training teacher trainers in new curriculum, a small number of fellowships (2-3) for Azerbaijani specialists to undertake studies abroad in curriculum development, and study tours to relevant countries in the region and Europe; o basic refurbishment for two curriculum development institutions; o equipment, materials and supplies for local curriculum developers to have access to resource materials on international and regional practices in curriculum development and reform; - 4 - o printing of new syllabi and teachers guides for Grade 1 nationally and Grade 5 in pilot schools in the first year, with phased roll-out into higher grades thereafter; and o operating costs (salaries; utilities, etc.) of the curriculum development institutions participating in project-financed activities to be part of the Government contribution. Component 2. Teacher Education and Pilot Schools (US$3.0 million) This component had two sub-components: Sub-component 2.1. Pilot teacher education (US$1.4 million). This component worked with seven in-service Teacher Training Institutes (TTIs) in five pilot regions. The project supported capacity building and reorientation of these selected institutions. This required development of new teacher training materials in selected areas, and working with the MOE and pilot TTIs to strengthen their training capacity and diversify provision of in-service training. The project focused on technical assistance on teacher training and teaching methods development, training of teacher trainers, minor rehabilitation and refurbishment of institutions, and financing of training materials, equipment, materials and supplies. This included an evaluation of how various teacher training initiatives financed under the project affected teaching and learning, both in pilot TTIs and in pilot schools. Sub-component 2.2. Pilot Schools in Five Proposed Regions (US$1.6 million). Twenty pilot schools in five pilot regions were selected in the areas where teacher training institutions are located to assess and evaluate newly developed curricula and teaching methods for selected grades. The project financed teaching materials (including textbooks, supplies, and other selected materials), equipment, training for both teachers and administrators, and modest rehabilitation and refurbishment of pilot schools as part of the improvement of the learning environment. In addition, the training and outreach activities aimed at strengthening parent and community involvement in school operations. The component included the following activities: o training of trainers through workshops, short courses, seminars, and study tours about new curricula and teaching methods, including non-TTI-based provision of training for teachers; o training of teachers and principals through seminars, workshops, and short courses about new curricula and teaching methods; o technical support to establish twinning arrangements between pilot teacher training institutes and pilot schools; and o support to pilot teacher training institutes to become teacher training resource centers in the selected rayon. In this respect, the support included textbooks, references, and other learning materials. Component 3. Project Management, and Monitoring and Evaluation (US$0.6 million) This component had two sub-components: Sub-component 3.1. Project Management (US$0.4 million). This component supported operations of the Project Implementation Unit (PIU) during project life, including training, equipment and salaries. This included support to PIU Director, PIU staff in financial management, procurement and administration, and basic equipment needs. Project management was supported by international consultancies for project advisory assistance (e.g., implementation specialist, financial management specialist, procurement specialist, etc.). Furthermore, additional capacity development was offered to the staff of the MOE. - 5 - Sub-component 3.2. Monitoring and Evaluation (US$0.2 million). This sub-component was an integral part of the project because of its learning and innovation aspects. For this reason, the project supported not only regular monitoring and evaluation (M&E) of the set of key performance indicators to monitor project impact, but also several studies to evaluate the implementation of innovative elements. This sub-component also financed several evaluation studies about the performance progress of the project, including Education Cost and Finance Study, and Textbook Publishing and Distribution Policy Study. However, Azerbaijan's planned participation in the Organization for Economic Cooperation and Development (OECD) Programme for International Student Assessment (PISA) study, which focuses on developing internationally comparable measures of student achievement, was not actualized. 3.4 Revised Components: There were no amendments to the original project components or the overall structure of the project. 3.5 Quality at Entry: According to the available references and considering the country circumstances, the quality at entry of the project was satisfactory. The project was prepared according to the Quality Assurance Group (QAG) standards of the Bank. It was subject to normal peer review process and had political support of the Government of Azerbaijan. The project design was clearly in line with the Country Assistance Strategy (CAS 15948-AZ, dated August 27, 1996) and with the Government's Education Reform Program. Relevant lessons from other International Development Association (IDA) Learning and Innovation Loans (LILs) were studied during the project design. Social assessment and education management review were financed by the Dutch Trust Fund and the implementation capacities were assessed through financing from a Policy and Human Resources Development Fund (PHRD) grant. No baseline data for project performance indicators were available at the start of the project implementation. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: The overall outcome of the project was satisfactory. The goal of the Education Reform Project was to speed up the education sector development by having an impact on teaching and learning in selected pilot schools and selected age groups of pupils in general education schools and to learn from those experiences for putting the reform in a national scale through a larger project. The intended outcomes of this project were strengthened policy and institutional frameworks and new knowledge through experimentation with new models and ideas in developing the education system. Since the instrument used was a LIL, the project goal was also to assess the impact of project interventions through continuous monitoring and evaluation of the Teacher Training Institutions and the pilot schools. Evaluating the achievement of the project objective and outputs should have two aspects: to what extent were quantitative and qualitative outputs reached as a result of the interventions? Furthermore, it is necessary to note that there were practically no sector studies available prior to this project and the Ministry of Education is a first time borrower, so building project management capacities was one important aspect of the project. The overall rating of the achievement of the project development objective depends on how these different aspects of the project are balanced and how they compensate each other as the education reform is moving forward in Azerbaijan. Several points need to be made here. First, the ERP was designed as a three-year intervention to strengthen the foundation for future lending to the sector. However, due to ineffective project management and the lack of understanding of the procedures, the MOE spent about a year selecting the consultant (to provide technical assistance to curriculum reform) who eventually started to work in October 2000. After this - 6 - recruitment and re-arranging of project management, the project implementation commenced. Second, being a LIL, this project needs to be looked at from the point of view of the main purposes of that particular intervention. The project design correctly focuses on two main areas of improving the quality of teaching and learning, namely curriculum and teacher development in selected grades and schools. The project also had a component on project management, and monitoring and evaluation of the experimentation in the pilot schools and institutions. The third component is therefore critical regarding the learning and innovation aspect of the entire project. In general, the ERP had a positive impact on the education sector development in Azerbaijan. During the visits to the pilot schools and institutions, it became obvious that teachers and pupils had gained a lot from in-service training and new curriculum materials that were provided under the project. Moreover, the general attitudes towards education reform were positive in two ways. First, there seemed to be a wide recognition of the need for changing education, not only in terms of structures and regulations but more fundamentally concerning what goes on in schools and classrooms. The awareness of the global trends in schooling, including active models of teaching and learning, alternative methods of assessment, school level improvement and school management, has also been consolidated in the Ministry of Education and its affiliated institutions. Second, the Ministry of Education has considerably better knowledge and skills today to steer and manage the ongoing sector reform after the learning period during the ERP. There seems to be a good understanding of the need for strategic thinking and monitoring of the implemented policies. The headmasters in the pilot schools recognized the importance of helping teachers in their schools to overcome the anxiety related to moving from traditional teaching to more interactive modes of instruction. Finally, there is a broader pool of experts now within the education system who are available for education reform work and advising schools how to cope with the existing expectations of renewal. Pilot approach to education reform is often criticized as a failing strategy to bring about sustainable improvement in education. Moreover, piloting sometimes has unexpected consequences within the system. The case of the ERP in Azerbaijan is no exception, and a remark needs to be made here since this was not among the issues under section Sustainability and Risks in the Project Appraisal Document (PAD). The development objective of this project generates a question: Are the pupils in pilot schools that use renewed curriculum, better facilities and where teachers have been retrained, learning better compared to pupils in other schools? The follow-up question is: What other consequences has participation in the pilot had to the participating schools? All the schools visited (six pilot schools) shared the same development: as the curriculum, facilities and teachers teaching capacities improved, the number of enrollments in these schools increased respectively. In some schools, as much as 30 percent (in one school from 1,800 to 2,500) due to new legislation that provides parents with free school choice. At the same time, the net enrollments have been declining during the past decade, which indicates a dramatic drop in number of pupils in other schools. In other words, the pilot schools are, for obvious reasons, attracting certain families from other schools' catchment areas. Together with increasing class sizes and school sizes, this has also had a positive impact on student performance and achievement based on teachers' experiences (not always objective evidence). However, it may be argued that the improved learning outcomes are due to biased intake which favors pupils from the better-off families. Overall, the conclusion of this report is that despite some shortcomings in terms of reaching the development objective, there was a satisfactory impact of the project in the education sector and its ongoing reform in Azerbaijan. Most of the quantitative targets (teachers and experts trained, schools rehabilitated, new syllabi designed) were achieved or exceeded. However, due to lack of reliable data, there is not enough evidence that the project objectives, as they are described in the PAD, were fully achieved. The project was able to produce new thinking as well as some new practices in the Ministry, pilot institutions and schools, but it partially failed to benefit from the initial idea of a LIL: to have recorded evidence of the advantages - 7 - or disadvantages of selected education improvement approaches to improving the quality of teaching and learning. As noted earlier, the concept of improved effectiveness of teaching and learning is complicated and this project provided only anecdotal evidence of improved effectiveness in schools. The key problem in judging the achievement of project objectives is that there is no data on which this judgement could be based. Indicatively, the Monitoring and Evaluation Study (2004) and Final Report on Curriculum Development, Preparation of Educational Materials and Student Assessment Consultancy (2003) carried out by an external consultant indicated that there was no significant difference in student achievement between the pilot schools and non-pilot schools after the interventions (although it should be noted that this was the first national assessment in Azerbaijan and was conducted more as a trial than as an actual assessment) nor improved parent and community participation and satisfaction in pilot schools. However, this project is a good example of the complex nature of designing education reforms - often the most important outcomes of these reforms are different than those initially expected. 4.2 Outputs by components: The main emphasis during the implementation of the ERP was on the curriculum reform and related teacher in-service training for teachers in pilot schools. The third project component (Project Management, and Monitoring and Evaluation) was only partially implemented regarding monitoring and evaluation, although this was regarded as a crucial element of the LIL. 1. Curriculum Reform (Appraisal estimate US$1.2 million; Actual US$1.32 million) - Unsatisfactory The intended implementation of this project component was jeopardized by the unsuccessful selection (the contract was signed but soon termonated due to unacceptable performance) of a consultant who was supposed to assist the MOE and provide technical assistance to the Working Groups and curriculum specialists in Azerbaijan. In other words, nothing significant happened under this component before 2001, except completed civil works in pilot institutions. The Ministry of Education tried hard to cope with the situation without external help and established Working Groups that drafted new syllabi for music and drawing, Azeri language for Grades 1-2, and Azeri language and literature, mathematics and informatics, history, English, music and drawing, and Russian language (in Russian schools) for Grades 5-6. The draft versions of these new syllabi were delivered to the pilot schools in August 2001. It is obvious that the quality of these pilot curricula was not what was expected when the project was designed. However, the MOE, together with the regional and local (rayon) education specialists, analyzed the experiences in the pilot schools and discussed them in the workshops and meetings during 2002. New curriculum (syllabi). A new consultant was hired to provide technical assistance to this component in January 2003. As a result, the consultant and its local counterparts prepared the concept of the framework curriculum for general education, supported the further development of Azeri language and mathematics curriculum for Grades 5 and 10, and prepared and conducted the small-scale national student assessment in eight pilot schools and twelve non-pilot schools. In addition, this consultant provided professional development to teachers, trainers and education authorities in the workshops and seminars. This component was supposed to experiment with innovations in curriculum content and curriculum development process and learn from those experiences in order to put the curriculum reform on a national scale in the proposed follow-up project. Due to unsuccessful performance of the initially selected consultant and weak consultant management, this component lagged behind in implementing the activities that would have enabled expected experimentation of curriculum innovations and delivery of intended outcomes. Most of the activities related to renewing curriculum took place during the last year of the project, and therefore the actual piloting aspect remained weak. Actually, this component of the project was not able to provide any significant lessons for the nation-wide reform of national curriculum, except for the conceptual - 8 - framework for curriculum development. Visits to pilot schools gave the impression, expressed also by some MOE experts, that teachers appreciated the new facilities, in-service training and pedagogic guidebooks but, at the same time, emphasized that the real curriculum change requires also new learning materials for pupils. Ideas of interactive learning methods discussed during in-service training sessions were far more often referred to by teachers than the actual new curriculum as primary outcomes of the project. Capacity building. Although this component failed to achieve its main objectives, it has had some significant positive impact on reform-related issues in Azerbaijan that deserve specific mention here. First, as a result of this component, the concept of 'curriculum' is beginning to be rooted in pedagogical and educational development discourse in Azerbaijan. It is well-known that during the Soviet era, curriculum was not a concept used in this context. Now, however, even in the pilot schools, the principals spoke about the need for 'curriculum development' and, in the MOE, the officials referred to 'curriculum reform' as one key means of transforming education to meet the needs of modern society. Second, through the emphasis on pupil-centered curriculum and respective pedagogics, the attitudes of teachers in pilot schools are changing in favor of active participation of their pupils instead of traditional presentation-recitation mode of classroom behavior. Although there is no evidence yet in Azerbaijan that what was piloted during the project would make a difference in terms of effectiveness of teaching and learning, teachers who were engaged in the new curriculum process tend to believe that there will be a change in outcomes of schooling. Third, this component convinced the MOE and other reformers of education that curriculum reform alone, regardless of how well it is designed, would not do much unless pupils have new learning materials and schools are equipped with appropriate facilities to promote intended new pedagogical patterns of thinking and acting. These and other lessons from the project are important indeed although they were not recorded as the intended outcomes of this component. 2. Teacher Training and Pilot Schools (Appraisal estimate US$3.2 million; Actual US$3.2 million) - Satisfactory In this component, teacher training refers to in-service training of teachers. Teacher Training Institutes (TTI) that were the pilot institutes (total of seven institutes) offer in-service training to teachers. The goal of this component was to pilot new teacher education content (including new teaching methods) and delivery mechanisms in order to improve the effectiveness of teaching and learning in pilot schools. As in Component 1, this component also suffered significantly due to delayed technical assistance that became available only in February 2003, when the last months of the planned project implementation were at hand. For the purposes of piloting new teacher training ideas, twenty pilot schools and seven TTIs were selected from five regions. Logically, this second component is interconnected to the first. The content of teacher training was therefore related to new curriculum, student assessment and especially to the methods of teaching that were reflected by the renewed syllabi and goals of teaching. Capacity building. There were two waves of training of trainers and teachers on active and interactive teaching methods. The first wave in 2000-2002 was mainly arranged internally without support from consultants. Through the workshops and seminars, 731 teachers from pilot schools (Grades 1-6) and 110 methodologists from the TTIs were trained. It was assumed that some of the trained teachers and teacher trainers would later be engaged in training other teachers. Based on the estimates done by the MOE, the number of willing and able trainers from this population is 20-25 percent. The second wave occurred in May - September 2003, when an external consultant was assigned to strengthen the training provided during the first wave. This trainers training focused on modern teaching and learning methods and reached 128 persons from pilot regions. As a result, Azerbaijan has a pool of teacher trainers with awareness, knowledge and initial skills to work with other teachers in providing them with training on modern teaching - 9 - and learning methods. The impact of the traning provided was clearly visible in pilot schools and TTIs. The concept of active and interactive teaching and learning was spelled out during all visits. Discussions with pupils and students were convincing that these ideas were also brought to the classrooms. Interestingly also, students who were much older than the target population in this project were able to describe the features and their personal experiences with new teaching and learning methods. However, it is possible that the expressed enthusiasm regarding the positive impact of these modern teaching methods will remain rather limited because there was no evidence that the new pedagogical practices were presented in any wider context or conceptual framework. Based on the contemporary conceptions of effective in-service training of teachers, it is easy to argue that what was implemented within this component does not promise any sustainable or more general improvement in teacher effectiveness or student achievement since what has been offered consists of rather conventional solutions towards quick fixes in teachers' teaching skills rather than permanent change of their beliefs and behaviors. Institutional development. One positive impact of this component was the realization of more demand-driven provision of in-service training to teachers. Trainers in the TTIs were trained on how to conduct needs assessment analyses and how to develop and redesign training programs based on the actual needs of teachers. Most of the TTIs changed their way of thinking as well as their actual training curricula as a result of this component. One intention was to establish twinning relations between pilot schools and the TTIs. Since the creation of Teacher Resource Centres (TRC) within the TTIs was not properly actualized, truly operational partnerships between pilot schools and TTIs never really came to exist. There was an opinion that these TRCs already exist, which is true in theory, but according to teachers, they are not useful, as they have outdated literature and limited resources, for alternative modes of teaching. One of the most visible and widely recognized inputs was the strengthening of educational facilities and equipment in pilot schools and pilot Teacher Training Institutes. 3. Project Management, and Monitoring and Evaluation (Appraisal estimate US$0.6 million; Actual US$0.76 million) - Satisfactory The objective of this component was to set up a Project Implementation Unit (PIU), hire necessary staff for that Unit, provide adequate training to the staff, and coordinate the overall monitoring and evaluation of the project implementation. In this respect, the component needs to be analyzed in two parts: project management (Appraisal estimate US$0.4 million) and monitoring and evaluation (Appraisal estimate US$0.2 million). Project management. The overall output of project management was satisfactory. It is necessary to note that at the beginning of the project (1999-2001), very little progress was made and output was highly unsatisfactory. Only after reorganization of the PIU management (in January 2001), the implementation of the project activities proceeded as planned. The project was then managed by a strong PIU director and supported by a committed Minister and his senior staff that brought the project back to track after great frustration in the early phase of the project. Since then, there has been a close and dynamic relationship between the PIU and the MOE, and also mutual trust that has enabled flexibility during the course of project implementation. The PIU director has been very successful in hiring consultants for the technical assistance needed for the first two components of the project. Moreover, he managed to recruit local experts who believed in the project and had the right knowledge and skills to carry out their assignments effectively. Most of the achievements in this project took place after the beginning of 2002. - 10 - Monitoring and evaluation. The overall output of this sub-component was unsatisfactory. First of all, monitoring and evaluation is a principal tool in a LIL (or any learning-oriented intervention) in order to understand the realities of reforming education under selected piloting (experimental) settings. In this project, monitoring and evaluation focused solely on quantitative performance as determined by the project performance indicators; in other words, it was limited to routine monitoring and evaluation as part of typical project management. This implies that the learning aspect of the LIL was not properly utilized. Second, no baseline data was collected to provide even tentative indications of the effects of selected innovations to teaching and learning. In short, it is impossible to say anything ex post facto on how teachers' thinking and skills have changed in terms of new pedagogical approaches introduced during the project, or whether pupils are actually learning more or better in pilot schools after active/interactive teaching methods have been used by their teachers. Third, technical assistance to improve monitoring and evaluation within the MOE was made available only at the very end of the project implementation. The idea of enhancing respective knowledge and skills in the MOE in tandem with training and on-the-job learning was therefore mostly missing. A total of six sector studies were conducted and completed: Textbook Publishing and Distribution (2000), Education Cost and Financing Review (2002), Parent and Community Mobilization (2002), Teacher Evaluation and Professional Development (2002), Azerbaijan Education Reform Project - Monitoring and Evaluation (2003) and Monitoring and Evaluation Study (2004). Azerbaijan did not participate in PISA 2003 due to financial reasons. The meaning and importance of the monitoring and evaluation sub-component in a project like ERP should have been much more carefully emphasized and operationalized from the very beginning. Project supervision and technical assistance should have played key roles in designing and implementing necessary and relevant monitoring and evaluation activities. This did not happen in the ERP. According to the MOE, what was supposed to be done under this component was realized too late when basically nothing could be done. Financial Management. The overall project financial management was satisfactory. One positive aspect was the PIUs attempt to comply with both financial and audit covenants by maintaining adequate financial management systems - including staffing, accounting and financial reporting systems, internal controls and auditing arrangements. Quarterly financial monitoring reports and annual audit reports were submitted to the Bank on time, and auditors recommendations on strengthening internal controls that were identified during the course of conducting the audits adhered to. Although the Project experienced Credit disbursements delays during the first year of implementation due to procurement process, overall disbursement progress was satisfactory. Total disbursements of IDA Credit 3220-AZ was US$4,825,596.72 and PHRD Grant TF 25581 was US$288,909.44. 4.3 Net Present Value/Economic rate of return: Analysis of the pre-project situation in the education sector in Azerbaijan and comparing it to the post-project implementation shows that the relatively high cost of experimenting curriculum and teacher training innovations, as far as eventual results are concerned, will probably pay off in the long term as the follow-up project will be fully under implementation. 4.4 Financial rate of return: No financial rate of return was calculated for the Education Reform Project. 4.5 Institutional development impact: The institutional development impact of the project has been moderate in the project environment and within the education community. Pilot institutions (schools and TTIs) have gained not only from participating in various in-service training sessions but also from their internal discussions as part of a - 11 - small network of changing schools. Wide-spread recognition of a need to change the ways in which schools function was common in all pilot institutions. This attitude change is a crucial element of institutional transformation because it constitutes the core of what is called the culture of schools. Moreover, together with these new patterns of thinking about teaching and learning, there is also a much better understanding today in the pilot schools and TTIs of what needs to be done to get there. Similarly, to TTIs in their shift to mapping teachers' training needs and expectations, teachers are also looking more at their pupils in terms of what they think and expect from school. This cultural change in pilot institutions is perhaps the most important single development impact of the project in its environment. Based on the technical advice delivered through the PHRD grant during the project preparation, the Government reorganized the structure of some key education institutions. On June 2000, a Presidential Decree merged the Pedagogical Research Center and the Methodological Center under the Institute on Educational Problems (IEP). The Government approved the institutional structure of the IEP in February 2001, and the hiring of the intended qualified staff was rather slow, which affected the expected operations of the IEP in the project. Given that the IEP is one of the pilot institutions supported under the project, this institution was also to receive development and capacity building support through the project. Although the actual establishment of the IEP is an asset in itself to the education system, the institutional capacity of the IEP to contribute to the progress of education reform in Azerbaijan remains weak. The IEP is responsible for curriculum development, student assessment, development and review of textbooks and learning materials, and educational research. The project also contributed positively to conceptualizing, designing and preparing the follow-up project, Education Sector Development Project (ESDP). More comprehensive basis for managing education project and education reforms in general was established. The MOE now has better capacity to truly lead the reform and be more independent in implementing the ESDP, compared to the situation in the beginning of the ERP. The improved capacity of the MOE to cooperate with the Bank and other international agencies should also be emphasized. The project also provided equipment and limited training to the MOE's departments of Economics and General Education to collect and analyze the key performance indicators for the project. This project has clearly had a positive impact in the MOE. The Minister and his senior staff are convinced now that developing the education system to perform better in terms of quality and effectiveness benefits from active international cooperation. Good examples and lessons learned in other education systems find much softer soil today in Azerbaijan compared to the pre-ERP era. Through study visits, international training programs, visiting consultants and the Bank supervision, the MOE has received continuous inputs of global ideas that may turn out to be useful in the ongoing education sector reform. This project has also catalyzed thinking on the overall goals of education as a result of good experiences with new teaching methods, increased focus on educational assessment and changes in school curricula. Teachers and also pupils and parents are beginning to have a stronger voice regarding how schools should be developed. The MOE is also clearly recognizing the necessity of investing in a system that would help in evaluating education outcomes as crucial element of the overall monitoring of the education system. It is, however, important to realize that this project included only twenty schools and seven TTIs. The big issue is how these positive results of the ERP can be disseminated to all schools in every part of Azerbaijan. Although some pilot schools have provided training to their neighboring schools, it is fair to say that the development impact in the level of schools and teachers has been minimal. Furthermore, this project does not provide advice on whether what has worked in pilot schools will work in other schools as well. As a conclusion, the LIL has helped to introduce the education reforms needed in Azerbaijan and showed that some change is possible with enthusiastic teachers and school directors. It has also proven that - 12 - if teachers receive appropriate help and resources, they, together with their pupils, will react positively and actually find these new ideas beneficial to their work. 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implementing agency: In general, the Government (Ministry of Education) showed commitment to the project purpose and objectives. The project became effective on time. Short preparation time, lack of sector work and inexperienced borrower reduced the active participation of the MOE in project preparation. Project implementation suffered from inexperienced project management in the beginning (three PIU Directors were replaced) and as a consequence, unsuccessful selection of a consultant who was not able to deliver what was expected. This caused considerable delay in implementation. The first year of the project focused on the selection of the consultants and procurement of goods and works. In reality, very little was done in curriculum development or teacher training by mid-2001. According to the available records, the project implementation actually started in the beginning of 2002 (until then the Project Status Report [PSR] rating was unsatisfactory). It may be argued that the Government should have selected the initial consultant more carefully, being in charge of selecting the consultant, should have been more careful in finding out the real capacity of the winning consultant. However, it is extremely difficult to predict in advance whether a competitively selected consultant will be able to deliver or not, especially for first time borrowers from the Bank. On the other hand, the lenthy selection process (in this case, it took about one year to see the selected consultant for the first time!) was partly due to the current procurement procedures set by the Bank. 5.2 Factors generally subject to government control: This project was implemented under a relatively strict centralized planning and implementation scheme although the local education authorities and individual schools had new areas of responsibilities due to the first waves of decentralization in the public sector. The positive factor that has had significant impact on project implementation and outcomes described earlier is the continuous and strong support by the Minister and his senior colleagues to the project. The MOE has also used media effectively in communicating to the people about the education reform and the project activities. The main negative factors subject to Government control affecting implementation were the lack of an action plan for the Education Reform Program, delayed institutional reform regarding the Institute on Educational Problems (IEP), and weak coherence between the project activities and the overall education sector reform. Lack of competent education development staff often led to ad hoc solutions and uncoordinated reform efforts, especially in curriculum development and teacher training that often affected the activities implemented under the ERP. The merger of the two previous institutes to IEP was also done without a strategic or operational plan, institutional analysis or management review that would have guided the organizational reform (although such review and a seminar were provided through the Dutch Trust Fund by external consultants). Since the organizational structure, staffing and operational functions remained undefined for almost a year, very little institutional development or support to the project implementation occurred. 5.3 Factors generally subject to implementing agency control: The ERP was the first IDA-financed investment credit in the education sector in Azerbaijan and, hence, was a valuable learning experience for the Government. After relatively smooth project preparation, both the Bank and the Government faced considerable challenges that accumulated during the first half (September 1999 - December 2001) of the project implementation. In the PAD, the Bank team forecasted some of the probable risks: lack of implementation capacity, weak practices in project management, and the project being too complex for a first-time borrower. Most of the constraints were related to the implementation agency and its affiliates (pilot TTIs). - 13 - The major problems in this project were related to delays in implementation. Analyzing the documentation during the course of the implementation, several factors can be identified. First, project management was the weak point from the very beginning of the project. After sixteen months of project implementation, the PIU Director was replaced and project management begun to gradually improve. Second, procurement capacity also remained weak until the latter part of project implementation and this had a negative effect on overall implementation and outcomes. The process of consultant selection and procurement of teaching and learning materials to pilot schools and TTIs was slow, leading to delays in receiving technical assistance and shortages in adequate materials in schools. A misprocurement issue that occurred in 2000-2001 was one indication of the complexity and related consequences of procurement, although it did not directly affect implementation. Third, lack of coordination of implementation activities between the PIU and the other implementing agencies (IEP and the TTIs) eventually led to the PIU Director becoming the coordinator general of practically all project activities. During most of the implementation period, the IEP and the seven TTIs did not have an agreed clear implementation plan that would have encouraged these institutions to take the lead and coordinate the implementation of the Teacher Training component. The project implementation was extended twice (until 31 March 2004) to be able to complete all planned activities. 5.4 Costs and financing: The Project was implemented between September 9, 1999 and March 31, 2004. It was approved for an amount of US$5.5 million of which US$5.0 million was provided by IDA and US$500,000 was provided by the Government. The planned project budget was very accurate, with financing totalling US$5.28 million. The counterpart funding from the Government was US$448,000 and credit disbursement was US$4.8 million. Table 1. The Education Reform Project budget and actual disbursements Component Allocation Percentage of Actual Percentage of (US$m) Total Disbursements Allocation 1. Curriculum reform 1.20 21.8 1.32 110 2. Teacher training 3.20 58.2 3.20 100 3. Monitoring and evaluation 0.6 10.9 0.76 126.6 Contingencies 0.5 9.1 TOTAL 5.5 100 5.28 96.0 The total cost of project preparation, supervision and completion was costly, as Annex 4 indicates. Disbursements were timely while the project was in full operation. All civil works were completed during the first year and a half of the project. There were delays in textbook procurement and hiring of the external consultant after first unsuccessful procurement. 6. Sustainability 6.1 Rationale for sustainability rating: The key purpose of the pilot approach in education is to test a selected approach or various models that would provide potential means to replicate the intended reforms on a national scale. If sustainability in this section means the probability of maintaining the achievements generated or expected to be generated in relation to its objective over the life of the project within the Government budget, the intended outcomes of the ERP are likely to be sustained. However, the outcomes achieved in this project required approximately US$100.0 investment per unit (school or TTI) in rehabilitation, improved facilities and equipment, and - 14 - training of teachers in pilot schools was expensive during the ERP. Given these costs, it would be difficult to multiply within the forecasted government budgets the reform efforts of the ERP in all or even most of the schools in Azerbaijan which has 4,450 schools and 160,000 teachers within the system. The ERP was able to train and prepare a pool of local trainers in the TTIs and in the pilot schools, many of whom will be able to work as trainers during the ESDP. This will bring the cost of teacher training down but is also a factor that will affect the effectiveness of this training. Unfortunately, due to the limited scope of project design, the ERP was not able to produce any further information on various options, such as low-cost/high-impact teacher development and school improvement models. For political and equity reasons, the MOE has to somehow include all schools in the ESDP as beneficiaries. There is a risk, therefore, that teachers and schools will receive what the pilot schools did in the ERP, but in much smaller and less-intensive scale. There is a lot of evidence in other projects and in research on school development about the ineffectiveness of this type of effort. Finally, the initial review and evaluation of the impact of the ERP on effectiveness of teaching and learning does not provide any evidence that the quality of pupils' learning in pilot schools has improved. According to the external consultant's study (2003) that was carried out under the ERP, the first external student assessment in Grade 5 mathematics and Azeri language achievement did not find significant differences between the pilot schools and non-pilot schools. There may be various reasons why the ERP interventions did not have visible impact on learning, although many teachers and their pupils warmly welcomed the new pedagogical ideas delivered through the ERP. Hence, the evidence from the ERP experience does not support repeating the same efforts or models of teacher training and curriculum to improve the quality of teaching and learning. Despite generally low expectations of overall sustainability of the selected reform approaches, there are several themes that will certainly be sustained by the Government, donors, or community level support. The commitment to bring about the changes in curriculum, including both content and models of teaching and learning, will be continued one way or the other. The follow-up ESDP will also provide significant value-added to this process. The recognition of the importance of better monitoring and evaluation of the education system performance will also be sustained and probably will receive even more attention from the Government as well as from the stakeholders in the near future. 6.2 Transition arrangement to regular operations: The Education Sector Development Project (ESDP) was designed towards the end of the ERP implementation and was actually launched during the last months of ERP effectiveness. Some of the successful aspects and good experiences of the ERP have been scaled up or modified to match with general dissemination (teaching methods training, student assessment). Some of the issues that have been raised in this Implementation Completion Report (ICR) were not available when the ESDP was prepared. However, there will be occasions when these issues can be discussed during the supervision of the ESDP, especially as the PIU will continue to be the key coordinating unit for the implementation of the EDSP. This also means that the project management capacities developed during the ERP are more likely to sustain under the ESDP. 7. Bank and Borrower Performance Bank 7.1 Lending: The Bank's overall lending performance was unsatisfactory. The identification of the ERP was done in coherence with the MOE's education sector reform priorities that were reflected in the Education Sector Strategy that was drafted during the project preparation. The Government saw the ERP an integral element - 15 - in the overall implementation strategy of the education sector development plan. The project supported the CAS (1996) objective of the "need for social policies that preserve Azerbaijan's human capital and protect those vulnerable to changes that are sweeping the economy." In particular, the CAS referred to the "need to develop a more rationalized system for delivery of basic social services." The ERP also supported the next CAS (1999) and further development of the education system by providing advice on curriculum reforms and teacher training programs. There was sufficient stakeholder participation during the project preparation and the Government demonstrated a high commitment to implementing its reform agenda. The project team took into account the relevant issues in project design at the time of preparation. Based on the situation analysis at the time of identification, the LIL was considered an appropriate lending instrument. The project team envisaged that a LIL would provide the framework for strengthening the policy and institutional framework and experimenting with new models within the sector. Afterwards, as the implementation and related outcomes of the project are analyzed, one wonders if the main rationale of a LIL was properly utilized. The learning aspect of the ERP was rather unfocused, touching mainly on capacity building within the PIU and in the pilot schools and TTIs. In terms of innovations, the ERP did not experiment with any really new teacher training arrangements or curriculum approaches. Therefore, it appears that the intended LIL was implemented as a small Specific Investment Loan (SIL), with the specific goal to change some core areas in the education system rather than to learn what would work and how the proven impact could be put onto a national scale to benefit all schools. Although this LIL had several positive impacts (of which some were among those unexpected), it is not clear to what extent the MOE understood the primary logic and purpose of the selected lending instrument. Another remark related to the project design concerns the level of complexity of the designed project. Education reforms are challenges to any government and, for the first-time borrower, partnership with the Bank in education development is a double-challenge. The project team quite correctly reflected in the PAD risk analysis that the complexity of the project may become an obstacle for the borrower. Therefore, a roadmap for education reform in general and curriculum development, in particular, could have been made more specific. Most complications occurred in procurement, coordination of teacher training and curriculum development activities, and establishing ownership-driven relationship with the consultants. None of these issues were appropriately predicted during the project design and preventive measures were therefore not sufficiently mobilized. It may be argued that the ERP design should have had stronger emphasis on management and leadership in general from the beginning of the project in order to create self-correction capacity within the PIU and the MOE, to indicate where the potential problems might be encountered. 7.2 Supervision: Supervision of the ERP was satisfactory. The project was supervised throughout the project implementation by means of semi-annual visits and more frequent assistance through telephone and electronic mail. The main focus of the supervision was on the progress in implementing the project components, identification of possible problems and general project management issues. The project implementation progress was adequately and timely reported and the performance ratings given in the PSRs were, for the most part, realistic. Supervision reporting was mainly impact-oriented and dealt with those implementation issues that were likely to affect implementation and outcomes. Supervision mission typically lasted for 5 to 8 working days in the field which in normal circumstances is an appropriate duration for an effective supervision. The Bank managers (two Sector Managers and Sector Director) were involved in three supervision missions during the critical phases of the implementation, and this probably improved the positive momentum in implementation. The field office in Baku played a crucial role during the supervision and also provided an important sustainability factor to the overall supervision. It appears that the Task Team Leader of the ERP should have been supported more in the beginning of - 16 - implementation by the Bank. All in all, cooperation between the Bank team, the Borrower, the PIU, other donors and the consultants was mainly constructive and successful. As was described in earlier chapters, the project suffered from severe implementation delays in the beginning due to lack of experience in Bank operations, insufficient project management capacity in the PIU, poor overall implementation coordination between the implementing agencies, and unsuccessful selection of good consultant in the early phase of the project. Moreover, there was a lengthy misprocurement case during the second year of implementation. These implementation problems were identified and addressed properly and pro-actively. However, since the supervision teams visited Azerbaijan only semi-annually, it was very difficult to perform any preventive or early corrective measures in order to help the PIU and the MOE to react properly to these problems. The PAD and the first supervision memoranda quite correctly underline that close cooperation, advice and help in the early stages of implementation of this project would be crucial for the project success. It may be argued that this principle of more intensive supervision was not fully practiced. Although some of the problems faced by the Borrower may have been inevitable, for example, selecting another consultant after failing performance of the first one, could have been done faster with more direct support from the Bank. The purpose and rationale of supervision seemed to be unclear to the Borrower until the very end of the project. Supervision was seen as auditing and control conducted by the Bank rather than technical advice and help to improve overall project-related performance, especially implementation. As a consequence, the Borrower was not sure whether they could influence the supervision process, for example, by requesting certain kinds of assistance or specific experts to join the supervision missions. Review of the supervision documents indicates that the Bank could have paid more attention to appropriate knowledge and skills mix on the Bank team (including consultants). With appropriate skills mix, the Bank team could have provided more relevant and useful advice to the PIU, especially with regard to the occasional major issues that arose in implementation. 7.3 Overall Bank performance: Satisfactory. The Bank's performance in general was professional and client-oriented. There were some serious weaknesses in the project design (low learning orientation and lack of focus on truly new solutions to teacher training and curriculum development) as well as project supervision (lack of support to the implementing agency in the beginning, lack of focus on education change, and lack of specific technical staff in supervision teams), but close partnership with the MOE and the PIU, and accurate processing of project issues keep the overall Bank performance rating satisfactory. Borrower 7.4 Preparation: Satisfactory. The ERP was the first education project with the Bank in Azerbaijan. The MOE had independently drafted the Education Reform Program and expected that this project would help to achieve some of the expected outcomes of the sector reform. The MOE education reform strategy has a strong emphasis on curriculum reform and related teacher in-service training as key priorities. Although the project design was mainly prepared and drafted by the Bank team, the MOE provided its support to the plan. Relatively low planning and research capacity in the Ministry of Education and in the education sector in general prevented more active involvement of the MOE in the project preparation. 7.5 Government implementation performance: Satisfactory. The Government described its education sector policy in the Education Reform Program. It identified the priorities that were included in the project design. The Government, through the Minister of Education, remained fully committed to the project throughout the implementation. The MOE was - 17 - pro-active in identifying problems during implementation, such as inability of the first PIU Director to fulfill his responsibilities as expected and low level of consultant performance during the beginning of the project. The MOE was very successful and quick in finding a competent and experienced new Director for the PIU. The Government also demonstrated strict and open professional behavior as the unfortunate misprocurement issue was processed and solved. The counterpart funds were disbursed as planned. Despite the overall satisfactory performance of the Government, it should be noted that the MOE did not fully utilize the new capacities gained during the Education Fellows placement programs and studies abroad. Many of these individuals were not immediately hired to benefit the project management and implementation. 7.6 Implementing Agency: Satisfactory (although initially Unsatisfactory). The implementing agency's performance from the management effectiveness point of view was poor until the last two years of the project. During the first two years of the project, practically nothing related to project objectives was achieved. Implementation of the Curriculum Reform and Teacher Training components remained uncoordinated and ineffective until the new PIU Director took the responsibility for all coordination in the project. Procurement of textbooks, teaching materials and other goods to pilot schools and TTIs was delayed. Monitoring and evaluation of the piloting process remained in at a low level throughout the project. Until the end of the project, the line units (Department of General Education, IEP, pilot TTIs) did not take full ownership, assuming that the PIU would be in charge of the project, including curriculum and teacher training activities. On the other hand, much credit should be given to the leadership role of the Minister, the Deputy Minister and their colleagues, including the new PIU Director, as he entered the project during an extremely complicated period. The project had been practically at a standstill until then (except for civil works and a few ad hoc teacher trainings) and had experienced a serious misprocurement case. Under the new management, the project was brought back to full operation, good relations with the beneficiaries were established, excellent communication with the Minister and his senior staff existed, and all project management related procedures were conducted and completed in a timely and satisfactory manner. The PIU Director and the MOE also performed well in selecting consultants and making sure that all their deliverables were disseminated effectively. During the second half of the ERP implementation, the implementing agency's (PIU) performance was satisfactory. 7.7 Overall Borrower performance: Overall Borrower performance was satisfactory during the course of project preparation and implementation. The Borrower was doing the best it could in often new and strange circumstances, and was willing to improve and learn continuously. Difficulties in project implementation were overcome constructively, openly and professionally. Throughout the project, the Borrower demonstrated strong commitment to the project and to education sector development. The MOE has shown understanding and patience with the requirements of the Bank and flexibility in adapting to new practices and external demands. 8. Lessons Learned General issues Learning and Innovation Loan. Although the selection of a LIL as a lending instrument for this project was well-justified and understandable, after completion of the project, it is not clear to what extent this project fulfilled the principles of a LIL (other than being prepared quickly). From the learning point of view, most of what occurred was the raised individual awareness and knowledge related to education - 18 - reform, especially with respect to curriculum and teacher development. Far less learning took place at the institutional levels or at the education system level. The Bank needs to be much more precise and disciplined in using LILs, as they have a great potential to promote learning and understanding among first-time borrowers about the partnership procedures with international organizations, and the nature and dynamics of sectoral reforms, and they can provide political leverage and acceptance of new ideas and necessity to change in general. It is important to set the development objective and short term goals of similar LILs realistically and focus them on learning and improving borrowers' personal and institutional capacities to cope with larger sector programs. This project also shows how shortened preparation time often leads to inappropriate identification of key issues and priorities, and obstacles in implementation. This implies that the shorter the preparation time, the greater the need for flexibility in implementation. Pilot approach and design. The most beneficial aspect of the pilot approach in education development comes from comparing various models to solve a given problem. This requires that the experiments within the pilot be carefully designed. In this project, basically, only one approach was piloted in selected schools and Teacher Training Institutes. Therefore, at the end of the project, very little could be said about how good this chosen method actually is as a general approach. Piloting potential models of teacher professional development or curriculum in schools would also require a systematic and continuous monitoring and evaluation system, that in this project was not successful. What is left after rather expensive piloting are mainly opinions of those involved in the exercise, not objective data on how the selected approach worked and what impact it had on teaching and learning in pilot schools. Hence, in similar pilot-based LILs, much more attention needs to be paid to designing of the pilot, mobilizing technical assistance during the first months of implementation and an adequate monitoring system from the very beginning of the pilot (including baseline data from pilot institutions). Role of supervision. First-time borrowers always need more and different supervision than experienced partners. This was well-recognized in the project documents and early supervision memoranda. However, despite sufficient time allocated to supervision by the Bank team, this project shows that with more client-oriented and needs-based supervision, some of the delays and other deficits in implementation could have been either avoided or worked out faster. The lesson is that, in LILs, the entire concept of supervision should be reconsidered. This means, first and foremost, emphasizing the planning for supervision visits in collaboration with the Borrower, indicating the particular areas where technical help is needed, including strong technical experts to provide sector-specific help to the client, and providing the Borrower and the key stakeholders with regular opportunities to discuss together the issues related to the project and the reform in general (more workshops and learning-events instead of traditional meetings). Finally, it is important to clarify the purpose of supervision to the Borrower very carefully and to make sure that all supervision has needs-based orientation throughout the implementation. Attention to educational change. The MOE saw this project as a potential engine for its own education sector reform strategy. Therefore, one important aspect of this project was to help the MOE, education authorities, principals and teachers to understand educational change. In other words, central expectation related to this project was to learn about how to make the intended reforms alive in schools and classrooms. The aspect of understanding and learning to cope with continuous educational change was only implicitly included in the ERP. Any education reform effort should provide opportunities for those who are in charge of implementing the changes to learn more about the experience in education reform elsewhere. Educational research community including teacher educators are playing a key role in doing this. - 19 - Project specific issues Need for continuous support to schools. Teachers and pupils indicated that this project was useful for them. Clearly, several new thoughts and practices were rooted in schools. However, in-service training that was offered to teachers was limited to one or a few workshops without systematic follow-up. Although the concept of 'clinical supervision' was experimented (but not analyzed in terms of its suitability in Azeri schools and its impact on teaching and learning), most teachers had to find their own ways to continue their professional development alone. Listening to teachers indicates that in addition to formal training, they also need follow-up support in order to maintain the momentum to implement new ideas in their work, and what is even more important, to continue learning. This project shows that it is possible to reach the level of high teacher motivation to change rather quickly but maintaining that spirit of renewal requires systematic social and resource support and help to teachers. Together with innovative and demand-driven teacher in-service methods, it is necessary to design more sustainable professional development system that will help to cultivate the knowledge and skills learned in these trainings. Curriculum versus textbooks. Curriculum reform was one of the main goals of this project. For many teachers, the actual curriculum is the textbook used by pupils. Conversion of the new syllabi and curriculum framework prepared in this project to teaching and learning in classrooms was slower than expected, mostly due to lack of teaching and learning materials that would have fully reflected the renewed curriculum. Therefore, the lesson learned, again, in this project is that development of textbooks and related teaching aids needs to go hand-in-hand with the new curriculum. One possible practical solution is to use the selected pilot schools as a testing site for the new curriculum and develop new textbooks simultaneously, using some of the teachers from these schools as authors. Unpredictable impacts. The project also had some unpredictable impacts. The general demographic trend in Azerbaijan includes declining birth rate that has influenced the number of pupils enrolling in schools. In other words, schools are having fewer pupils every September than before. Generally, this means declining school sizes. However, most of the pilot schools have experienced an opposite development. Due to the legislation, parents have a right to choose the school for their children, and in the rayons where there is a pilot school, many parents have chosen these schools, for obvious reasons. Some of the pilot schools are experiencing a more than 30 percent increase in enrollment compared to the situation five years ago. This naturally has some negative impacts to the life in school in general. Argued improvement of pupil achievement may be because of the biased additional intake in pilot schools, together with or instead of more effective methods of instruction. Pilot-approach can, hence, have both positive and negative impacts on schools, teachers and pupils. Conceptions of learning. This project had a strong focus on learning through curriculum and teacher training. Moreover, this was a Learning and Innovation Loan. Since learning seems to be the key concept of this and similar operations, it would have been beneficial to pay more attention to better understanding of this concept. It is certain that learning has a different connotation in Azerbaijan than it has among contemporary school development experts or at the Bank. These differing conceptions of learning became very dominant when the lessons learned were discussed with the Borrower and the stakeholders in the end of the Project. The conclusion is that understanding how pupils learn and what learning means in general could become an integral theme in education sector reform programs in general and in LILs in particular. Within these expanded conceptions of learning such complex issues as educational change (as one form of learning) and monitoring and evaluation could be seen in a new light. - 20 - 9. Partner Comments (a) Borrower/implementing agency: Overall, the Borrower experienced the project execution satisfactory. Although there were constraints during implementation of the planned project activities, most of them were implemented during the project duration. Details of Borrower's views of the project are attached in Additional Annex 8. (b) Cofinanciers: Not applicable. (c) Other partners (NGOs/private sector): Not applicable. 10. Additional Information - 21 - Annex 1. Key Performance Indicators/Log Frame Matrix Outcome / Impact Indicators: 1 Indicator/Matrix Projected in last PSR Actual/Latest Estimate (i) Improved teacher effectiveness and No baseline survey. The results will be based The definition and respective measuring of satisfaction in pilot schools and TTIs. on the findings of a monitoring and evaluation improved teachers' effectiveness is not easy. study under the Project. The study was However, 11 out of 13 headmasters underline completed in March 2004. changes in teachers' methodical approach to instruction and higher variability of teaching methods, more orientation towards the students, more active and independent, it indicates that a certain basis for improved teachers' effectiveness was set. Most of 45 teachers who were asked whether they changed their teaching behavior said "yes" explicitly by illustrating their behavior changes. They feel more responsibility for the teaching-learning process, they feel influenced by their students' behavior. The first external assessment of student achievement in Azeri language and Mathematics (Grade 5) conducted by the foreign consultant showed no significat differences bewteen pilot and non-pilot schools. (ii) Improved student learning outcomes in No baseline data. Pilot national student assessment on Mother pilot schools. tongue and Mathematics for Grade 5 was carried out as part of the LIL. No baseline data on student achievement was available. However, the average scores of pilot and non-pilot students were compared in order to find out whether there was already an effect of being thought with the new curriculum materials. In both subjects, no significant differences were found between the average scores of pilot school and non-pilot school students. Only evidence of improved student learning outcomes are anecdotal and based on non-formal visit to schools and conversations with teachers and school principals. (iii) Improved teacher training and curriculum No baseline survey. The assessment is Pool of trainers and curriculum developers development capacity in the MOE and pilot based on the findings of a Monitoring and exist. There were 110 methodologists of institutions. Evaluation Study (2004). Teacher In-Service Institutions trained either in new teaching methods or in using the new curriculum materials, and 61 methodologists of in-service training institutions were trained in modern teaching methods and training skills. A third group of 36 methodologists of in-service training institutions and specialists of the Institute of Educational Problems were specially trained and qualified in developing curricula for teachers' in-service training. 61 people from the MOE and pilot institutions improved their teacher training and curriculum development capacities in different international trainings and fellowships organised under the LIL. Capacities have improved mostly at personal levels, less so at institutional levels. Institutions were slow in employing internationally trained experts. - 22 - (iv) Improved project management and No baseline survey. The results are based on No evidence to show specifically which implementation capacity in the MOE. the findings of the Monitoring and Evaluation capacities have been developed and how the (2004). newly created knowledge and skills are distributed over various institutions.During the project 33 staff of the MOE/PIU participated in international training on different aspects of management of World Bank financed projects. (v) Improved parent and the community No baseline survey. The results are based on It is difficult to give reliable data regarding participation and satisfaction in pilot schools. the findings of the Monitoring and Evaluation improved parents and community (2004). participation in pilot schools. Many parents were reserved in the beginning of the project, when the new teaching methods were introduced. Most of the headmasters reported changes in parents' attitudes by the end of the project. However, there are still parents (about 10 to 20 percent) who reject the pilot program. Of course, most of the parents like and highly appreciate the new equipment for the pilot schools. Output Indicators: 1 Indicator/Matrix Projected in last PSR Actual/Latest Estimate 1. Number of curriculum developers 96 239 participating in study tours and workshops, and training. 2. Development and availablity of new syllabi, 10 syllabi for Grades 1-2 and 5-6, 12 11 syllabi for Grades 1-2 and 5-6; textboks and teacher guides. textbooks and 7 teacher guides 6 textbooks for Grades 5-6; 6 workbooks for Grades 5-6; 3 teacher guides 3. Number of trainers of teachers in pilot 120 180 TEIs trained through workshops, seminars, and study tours. 4. Number of teachers in selected grades of 1,500 1212 pilot schools trained in new curricula and teaching methods. 5. Number of teacher resource centers 5 pilot Teacher Training Institutes 5 pilot Teacher Training Institutes established in pilot TEIs. 6. Number of classes in pilot schools with 465 728 new curricula/teaching methods being implemented. 7. Number of schools and institutes 20 pilot schools, 5 Teacher Training 20 pilot schools, 5 Teacher Training rehabilitated and refurbished and provided Institutes, 2 curriculum development Institutes, 2 curriculum development with supplies. institutes institutes 1End of project - 23 - Annex 2. Project Costs and Financing Project Cost by Component (in US$ million equivalent) Appraisal Actual/Latest Percentage of Estimate Estimate Appraisal Component US$ million US$ million 1. Curriculum Development 1.20 1.32 101.5 2. Teacher Education 3.20 3.20 1 3. Monitoring and Evaluation 0.60 0.76 126.6 Total Baseline Cost 5.00 5.28 Physical Contingencies 0.40 0.00 Price Contingencies 0.10 0.00 Total Project Costs 5.50 5.28 Total Financing Required 5.50 5.28 Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent) Total Cost Expenditure Category Procurement Method1 (including contingencies) ICB NCB Other2 N.B.F. 1. Works Civil Works 0.00 1.30 0.00 0.00 1.30 (0.00) (1.20) (0.00) (0.00) (1.20) 2. Goods Equipment and materials 0.00 0.00 1.20 0.00 1.20 (0.00) (0.00) (1.20) (0.00) (1.20) Printing 0.00 0.53 0.00 0.00 0.53 (0.00) (0.53) (0.00) (0.00) (0.53) 3. Services Technical Assistance & Consulting Services 0.00 0.00 1.20 0.00 1.20 (0.00) (0.00) (1.20) (0.00) (1.20) Training, study tours, fellowships, etc. 0.00 0.00 0.80 0.00 0.80 (0.00) (0.00) (0.80) 0.00 (0.80) 4. Miscellaneous 0.00 0.00 0.45 0.00 0.45 Recurrent Costs 0.00 0.00 (0.05) (0.00) (0.05) Total 0.00 1.83 3.65 0.00 5.50 (0.00) (1.73) (3.25) (0.00) (5.0) 1Figures in parenthesis are the amounts to be financed by the IDA credit. All costs including contingencies. 2Includes civil works and goods to be procured through national shopping, consulting services, services of contracted staff of the project management office, training, technical assistance services, and incremental operating costs related to (i) managing the project, and (ii) re-lending project funds to local government units. - 24 - Project Costs by Procurement Arrangements (Actual/Latest Estimate) (US$ million equivalent) Expenditure category Procurement Method1 Total Cost ICB NCB QCBS IS NS CQ IC DC SS Other Civil Works 1.27 1.27 Consultant services, Training, study tours and fellowships 0.96 0.56 0.41 0.41 2.34 Goods 0.75 0.57 0.21 0.01 1.54 Operating costs 0.13 0.13 Total 5.28 1Figures in parenthesis are the amounts to be financed by the IDA credit. All costs including contingencies. 2Includes civil works and goods to be procured through national shopping, consulting services, services of contracted staff of the project management office, training, technical assistance services, and incremental operating costs related to (i) managing the project, and (ii) re-lending project funds to local government units. Project Financing by Component (in US$ million equivalent) Percentage of Appraisal Component Appraisal Estimate Actual/Latest Estimate IDA Govt. CoF. Bank Govt. CoF. IDA Govt. CoF. 1. Curriculum 1.20 0.10 0.00 0.00 0.00 0.00 0.0 0.0 0.0 Development 2. Teacher Education 2a. Pilot TEIs 1.00 0.30 0.00 0.00 0.00 0.00 0.0 0.0 0.0 2b. Pilot Schools 1.50 0.40 0.00 0.00 0.00 0.00 0.0 0.0 0.0 3. Monitoring and Evaluation 3a. PIU 0.00 0.40 0.00 0.00 0.00 0.00 0.0 0.0 0.0 3b. M&E 0.00 0.20 0.00 0.00 0.00 0.00 0.0 0.0 0.0 - 25 - Annex 3. Economic Costs and Benefits Not applicable. - 26 - Annex 4. Bank Inputs (a) Missions: Stage of Project Cycle No. of Persons and Specialty Performance Rating (e.g. 2 Economists, 1 FMS, etc.) Implementation Development Month/Year Count Specialty Progress Objective Identification/Preparation 09/12/1998 - 7 Team Leader, 1 Sr. Education 09/26/ 1998 Specialist, 1 Education Economist, 2 Operations Officer, 1 Operations Analyst, 1 Education Specialist Consultant 12/ 07/1998 - 3 Team Leader, 1 Education 12/17/1998 Economist, 1 Operations Officer Appraisal/Negotiation 03/14/1999 - 7 1 Team Leader/Education 03/28/1999 Economist, 1 Education Economist, 1 Senior Education Specialist, 1 Operations Officer, 1 Operations Analyst, 1 Senior Procurement Officer, 1 Financial Management Specialist, Supervision 08/29/1999 - 5 1 Team Leader, 1 Operations S S 09/10/1999 Analyst, 1 Financial Management Specialist, 2 Operations Officer 11/22/1999 - 3 1 Team Leader/Education S S 11/26/1999 Economist, 1 Operations Officer, 1 Program Team Leader 05/18/2000 - 5 1 Team Leader/Education S S 05/19/2000 Economist, 1 Education Specialist, 1 Procurement Specialist, 1 Senior Operations Analyst, 1 Operations Officer 11/01/2000 - 2 1 Team Leader/Education S S 11/11/2000 Economist, 1 Operations Officer 02/26/2001 - 5 1 Team Leader/Education U S 03/10/2001 Economist; 2 Operations Officers; 1 Financial Management Specialist; 1 Program Team Leader 09/02/2001 - 5 1 Team Leader/Education U S 09/10/2001 Economist, 3 Operations Officers, 1 Senior Implementation/Procurement Specialist 11/27/2001 - 4 1 Team Leader/Education U S 12/07/2001 Economist, 1 Operations Officer, - 27 - 1 Education Sector Manager, 1 Social Scientist, 06/20/2002 - 2 1 Team Leader/Education S S 06/28/2002 Economist, 1 Operations Officer 06/23/2003 - 3 1 Team Leader/Education 06/27/2003 Economist, 1 Program Team Leader, 1 Education Sector Manager 11/03/2003 - 3 1 Team Leader/Education S S 11/05/2003 Economist, 1 Operations Officer, 1 Financial Management Specialist 01/19/2004 - 4 1 Team Leader/Education S S 01/24/2004 Economist, 1 Operations Officer, 1 Financial Management Specialist, 1 Procurement Specialist ICR 04/19/2004 - 7 1 Team Leader/Education 04/30/2004 Economist, 1 Senior Education Specialist, 1 Education Economist, 1 Operations Officer, 1 Senior Procurement Specialist, 1 Financial Management Specialist, 1 Architect/Project Management Specialist Consultant (b) Staff: Stage of Project Cycle Actual/Latest Estimate No. Staff weeks US$ ('000) Identification/Preparation 34.5 81 Appraisal/Negotiation 20.4 48 Supervision 147.0 343 ICR 11.1 31 Total 213.0 503 - 28 - Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating Macro policies H SU M N NA Sector Policies H SU M N NA Physical H SU M N NA Financial H SU M N NA Institutional Development H SU M N NA Environmental H SU M N NA Social Poverty Reduction H SU M N NA Gender H SU M N NA Other (Please specify) H SU M N NA Private sector development H SU M N NA Public sector management H SU M N NA Other (Please specify) H SU M N NA - 29 - Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating Lending HS S U HU Supervision HS S U HU Overall HS S U HU 6.2 Borrower performance Rating Preparation HS S U HU Government implementation performance HS S U HU Implementation agency performance HS S U HU Overall HS S U HU The implementation agency's performnace rating is based on two different periods in the project. The first part (1999-2001) was concidered to be unsatisfactory while the second period (2001-2004) was rated highly satisfactory. - 30 - Annex 7. List of Supporting Documents 1. Azerbaijan Country Assistance Strategy (Report No. 15948-AZ). World Bank, August 27, 1996. 2. Azerbaijan Country Assistance Strategy (Report No. 19892-AZ). World Bank, November 29, 1999. 3. Project Appraisal Document: Education Reform Project (Azerbaijan). World Bank, May 4, 1999. 4. CITO Group. Final Report. Curriculum development. Preparation of educational materials. Student assessment consultancy. August 2003. 5. Wagner, Joachim. Azerbaijan Education Reform Project. Project Monitoring and Evaluation Component. Final Report. September 2003. 6. CITO Group. Monitoring and Evaluation Study. March 2004. - 31 - Additional Annex 8. Borrower's Completion Report (Summary) Report on Implementation of the Credit Agreement 3220-AZ Education Reform Project in Azerbaijan between the Republic of Azerbaijan and the International Development Association (the World Bank) Ministry of Education Republic of Azerbaijan April 26, 2004 The "Development Credit Agreement" 3220-AZ was signed on July 29, 1999 between the Government of Azerbaijan and the International Development Association and it was ratified by the subsequent decree of the President of Azerbaijan Republic. According to the World Bank classification the Credit is characterized as a Learning and Innovation Loan. In other words, one of the objectives of the Credit is to create necessary capacity and skills regarding the rules, procedures and regulations of the World Bank, provide proper training of staff, support educational innovations in Azerbaijan education system. The overall cost of the Project was 5.5 million US dollars, and 500 thousand dollars out of this sum was the share of Azerbaijan Government. 27 pilot institutions were identified for the implementation of the Project. These included 20 pilot schools in Nakhchivan Autonomous Republic (4), Baku (8), Ganja (3) and Sumgait (1) cities and Lankaran (4); the former Head Institute for In-service Training and Re-training of Pedagogical Staff and its Lankaran branch; teacher in-service training institutes located in Nakhchivan and Ganja; Baku City Institute for In-service Training and Re-training of Pedagogical Staff; former Scientific-Methodological Center for Educational Problems and Institute of Pedagogical Sciences Scientific Researches. The Education Reform Project focused on specific approaches for carrying out reforms in the following three main directions: 1. Reforms in the provision of educational plans and syllabi, and instructional-methodological materials 2. Reforms in teacher in-service training 3. Project monitoring and evaluation. Among the successful accomplishments of the Ministry of Education in the area of education reform is development and printing of 9 (nine) new generation syllabi for grades I-II and V-VI of pilot schools based on national standards for general education and Basic educational plan. The prominent scientists-pedagogues and education specialists, progressive teachers of the Republic were involved in this work. Starting from 2001-2002 school year the subjects of Azerbaijan language and literature, math and computer science, history, English language (grades V and VI), music and arts (grades I-II, V-VI), Azerbaijan language (for grades I-II, V-VI of schools with Russian as the instruction language), Russian language and literature (for grades V-VI of schools with Russian as the instruction language) are being taught according to the new syllabi. In connection with the implementation of these syllabi, 252 teachers and methodologists have passed special training courses based on the system "training of trainers". Staring from the year 2001 major attention was given to the fundamental change in learning environment at pilot schools, and application of active and interactive teaching/learning methods in instruction process. 204 primary grade teachers from - 32 - pilot schools in the year 2000, and 527 subject teachers instructing at grades V-VI and 110 methodologists from pilot in-service training institutes in the year 2001have participated in special courses on modern teaching methods. And 66 teachers from pilot schools and methodologists from related REOs, as well as 62 methodologists from in-service training institutes have attended in May-August of the year 2003 the 104-hour training course conducted in three stages based on the "training of trainers" model and have subsequently received certificates. Those teachers and methodologists will train their colleagues in active and interactive learning methods starting from the new school year. Special seminars devoted to issues related to the application of those methods have been organized each month at one of the pilot schools. Seminars of this type with participation of teachers and methodologists from pilot schools, prominent scientists-pedagogues were held in Baku schools # 7, 18, 70, 82, 181, 275, 287, Sumgait city school # 25, Nakhchivan city school # 11 and Aralig village school in Sharur region. In accordance with modern pedagogical tendencies aiming to apply new experiences in Azerbaijan educational system, three education specialists selected on a competitive basis received a master's degree in Britain, at Leads University and King's College on "Integrative subjects", "Math and science" and "Assessment in education" on a special scholarship from the Ministry of Education. In May 2000, 7 employees of the Ministry of Education were sent to the Republic of Turkey with the purpose of learning the features of implementation of World Bank financed education reform projects. A delegation representing the Ministry and the pilot institutions and consisting of 10 persons has closely familiarized with the education system of the United States of America in April 2001, and held 18 official meetings, participated in 15 seminars and got acquainted with the work practices of 12 general schools and other educational institutions in 5 big cities of the US. In July of the same year 28 officials of the Ministry and pilot institutions have participated in the training course on "Modern methods of general school management in the course of democratization of the society" organized in Israel. In May 2002 a delegation of the Ministry of Education consisting of 8 persons have closely familiarized with the educational system of the Netherlands and Britain. During the course of the study tour the delegation has studied in both countries the functioning of 10 units dealing with development of syllabi, textbooks and other materials, evaluation of activities of educational institutions, visited 2 universities and 2 general schools, participated in 21 seminars. In October 2002, seminars on "Modern views on education quality" and "Monitoring and evaluation of educational system" were organized by German specialists for 80 leading officials in the field of education and in December of the same year British and US specialists conducted seminars on "Evaluation of teacher performance and development of professional level" for the leadership and employees of the Ministry of Education, in-service training institutions and pilot schools. In July 2003 a delegation of 20 persons consisting of 2 deputy ministers and heads of departments and divisions of the Ministry passed a two-week training course on "Management of the education reform project" organized in Training Center of the International Labor Organization located in Italy. In the period of 1-14 September 9 educational employees were on a study tour in Finland and Sweden devoted to the issues of curriculum development. In the period of 2000-2002 international consultants have carried out special researches and made relevant recommendations for textbook provision in Azerbaijan, educational expenses and finance, parent and community involvement in school life, development of teacher professional level. Specialists from an international consulting company conducted research on "Monitoring and Evaluation of the Education Reform Project". Under the Education Reform Project it is also envisaged to strengthen the material-technical resources of pilot institutions. Renovations were carried out in pilot institutions in 2000 and approximately 1.28 million - 33 - US dollars were spent for this purpose. Computers, copy machines and other equipment, as well as furniture in the amount of approximately 1.50 million US dollars have been purchased for the Ministry of Education and a number of pilot institutes. In the years of 2000-2003 512 sets of computers, 77 printers, 34 copy machines, 100 TVs, 208 stereo tape recorders, 92 videos, 189 various projectors, including multimedia projectors have been purchased for all pilot schools and institutes. In addition pilot schools were supplied with 5,950 sets of student's desks/chairs, 470 boards for using with chalk and marker, 1897 chairs for conference rooms etc. (see, Table 1). TABLE 1 Equipment supplied MOE PIU IEP TTIs PS TOTAL Computer 37 9 37 152 277 512 Printer 9 4 7 21 36 77 Copy machine 4 1 3 6 20 34 TV 6 1 5 32 56 100 Stereo tape recorder - - - 36 172 208 Video, DVD/VCR - 1 5 30 56 92 Projectors 3 2 6 56 122 189 Fax machines 8 1 2 5 20 36 Video camera 1 1 1 6 20 29 Student's desk with two chairs 5950 sets 5950 sets Writing boards 470 470 Cabinet furniture (set) 15 1 2 4 22 Office desk 67 22 170 290 549 Office chair 66 50 330 290 736 Bookcase 28 20 110 120 278 Safe 7 2 6 15 Chairs for conference room 1897 1897 On the whole, 92.85 percent of 5 million US dollars allocated by the World Bank and 87.34 percent of 500 thousand dollars of the contribution of the Azerbaijan Republic have been disbursed so far (see Table 2 for details). TABLE 2 IDA Credit funds - 3220AZ # Category Description Allocated Disbursed Undisbursed 1 CIVILWORKS 1154955 1152279 2676 2 CONSULTANTSERVICES 1816792 1721073 95719 3 TRAINING 622577 616350 6227 4 GOODS 1307273 1272810 34463 5 OPERATINGCOSTS 70860 64591 6269 TOTAL 4972457 4827103 145354 % 100.00 97.08 2.92 - 34 - Azerbaijan Government contribution # CategoryDescription Allocated Disbursed Undisbursed 1 CIVILWORKS 128544 127457 1087 2 CONSULTANTSERVICES - - - 3 TRAINING - - - 4 GOODS 321456 272850 48606 5 OPERATINGCOSTS 50000 46649 3351 TOTAL 500000 446956 53044 % 100.00 89.39 10.61 Due to regression in the first 2 years of the Credit Agreement and because of some mistakes made, a range of activities envisaged were not carried out in time. Thus, due to a number of shortcomings in the activities of the Consortium for International Development selected in October 2000 for rendering consultancy services in "Curriculum development and teacher in-service training" and because of its careless performance of the terms of the contract and the activities it envisaged the Ministry of Education started in July 2001 the procedure for cancellation of the contract with the consortium and this process was completed at the end of November of the same year. As a result, a range of activities provided for in pilot schools in the fields of education content, curriculum development, teacher in-service training have not been carried out and funds for this purpose amounting to 405 thousand US dollars have remained unused. In the Project Appraisal Document funds in the amount of 400 thousand US dollars are provided for the purchase of textbooks for pilot schools, 130 thousand US dollars are provided for development and printing of methodological aids for teachers. Because this issue has not been precisely detailed in the Project Appraisal Document a series of negotiations have been held between the Ministry of Education and the World Bank, suggestions were made and only in April 2002 a suitable agreement was reached for solving the issue. Because completion of these two large-scale activities till the end of the Credit Agreement, i.e. the 31st of December was not feasible, with consent of Azerbaijan Government the World Bank was addressed to in order to prolong the Credit Agreement for 9 months and on 16th of July, 2002 the World Bank approved the prolongation of the Credit Agreement till September 30th, 2003. During this time-period a bid for preparation of new textbooks, workbooks and teachers' methodological aids was conducted according to the plan. For the first time in Azerbaijan, starting from 1st of September textbooks, workbooks and teachers' methodological aids will be provided for students of grades V and VI in pilot schools on the subject of "Literature", and for the schools with Russian as the language of instruction on the subject of "Russian language" and "Literature". Appropriate activities have been carried out in order to complete the unfinished activities in the field of curriculum development, assessment of student achievements and teacher in-service training. Selected external consultants have commenced their consultancy services in January-February of the current year and these activities are almost completed. These consultants conducted seminars on "Curriculum development", "Student assessment and national assessment" for specialists from the Institute of Educational Problems, in-service training institutes and a group of progressive teachers (total of 62 persons) in the period of February-June of the current year. For the first time a national student assessment in compliance with the international standards was organized for 1,600 grade 5 students from 8 pilot and 12 non-pilot schools located in Baku, Ganja, Sumgait and Lankaran. Task groups involving the - 35 - international and national consultants and 26 education specialist have developed curricula on Native Language and Math (V and IX grades), as well as models for national and student assessments. Another international consultant conducted seminars for 155 methodologist from in-service institutes, IEP employees and pilot school teachers on "Curriculum development for in-service institutes", "Active and interactive teaching/learning methods". Curricula in 20 fields based on 6 subjects were developed for the first time, with help of this Center. Taking into consideration the second prolongation of the Credit Agreement the Ministry of Education (PIU) prepared a corresponding implementation plan and is now implementing this plan. The Credit 3220-AZ was closed March 31, 2004. - 36 - - 37 -