Document of The World Bank Report No: ICR00001134 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-37690) ON A CREDIT IN THE AMOUNT OF SDR 13.2 MILLION (US$ 18 MILLION EQUIVALENT) TO THE REPUBLIC OF AZERBAIJAN FOR AN EDUCATION SECTOR DEVELOPMENT PROJECT May 17, 2010 Human Development Sector Unit South Caucasus Country Unit Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective May, 2010) Currency Unit = New Azerbaijan Manat (AZN) AZN 1.00 = US$ 1.24 US$ 1.00 = 0. 80AZN FISCAL YEAR January 1 ­ December 31 ABBREVIATIONS AND ACRONYMS APC Adaptable Program Credit NGO Non-Governmental Organization APL Adaptable Program Lending OECD Organization for Economic Cooperation and Development CAR Center for Applied Research OSI Open Society Institute CAS Country Assistance Strategy PAD Project Appraisal Document CIS Commonwealth of Independent States PAPU Policy Analysis and Planning Unit DCA Development Credit Agreement PCU Project Coordination Unit ECD Early Childhood Development PDO Project Development Objective EMIS Education Management Information System PHRD Policy and Human Resources Development Fund EMISU Education Management Information System Unit PIRLS Program on International Reading Literacy Study EMP Environmental Management Plan PISA Program for International Student Assessment ERP Education Reform Project PRSP Poverty Reduction Strategy Paper ESDP Education Sector Development Program PSA Production Sharing Agreement FSU Former Soviet Union POM Project Operational Manual GDP Gross Domestic Product QAG Quality Assessment Group GOA Government of Azerbaijan QER Quality Enhancement Review ICTs Information and Communication Technologies SAPPU Strategic Analysis, Planning and Personnel Unit IDA International Development Association SDR Special Drawing Rights IDP Internally Displaced People SGP School Grant Program IEP Institute for Education Problems SIL Specific Investment Loan ISR Implementation Status and Results Report SPPRED State Program on Poverty Reduction and Economic Development KPI Key Performance Indicators TTIs Teacher Training Institutes LIL Learning and Innovation Loan TIMSS Trends in International Mathematics and Science Study MOE Ministry of Education UNICEF United Nations International Children's Fund MOF Ministry of Finance VET Vocational Education and Training NCSA National Center for Student Assessment Vice President: Philippe Le Houerou Country Director: Asad Alam Sector Manager: Mamta Murthi Project Team Leader: Juan Manuel Moreno Olmedilla ICR Team Leader: Nino Kutateladze REPUBLIC OF AZERBAIJAN Education Sector Development Project CONTENTS Data Sheet..........................................................................................i 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 7 3. Assessment of Outcomes .......................................................................................... 12 4. Assessment of Risk to Development Outcome......................................................... 19 5. Assessment of Bank and Borrower Performance ..................................................... 20 6. Lessons Learned ....................................................................................................... 23 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 23 Annex 1. Project Costs and Financing .......................................................................... 25 Annex 2. Outputs by Component ................................................................................. 26 Annex 3. Economic and Financial Analysis ................................................................. 37 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 38 Annex 5. Beneficiary Survey Results ........................................................................... 40 Annex 6. Stakeholder Workshop Report and Results................................................... 41 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 43 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders...................... 65 Annex 9. List of Supporting Documents ...................................................................... 66 MAP .............................................................................................................................. 67 A. Basic Information Education Sector Country: Azerbaijan Project Name: Development Project Project ID: P070989 L/C/TF Number(s): IDA-37690 ICR Date: 05/17/2010 ICR Type: Core ICR AZERBAIJAN Lending Instrument: APL Borrower: REPUBLIC Original Total XDR 13.2M Disbursed Amount: XDR 13.0M Commitment: Revised Amount: XDR 13.0M Environmental Category: B Implementing Agencies: Project Coordination Unit Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 09/30/2002 Effectiveness: 12/18/2003 Appraisal: 02/24/2003 Restructuring(s): Approval: 05/27/2003 Mid-term Review: 03/06/2006 06/09/2006 Closing: 03/31/2008 09/30/2009 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Satisfactory Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: i C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of No None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 25 25 Primary education 40 40 Secondary education 20 20 Sub-national government administration 5 5 Tertiary education 10 10 Theme Code (as % of total Bank financing) Education for all 33 33 Education for the knowledge economy 67 67 E. Bank Staff Positions At ICR At Approval Vice President: Philippe H. Le Houerou Philippe H. Le Houerou Country Director: Asad Alam Asad Alam Sector Manager: Mamta Murthi Mamta Murthi Project Team Leader: Juan Manuel Moreno Olmedilla Meskerem Mulatu ICR Team Leader: Nino Kutateladze ICR Primary Author: Nino Kutateladze F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The Project constitutes the first phase of a three phase Adaptable Program Loan (APL). The overall objective of the Program is to increase learning achievements for all students to meet the needs of a market economy and improve efficiency of general education. The development objective of this Project was to strengthen the capacity to plan, manage and ii monitor the reform program more effectively and efficiently and initiate the highest priority investments in selected districts. Revised Project Development Objectives (as approved by original approving authority) Not applicable. (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years National curriculum framework for grades 1-11 developed, standards developed Indicator 1 : and approved for grades 1-4 by the MoE; new syllabi developed at least for two grades and ready for piloting. National National Curriculum curriculum framework for framework for general education grades 1-11 developed and developed, adopted. Subject Value standards curricula for quantitative or None developed and primary grades Qualitative) approved for approved. Grade 1 grades 1-4 by the curricula piloted in MoE; new syllabi the 2007-2008 developed at least academic year and for two grades and introduced nation- ready for piloting. wide in 2008-2009. Date achieved 03/15/2004 03/31/2008 03/31/2008 Comments (incl. % Achieved. achievement) At least 90 percent of children in grades 1-5 have free of charge textbooks in core Indicator 2 : subjects. Wide disparities in access to textbooks were At least 90 percent reported among schools of children in 95% of children in Value and particularly in urban grades 1-5 have grades 1-5 received quantitative or versus rural areas. free of charge free of charge Qualitative) Source: 2005 Baseline textbooks in core textbooks. Report on Preliminary subjects. Information on Abundance of Textbooks. Date achieved 06/30/2005 09/30/2007 09/30/2007 Comments Achieved. (incl. % Source: Evaluation survey carried out in 2007. Textbook Survey carried out in achievement) May 2009 by Sigma confirmed lack of disparities in access to core textbooks. Indicator 3 : New funding formula and mechanism prepared and implemented on an iii experimental basis in three pilot districts. New funding Per-student formula and financing formula mechanism was developed and Value prepared and implemented on an quantitative or None implemented on an experimental basis Qualitative) experimental basis in 59 schools in the in three pilot three pilot districts districts. in 2007. Date achieved 03/15/2002 03/31/2008 03/31/2008 Comments (incl. % Achieved. achievement) District rationalization plans prepared in three pilot districts and approved by the Indicator 4 : Government and IDA. Plans were developed and District approved by the rationalization Ministry of Value plans prepared in Education. The quantitative or None three pilot districts district Qualitative) and approved by rationalization the Government plans were and IDA. implemented in the three pilot districts. Date achieved 03/15/2004 03/31/2008 03/31/2008 Comments (incl. % Achieved. achievement) Completion of draft organizational and staffing plan for the Ministry of Indicator 5 : Education (including the Institute of Educational Problems) Completion of draft organizational and staffing plan for Value the Ministry of quantitative or None Plan delivered. Education Qualitative) (including the Institute of Educational Problems) Date achieved 03/15/2004 03/31/2008 03/31/2008 Comments (incl. % Achieved. achievement) Completion of a pilot sample-based assessment of grade 4 student achievement Indicator 6 : in Math and Azeri language. Value Completion of a The sample-based None quantitative or pilot sample-based assessments in iv Qualitative) assessment of mother tongue and grade 4 student mathematics in achievement in grade 4 and 9 were Math and Azeri conducted in April- language. May 2006. Analysis of the assessment results was made available in 2007 and widely discussed with the MOE and stakeholders. Date achieved 03/15/2004 11/30/2006 11/30/2006 Comments (incl. % Achieved. achievement) Curriculum development capacity established in the restructured Institute for Indicator 7 : Education Problems by 2006. Curriculum development capacity Adequate capacity Value established in the was established quantitative or None restructured within the Qualitative) Institute for Curriculum Center. Education Problems by 2006. Date achieved 03/15/2004 12/31/2006 12/31/2006 Comments (incl. % Achieved. achievement) Management and planning capacity of the MOE and local governments Indicator 8 : strengthened and performs its functions efficiently and effectively. Management and MOE capacity was planning capacity strengthened by the of the MOE and introduction of Value local governments planning and quantitative or None strengthened and reporting system Qualitative) performs its and establishment functions of key new units, efficiently and such as SAPU, effectively. MAU and EMISU. Date achieved 03/15/2004 03/31/2008 03/31/2008 Comments (incl. % Achieved. achievement) Education statistics, including learning outcomes, are readily available at the Indicator 9 : national level and pilot districts and published in the form of "Education Indicators" for policy makers and the public by 2006. Value None. Education The MOE statistical v quantitative or statistics, report using new Qualitative) including learning EMIS was issued. outcomes, are Results of the readily available at examinations and the national level student assessments and pilot districts are available in the and published in Ministry's website. the form of "Education Indicators" for policy makers and the public by 2006. Date achieved 03/15/2004 12/31/2006 03/31/2008 Comments (incl. % Achieved. achievement) Policy makers and education stakeholders use the results of learning outcomes Indicator 10 : and reliable statistics in the policy analysis and strategy development. Policy makers and education ESDP made stakeholders use excellent progress the results of in making available Value learning outcomes reliable school quantitative or None and reliable statistics and data Qualitative) statistics in the on student learning policy analysis outcomes for and strategy decision-making. development. Date achieved 03/15/2004 03/31/2008 03/31/2008 Comments (incl. % Achieved. achievement) 25 percent of primary teachers (about 9,000) are familiar with the new Indicator 11 : curriculum, teaching methods, and assessment targets and begin to reflect those changes in their teaching practices during the 2007-2008 school year. 25 percent of primary teachers About 18,000 (about 9,000) are primary school familiar with the teachers were new curriculum, trained. 40 percent Value teaching methods, of 1st grade quantitative or None. and assessment teachers are Qualitative) targets and begin satisfied with the to reflect those quality of trainings changes in their received, 80 percent teaching practices indicate the need during the 2007- for more trainings. 2008 school year. Date achieved 03/15/2004 09/01/2007 09/01/2007 vi Comments Achieved. (incl. % Source: 2009 Textbooks and Curriculum Implementation Survey. achievement) Present disparities in access to core textbooks (based on new curricula) in basic Indicator 12 : education grades (1-9) eliminated by 2006. Present disparities Over 95% of in access to core children in grades textbooks (based Value 1-5 received free of on new curricula) quantitative or None charge textbooks. in basic education Qualitative) No disparities were grades (1-9) reported in access eliminated by to core textbooks. 2006. Date achieved 03/15/2004 12/31/2006 12/31/2006 Comments Achieved. (incl. % Source: 2007 ESDP Evaluation Survey and 2009 Textbooks and Curriculum achievement) Implementation Survey. Student-teacher ratio in general education increased from 10:1 in 2002 to 12:1 in Indicator 13 : 2007 in the pilot districts. Moderate efficiency Student-teacher gains were ratio in general achieved in the Value education pilot districts, as quantitative or None increased from measured by Qualitative) 10:1 in 2002 to increases up to 11 12:1 in 2007 in the percent in the pilot districts. average student- teacher ratios. Date achieved 03/15/2002 12/31/2007 03/31/2008 Comments (incl. % Partially Achieved. achievement) Efficiency and finance reforms, including district rationalization plans, prepared Indicator 14 : and implemented successfully in the three pilot districts by 2007. Efficiency and Efficiency and finance reforms, finance reforms including district were carried out in Value rationalization the pilot districts, quantitative or None. plans, prepared although Qualitative) and implemented implementation successfully in the faced some three pilot shortcomings. districts by 2007. Date achieved 03/15/2004 12/31/2007 12/31/2007 Comments (incl. % Partially Achieved. achievement) Indicator 15 : Society at large express support for education reform by 2007. Value None. Society at large Survey of parents, vii quantitative or express support for teachers and school Qualitative) education reform officials shows a by 2007. moderate increase from the baseline survey conducted in 2005. However, satisfaction of rural population has decreased from 42 percent at baseline to 37 percent in the follow on survey. Date achieved 03/15/2004 12/31/2007 03/31/2008 Comments Partially achieved. (incl. % 2007 ESDP Evaluation Survey. achievement) (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 : Policy and Planning Unit Established. Strategic Analysis Value Policy and and Planning Unit (quantitative None Planning Unit (SAPU) was or Qualitative) Established. established in 2004 and fully staffed. Date achieved 03/15/2004 03/15/2005 03/31/2005 Comments (incl. % Achieved. achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 06/30/2003 Satisfactory Satisfactory 0.00 2 12/29/2003 Satisfactory Satisfactory 0.00 3 03/02/2004 Satisfactory Satisfactory 0.25 4 06/16/2004 Satisfactory Satisfactory 0.25 5 12/21/2004 Satisfactory Satisfactory 1.29 6 03/28/2005 Moderately Satisfactory Moderately Satisfactory 2.27 7 08/15/2005 Moderately Satisfactory Moderately Satisfactory 2.49 8 02/28/2006 Moderately Satisfactory Moderately Satisfactory 3.32 9 02/22/2007 Moderately Satisfactory Moderately 8.86 viii Unsatisfactory 10 09/12/2007 Satisfactory Satisfactory 15.35 11 06/18/2008 Satisfactory Satisfactory 18.58 12 12/19/2008 Satisfactory Satisfactory 18.87 13 06/26/2009 Satisfactory Satisfactory 19.02 H. Restructuring (if any) Not Applicable I. Disbursement Profile ix 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal Country Background 1. Azerbaijan declared independence from the Soviet Union in 1991. The country experienced many of the similar challenges as other transition countries in the Commonwealth of Independent States (CIS) ­ severe economic contraction during the early 1990s, deterioration in social services and infrastructure, and a sharp increase in poverty. In addition, the early years of independence were overshadowed by the Nagorno- Karabakh armed conflict with neighboring Armenia resulting in nearly one million Internally Displaced People (IDPs) and refugees. On the positive side, the country emerged from the Soviet Union with significant oil and gas resources. The signing of a number of Production Sharing Agreements (PSAs) with oil companies in the mid-1990s and initiation of a stabilization and structural reform program marked the beginning of a turnaround in Azerbaijan's development prospects. Economic recovery began in 1996 with GDP growth averaging more than 8 percent over the period of 1997-2002. Despite this progress, at the time of project preparation (2002), GDP was still 40 percent less than in 1989 and nearly half the population was below the national poverty line. 2. The task of overcoming the legacy of inefficient production, outdated technology, monopolies, environmental neglect and the absence of the institutions of a market economy proved more difficult in Azerbaijan than initially anticipated. The record of policy and institutional reform had been mixed, but there had been many clearly positive development achievements since 2001. The success of the stabilization program, growing foreign investor interest, and the prospect and associated risks of large-scale oil revenues seemed to have strengthened the authorities' commitment to reforms. As part of the Poverty Reduction Support Program (PRSP), formerly known as the State Program on Poverty Reduction and Economic Development (SPPRED), the Government developed a medium- term strategy emphasizing tackling poverty and strengthening governance, with technical cooperation from the international community. The most significant reforms included establishing a State Oil Fund; strengthening governance in the banking sector; starting to address implicit subsidies and the need for greater financial transparency in the utilities sectors; some improvements in the structure of the public sector including consolidation of several activities in the new Ministry of Economic Development; enactment of a modern tax code; and the adoption of a more transparent budget execution and accounting, and strengthening of internal and external audit capacity. Sector Background 3. The education sector suffered during the transition years. Although enrolment rates remained high at independence and Azerbaijan was on track to meet the education related Millennium Development Goals (MDGs), the quality of education suffered due to sharp declines in real public expenditures on education. As a percentage of GDP, the public spending on education in Azerbaijan had continually fallen since the end of the Soviet period from 7.6 percent in 1993 to 2.7 percent of GDP in 2006. Even though there were no conclusive data to monitor trends in student learning outcomes during the transition, the results from the Student University Entrance Examination at the end of general education 1 (grade 11) indicate poor quality of general education. For example, more than half of the candidates score below the failing mark (300 points). The following main problems were likely to have contributed to the decline in the quality of general education: outdated curriculum and teaching methods, ineffective teacher education and training system, low teacher salaries and poor learning environment as suggested by shortages of textbooks, reading materials and supplies. The latter point was further influenced by a shift from Cyrillic to Latin script in Azerbaijan, in the early 1990s. Partly as a result of this shift, there was a severe shortage of reading materials in Latin script that could foster reading habits of students, support core curricula and encourage independent thinking. Moreover, growing inequalities in access to high quality general education, inefficient use of available resources and weak management, planning and monitoring capacity at the central and local levels posed further challenges to the sector. 4. To address the above challenges, the Government developed and approved a comprehensive Education Reform Program in 1999, which had a strong focus on improving the quality and relevance of student learning to meet the needs of a market economy and strengthening the capacity for managing resources for education effectively, efficiently and equitably. The Program targeted: (i) curriculum reform and teacher training; (ii) provision of educational materials including textbooks; and (iii) educational management and finance. As such, the Government followed through with reforming the curriculum and in-service teacher training, and developed new textbooks and materials for general education. By the time of appraisal (Spring 2003), the Government had identified further priorities and specific directions for the development of the education sector in the draft Ten-Year Education Reform Strategy (2003-2013) prepared by the Ministry of Education. 5. The capacity to carry out these ambitious reforms as well as to introduce the required changes was largely dependent on access to additional financial and technical resources. The Education Sector Development Project (ESDP), designed as a three phase Adaptable Program Loan (APL), sought to support the Government's Program by addressing four main sectoral issues: (i) The need to improve the quality and relevance of general education. (ii) The need to enhance efficiency in and effectiveness of the use of resources in the general education sector. (iii) The need to reverse the deterioration of the coverage of general education enrolment. (iv) The need to raise the capacity of the MOE, local education authorities and school principals to plan, manage, and monitor the education sector more efficiently, effectively, and equitably, including strengthening its accountability to the key stakeholders. Rationale for Bank Assistance 6. The ESDP was targeted at improving the quality of general education (grades 1 to 11) and realigning the entire sector with the needs of the emerging market economy and social conditions. While the needs were pressing at all levels of education, it was agreed that the proposed Program would focus on general education because of limited IDA resources and the need for critical reforms at the general education level, which were essential for successful reforms at the other levels. Moreover, the inclusion of other sub- 2 sectors in the proposed Program would add complexity to the Program and the weak implementation capacity of the MOE at various levels could jeopardize the reform process. 7. The focus of the Project interventions on general education was consistent both with the Government's State Program on Poverty Reduction and Economic Development (SPPRED) (2003-2005) and 2003-2005 Azerbaijan Country Assistance Strategy (CAS) since investments in general education were critical in the production of quality secondary education graduates and poverty alleviation. The longer-term perspective of the APL was to support the Government to carry out a system-wide reform, while providing the flexibility to adjust and refine interventions of an ongoing basis. Moreover, the experience, knowledge and skills gained under an earlier Bank supported education project (Learning and Innovation Loan - LIL) had created a good prerequisite for further expansion of the education sector reforms in Azerbaijan. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 8. The Project constituted the first phase of a three-phase Program, which had the overall objective of increasing learning achievements for all students to meet the needs of a market economy and improve the efficiency of general education. 9. The specific Project Development Objective (PDO) of phase 1 was to strengthen the capacity to plan, manage and monitor the reform program more effectively and efficiently and to initiate the highest priority investments in selected districts.1 10. The following Key Performance Indicators (KPIs) were identified for monitoring the Program, with six of them also serving as the outcome indicators for this Project. Additionally, achievement of these six selected performance indicators would serve as "triggers" for moving to the next phase of the Program. 11. The key performance indicators were as follows: (i) Quality and Relevance of General Education National curriculum framework for grades 1-11 developed, standards developed and approved for grades 1-4 by the MOE; and new syllabi at least for two grades developed and ready for implementation on a pilot or nationwide basis. (Trigger 1) At least 9,000 general education teachers having participated in in-service teacher training programs. The results of national sample-based student assessment of grade 4 and 9 students in Mathematics and Azeri Language showing improved achievement between 2005 and 2007. At least 90 percent of children in grades 1-5 have free of charge textbooks in "core subjects." (Trigger 2) 1 The second phase of the Program would expand coverage and extend the scope of interventions and develop further management skills, particularly at local levels, based on evolving needs. The final phase would achieve full national coverage, consolidate interventions, and review the achievements of the reform program. 3 (ii) Efficiency and Finance Reforms New funding formula and mechanism prepared and implemented on an experimental basis in the three pilot districts. (Trigger 3) District rationalization plans prepared in the three pilot districts, under terms of reference satisfactory to IDA, and approved by the Government and reviewed by IDA. (Trigger 4) Student-teacher ratio in general education increased from 10:1 in 2002 to 12:1 in 2007 in the three pilot districts. (iii) Equity and Access to Quality General Education Gross enrollment rates of the age cohort 6-16 years olds increased from about 94 percent in 2002 to about 96 percent in 2007. (iv) Management, Planning, and Monitoring Capacity Completion of a draft new organizational and staffing plan for the Ministry of Education (including the Institute for Education Problems). (Trigger 5) A Policy Analysis and Planning Unit (PAPU) in the MOE established and functioning by March 2004. Completion of a pilot sample-based reliable assessment of grade 4 student achievement in Mathematics and Azeri language. (Trigger 6) A department of education management information system in the MOE established and functioning to support planning, policy analysis and monitoring. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 12. There were no changes to the Project Development Objective and key performance indicators. 1.4 Main Beneficiaries 13. The 2003-2013 Program, including this Project, sought to benefit the entire Azeri population because of improved quality and relevance of general education. 14. More specifically, all children in primary and general secondary education, students and teaching staff in the existing teacher training institutes, and all those working in the education sector have benefited from the curriculum reform, teacher development activities, the provision of reading materials, and strengthened planning, management and monitoring capacity in the sector supported by the Project. This currently represents about 1.6 million students in grades 1-11. 15. Benefits of the Project have also accrued to about 45,000 students and 5,000 teachers in the three pilot districts who have benefited from the school improvement and rationalization efforts as well as a school grant program. 16. Long term benefits will be derived from the projected improvement in quality and relevance of student learning as a result of better management and effective leadership, and implementation of quality improvement programs in general education. 4 1.5 Original Components (as approved) 17. The lending instrument was a three-phase Adaptable Program Credit (APL) designed for a ten year period (2003-2013). This Project was the first phase of the APL and consisted of five components, as follows: Component 1. Quality and Relevance of General Education. (Original Estimated Cost: US$ 7.05 million equivalent) 18. The objective of this component was to enhance the quality and relevance of general education through the following subcomponents: Curriculum Development. This subcomponent aimed to strengthen the curriculum development capacity in the Institute for Education Problems (IEP), design and adopt the National Curriculum Framework, and prepare and implement new standards and syllabi for grades 1-11 and develop new syllabi at least for two grades. Teacher Development. This subcomponent was designed to support the MOE in two main areas: (a) continued development of in-service teacher training, including the mass-training of about 9,000 teachers (or 25 percent of the present primary education teachers), and (b) initial reforms for pre-service teacher education. Textbooks and Reading Materials. The objective of the subcomponent was to stimulate reading habits of grades 1-11 students and provide co-curricular reading materials as well as assist the Government in the development of its textbook policy. It was agreed that OSI-Azerbaijan would provide some of the technical assistance and training within the framework of the objectives of the Project in the area of textbook policy. Component 2. Efficiency and Finance Reforms. (Original Estimated Cost: US$8.86 million equivalent) 19. This component aimed to support the design and implementation of reforms in the financing of education, rationalization and school improvement in the three pilot districts through the following subcomponents: Financing and Budgeting Reforms. The subcomponent was designed to assist the MOE to institute reforms in financing and budgeting in order to gain efficiencies in the use of financial, human and physical resources in the education sector. Rationalization and School Improvement. The subcomponent sought to assist the MOE in three pilot districts to: (i) develop plans and rationalize the school network in order to gain efficiencies in school management and operations; (ii) develop standards for schools and school rehabilitation; and (iii) select and rehabilitate schools and provide furniture, basic equipment and teaching aids to the schools. In addition, this subcomponent sought to support improvement of school libraries, and the provision of information and communication technologies in a selected number of schools as part of the school improvement plans. 5 Component 3. Upgrading Schooling Quality in Less Advantaged Rayons: School Grant Program. (Original Estimated Cost: US$0.87 million equivalent) 20. The School Grant Program (SGP) was designed to provide supplemental financing to schools through a competitive process to improve attendance and enrollments of the poor, as well as improve teaching and learning in poor schools. OSI-Azerbaijan support was envisaged for the implementation of SGP through the provision of technical assistance and Training. Component 4. Management, Planning and Monitoring Capacity. (Original Estimated Cost: US$2.34 million equivalent) 21. This component aimed to strengthen the management and planning capacity in the MOE to use available resources more effectively and efficiently. The component included three subcomponents: Establishment of a New System of Student Assessment. The subcomponent sought to assist the MOE in developing a reliable and diagnostic system for the assessment of student learning outcomes. Establishment of Education Management Information System (EMIS). This subcomponent aimed to support the development of Education Management Information System (EMIS) in the MOE, the pilot districts and institutions integral to the implementation of education reforms (e.g. IEP). Development of Management and Planning Capacity of MOE. This subcomponent aimed to strengthen the capacity of the MOE to manage and plan the delivery of education services more effectively and efficiently, and increase accountability to its key stakeholders. In addition, limited support was provided for the development of the Center for Applied Research (CAR) in the Institute for Education Problems. Component 5. Project Coordination and Monitoring. (Original Estimated Cost: US$1.87 million equivalent) 22. The objective of this component was to support project management and implementation, monitoring and evaluation, and the design and implementation of communication strategy. The component would support strengthening a Project Coordination Unit (PCU) under the office of the Deputy Minister of Education (Rural Education). 1.6 Revised Components 23. No changes were made to the original components. 1.7 Other significant changes 24. Appreciation of the Special Drawing Rights (SDR) in relation to the US dollar (US$). In the course of project implementation, ESDP benefited from additional project financing resulting from the exchange rate gains. As a result of favorable exchange rate of the SDR vis-à-vis the US Dollar, the original IDA Credit was increased from US$18 million equivalent to US$20 million equivalent. These extra resources were spent by project completion to finance mostly participation in the international assessments of 6 student learning achievement (PIRLS, TIMSS, and PISA), as well as other activities chosen selectively to strengthen ongoing efforts aimed at improving quality of education and consistent with the approved PDO. 25. Project Extension. The Project closing date was extended two times from the original closing date of March 31, 2008 to September 30, 2009: The first extension of the closing date was for one year, until March 31, 2009. The extension was granted in December 2007 to enable more time for carrying out activities related to the implementation of the new curriculum in Grade 1. The second extension was for six months, until September 30, 2009. The extension was granted in March 2009 to allow the Borrower additional time necessary to complete the selected activities with the additional funds resulting from the appreciation of the SDR. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 26. Participatory Design and Comprehensive Background Analysis. The Project's quality at entry was positively influenced both by its participatory design and thorough background analysis. The preparation of the Project relied heavily on two highly participatory studies: (i) a social assessment in which local level stakeholders were consulted on the issues concerning education and about their needs and priorities, and (ii) a study of informal payments in education, which was based on a nationally representative sample. In addition, two two-day participatory stakeholder workshops were organized to discuss the Government's Education Reform Strategy, priorities for the next education project, and the draft design of the proposed project. 27. As part of project preparation, the Bank coordinated a joint identification mission with the available donors in February 2002 to take advantage of other donor activities, in particular, the work started by the Open Society Institute (OSI) in textbook policies and teacher training programs and by UNICEF in pilot teacher education program and preschool education program. It was agreed that OSI-Azerbaijan would finance limited activities related to (i) the preparation of the school improvement plans, (ii) support of the School Grant Program through the provision of technical assistance and training for schools to write and implement grant proposals, and (iii) textbook policy and standards. 28. Project preparation was supported by a PHRD grant in the amount of US$ 461,503, which financed several studies laying the foundation for the ESDP components. The studies were carried out in the following key areas: national standards and curriculum reform; teacher education; education economics and finance; education management; education research and policy; EMIS and school mapping; information technology; instructional technology; monitoring, evaluation and assessment; and textbook development. 29. Strategic Relevance. The Project was consistent with the strategic goals of the 2003 Country Assistance Strategy (CAS) and responded to the needs identified in the Government's Ten-Year Education Reform Strategy (2003). 7 30. Given the scope of education reform and the Government's commitment to a long- term strategic vision, the three-phase, 10-year programmatic approach was chosen for ensuring achievement of the Project's objectives and long-term sustainability. Project design focused on quality improvement, efficiency and financing, equity and access, and management strengthening at the general education level. Despite the various needs in the education sector, the decision to limit the first phase of the Program to general education (including both primary and secondary) by financing priority investments in three pilot districts while supporting substantial reforms at the national levels was based on three main reasons: (i) the need for critical reforms at the general education level; (ii) the need for simplicity and clarity in the design and flexibility in implementation given the weak implementation capacity of the MOE; and (iii) the need to strengthen the organization and institutions tasked to implement the proposed reforms before comprehensive reforms could be expanded at other levels of education. 31. Lessons Learned from the Previous Bank Projects. Project design incorporated lessons and experience from the Bank's first Education Reform Project in Azerbaijan (Credit No. 3220-AZ), namely a US$5 million equivalent - Learning and Innovation credit (LIL) - which was launched in 1999. The LIL piloted curriculum reform and teacher in- service training in selected pilot institutions and schools in the country. One important lesson from the above-mentioned LIL was the need for the Ministry to accept responsibility for the reform process and take ownership of project implementation. During the implementation of the LIL, the lack of involvement of key implementing units of the Ministry in the initial design of the Education Reform Project and a lack of understanding of their responsibility for project implementation led to weak initial ownership by the Ministry as well as weak project implementation. For that reason, the design of the Education Sector Development Project (ESDP) envisaged closely integrating project management into the overall operation of the Ministry of Education. Namely, direct responsibilities for the implementation of project components were assigned to relevant Ministry officials, whereas PCU's responsibilities were limited to project financial management, procurement and coordination. 32. Design of efficiency and finance interventions. While the comprehensive sector analysis carried out during project preparation made it possible to clearly diagnose the needs and priorities for the reform of the education sector in Azerbaijan, the ICR team believes that the design of the efficiency and finance reform component and related outcome indicators to be overly ambitious since the political economy was not propitious for advancing these reforms. In order to implement finance and efficiency reforms successfully, strategic decisions were to be made at different levels of the Government and by different Ministries for which political readiness was low coupled with technical capacity constraints. 33. Risk rating. The Project was rated as of Substantial Risk at the time of preparation. Project risks were adequately identified, including the one related to the implementation of finance and budgetary reforms. The latter required major changes to be instituted by the Ministry of Finance in the existing finance and human resource policies and procedures and were beyond the powers of the MOE. However, as evidenced during project implementation, mitigation measures envisaged during preparation were not sufficient to ensure successful implementation of those components that required support and resources from other levels of government. 8 2.2 Implementation 34. The Project was approved by the Board of Executive Directors on May 27, 2003 and the Credit became effective six months after Board approval. Delays in the date of signing and then in the date of effectiveness were partly the result of the presidential elections (October 15, 2003) and government approval procedures to meet the conditions of effectiveness. 35. The early years of project implementation (2004-2005) were marked by significant implementation delays due to the lack of readiness and capacity constraints of the Borrower. Based on supervision documentation, it is apparent that the initial implementation setbacks were related to the following key factors: (i) the Bank's complex procurement procedures coupled with relatively low procurement capacity and lack of readiness in the Ministry resulting in delays in the identification and procurement of technical assistance and civil works for the key activities; (ii) weak coordination with the Ministry of Finance often resulting in long delays in the approval of contracts; and (iii) lack of coordination between different departments of the Ministry of Education responsible for project implementation. 36. The Bank conducted a Mid-term Review (MTR) after two and a half years of implementation (June 2006) and concluded that while project implementation had accelerated throughout 2006, the overall Implementation Performance rating was Moderately Unsatisfactory due to the accumulated effect of initial implementation delays. Namely, the new primary education curriculum was not ready for nation-wide implementation in 2007-2008 school year and the implementation of the financing and budgeting reforms was also behind the schedule. 37. Despite the initial implementation delays, the Project made significant progress in the aftermath of the MTR as it resulted in renewed commitment to an accelerated implementation schedule. Significant progress was made in several key areas: (i) design of the curriculum reform and introduction of a new textbook policy entailing the free distribution of textbooks in core subjects; (ii) establishment of a national system of student assessment involving national testing and participation in international assessment; (iii) establishment of the EMIS and the Strategic Analysis, Planning and Personnel Units at the Ministry of Education; (iv) approval of a plan for the reorganization and staffing of the Ministry of Education; and (v) adoption of a national strategy for the professional development of teachers. By July 2007 all six triggers to move toward the next phase of the Program had been met. 38. By the end of 2007, most of the planned activities were on track, with total disbursements reaching 61 Percent of total credit proceeds. However, there was a considerable delay with the envisaged in-service teacher training activities. The initial delay with the development of the primary school curriculum had prevented the timely development and launching of the related in-service teacher training for the 9,000 teachers needed for the nation-wide implementation of the new curriculum starting for the 2008- 2009 academic year. It became evident that the time left before project closing was not sufficient to carry out the training and that about US$ 800,000 equivalent would be left undisbursed. Therefore, a one year extension was granted to support the nationwide teacher training for the implementation of the new curriculum in Grade 1. 9 39. As previously stated, due to exchange rate gains and savings from successful procurement processes, the original IDA Credit was increased by about US$2 million equivalent. A decision was made to use these funds to finance the country's participation in international assessments of student learning achievement (PIRLS, TIMSS, and PISA) as well as other activities selectively chosen to strengthen ongoing efforts consistent with the Project's Development Objective. Thus the Project was granted another six month extension to allow the Borrower additional time necessary to complete those activities. 40. Overall, the effectiveness of project implementation improved in the aftermath of the mid-term review and ESDP was able to eventually meet its goals. This was primarily due to the fact that project implementation, though carried out by a Project Coordination Unit (PCU), was well integrated into the Ministry. While the PCU was responsible for procurement, financial management, and project monitoring and evaluation, it was the heads of the relevant departments/units of the Ministry who were directly responsible for the implementation of day-to-day project activities associated within their mandate. Such implementation arrangements have gradually triggered stronger ownership for the Project and, at the same time, have contributed to building adequate reform implementation capacity at the MOE level. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 41. Project design envisioned the elaboration and implementation of a monitoring and evaluation plan to support evaluation and policy studies to monitor and assess the outcomes of the Program against a set of performance indicators. The PCU, under the guidance and supervision of the office of Deputy Minister of Education, was responsible for the design and implementation of the project monitoring plan. Largely, the Ministry was able to carefully monitor both overall reform and implementation progress. The Project Implementation Supervision Reports (ISRs) demonstrate that project indicators were monitored and updated regularly. 42. Monitoring and evaluation of the Project was undertaken through: Baseline and Evaluation Surveys. Baseline surveys for the Project were carried out in May 2005 exploring the following issues: teaching methods, school principals' attitudes, parents' attitudes, textbook availability, library resources, management review and performance audits in the pilot districts, and policy analysis and strategy development. The baseline survey was replicated in May 2007 and the comparative analysis report of baseline and follow up surveys were produced by the PCU. Progress Reports. The detailed quarterly reports were produced regularly providing information on the status of impact/outcome and output indicators set out in the project Results Framework. Selected evaluation studies. Additionally, the following evaluation studies were carried out: evaluation of School Grants Program and the pilot funding experience. 43. The ICR benefited from the data and information extracted from the above sources. 10 2.4 Safeguard and Fiduciary Compliance 44. Safeguard. The environmental category for the Project was B (Partial Assessment) as it included financing the rehabilitation and refurbishment of schools in three pilot districts (Ali Bayramli, Ismaili, and Ujar). As part of preparation, an Environmental Management Plan (EMP) was developed by the Government and published in local papers. Following the disclosure of the EMP, the MOE organized consultative meetings in Baku and in the three districts, involving key stakeholders (school directors, teachers, parents and NGOs). While both PAD and Development Credit Agreement (DCA) included a provision for rehabilitation and refurbishment of school buildings and of the National Center for Student Assessment, neither document included a provision for new construction. However, as part of the region-wide Portfolio Review carried out in December, 2008, the team realized that one construction had been carried out with Project funds in 2004 - an extension wing was built to the Ministry of Education to provide office space for the National Center for Student Assessment. The Bank had provided the no objection to the NCB bidding documents for the construction of the expansion. The expansion took place on government land and did not involve any resettlement. 45. While the expansion was fully consistent with the Project Development Objectives and neither triggered a new safeguard nor changed an existing one, the Bank's team consulted with OPCS to determine the correct way to proceed in terms of ensuring compliance with Bank procedures and consistency in project documentation. The discussion took into account the fact that a region-wide safeguard review was carried out in 2007 in which the Project was reviewed and no issues were raised. The team was advised to carry out a 3rd order restructuring with a corresponding amendment to the DCA and subsequent disclosure to the Board via Monthly Operational Summaries. The amendment to DCA was introduced on May 8, 2009 to retroactively record said change in the project description in Part D.1 (iii) of Schedule 2 to the DCA to ensure a full compliance with the Bank policies and procedures. 46. Fiduciary. Project implementation was facilitated by a Project Coordination Unit (PCU) established within the Ministry of Education. Procurement, contracting processes and financial management were the responsibility of the PCU. The Project's procurement plan was approved and closely monitored by the Ministry of Finance. During the initial years of project implementation there were significant delays in the procurement of technical assistance and civil works mostly due to the slow approval of contracts by the Ministry of Finance as well as the lack of familiarity of the PCU with the Bank's procurement procedures. However, starting toward the end of 2005, the PCU was able to build sufficient procurement capacity and the procurement processes significantly improved. During implementation, the Project's annual financial and external audit reports were presented with no significant delays, and no serious internal control observations were reported. The Project's audit reports were unqualified throughout implementation. 2.5 Post-completion Operation/Next Phase 47. The sustainability of the Project's post-completion operation is very likely both in terms of the availability of adequate financial resources to continue interventions launched under ESDP and government commitment to further advance education reforms. Given that the second phase of the APL (Second Education Sector Development Project ­ Credit No. 4394-Az) was approved in April 2008 and became effective in December 2008, adequate 11 financial resources will be available to move towards a full-scale implementation of the preparatory work carried out under the first phase of the Program. The two phases of the APL will ensure support for the implementation of the many interventions launched under the first phase, such as curriculum reform, teachers' professional development, student assessment, and education reform management capacity at the national, regional and local levels. 48. The approval of the follow-up operation is a good demonstration of the Government's continued commitment to long-term education reform agenda. Government commitment is also well evidenced by mainstreaming some of the ESDP interventions into MOE's regular operations, as described below: 49. Pre-service teacher training. The Higher Education Unit of the MOE is responsible for addressing the reform of pre-service teacher education. In pursuing pre- service teacher education reforms, the Unit plans to build on the achievements made in this area as part of ESDP. 50. Textbook Policy. The Textbook and Publishing Unit of the MOE in collaboration with the Textbook Approval Board has already taken ownership over systematic implementation of the new Textbook Policy. 51. The School Grant Program. Annual competitions for the allocation of grants to the 100 best teachers as well as 50 best schools are evidence of sustainable implementation of the ideas promoted by the School Grant Program implemented under the ESDP. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 52. The Project supported key interventions targeted at improving quality, efficiency and financing, equity and access to general education and management strengthening. 53. The Project's objectives were relevant at the time it was prepared, and continue to be relevant today. They are fully consistent with the Bank's 2006 Country Partnership Strategy (CPS) for Azerbaijan as well as the new State Program on Poverty Reduction and Sustainable Development (SPPRSD) for 2006-2015. Likewise, the Program remains consistent with the new CPS (FY 2007-2010) strategic objective of improving the quality of and access to social services. With respect to education, the CPS specifically mentions the importance of developing a modern education system and globally competitive knowledge economy. 54. The progress made under the first phase of the APL has laid a strong foundation to move the education reform to the next level as part of the next phase of the Program. 12 3.2 Achievement of Project and Program Development Objectives Achievement of Project Development Objectives: 55. The Project's achievement of its Development Objective is assessed as Moderately Satisfactory given that it was able to achieve many of its expected outcomes albeit with some delay. Significant progress has been made in several key areas, namely in the design of curriculum reform and introduction of a new textbook policy entailing the free distribution of textbooks in core subjects; establishment of a national system of student assessment involving national testing and participation in international assessment; establishment of the EMIS and the Strategic Analysis, Planning and Personnel Units at the Ministry of Education; the approval of a plan for the reorganization and staffing of the Ministry of Education; adoption of a national strategy for the professional development of teachers. 56. One challenge in assessing performance is that the Program's 12 key performance indicators listed in the main body of the PAD, including the six triggers applicable to this Project, are not fully compatible with the outcome indicators listed in the PAD's results framework (Annex 1) and in the Development Credit Agreement. Likewise, whereas the Project's ISRs only reported on key performance indicators, the PCU quarterly reports and evaluation surveys measured the outcome indicators of Annex 1 of the PAD. Therefore, in order to rate the Project's achievement of its development objective, the Bank team reviewed both sets of indicators, focusing on both the six "trigger" performance indicators, and the nine outcome indicators listed in the results framework. 57. The assessment of Project's outcome indicators in this ICR is consistent with the Borrower's own completion report provided in Annex 7. The latter report indicated that all project key performance indicators were fully met and assessed achievement of most project outcome indicators as "Satisfactory" except for the two indicators (related to efficiency and finance reforms and student to teacher ratio) that were assessed as "Moderately Satisfactory". 58. In terms of the key performance indicators, the Project was able to meet all six indicators, as discussed below: 59. Trigger 1: National curriculum framework for grades 1-11 developed, standards developed and approved for grades 1-4 by the MOE, and new syllabi for at least two grades developed and ready for implementation on a pilot or nationwide basis. Evidence to date: Achieved. The National Curriculum Framework, including content standards for general education (grades 1-11), was developed by the MOE and approved by the Cabinet of Ministers in October 2006. Subject area curricula for primary grades were approved in July, 2007. The MOE carried out the piloting of the curricula for first grade in 23 schools of Baku in the 2007-2008 academic year. The nationwide introduction of Grade 1 subject curricula was launched in the 2008-2009 academic year. 60. Trigger 2: At least 90 percent of children in grades 1-5 have free of charge textbooks in "core subjects." 13 Evidence to date: Achieved. The project evaluation survey carried out in 2007 revealed that over 95 percent of children in grades 1-5 received free of charge textbooks. The textbook survey carried out in May 2009 by Sigma (a firm hired by the PCU for this purpose) confirmed lack of disparities in access to core textbooks. 61. Trigger 3: New funding formula and mechanism prepared and implemented on an experimental basis in the three pilot districts. Evidence to date: Achieved. A per-student financing formula was developed, approved by the Cabinet of Ministers in April 2006 and implemented on an experimental basis in 59 schools in the three pilot districts in 2007. 62. Trigger 4: District rationalization plans prepared in the three pilot districts, under terms of reference satisfactory to IDA, and approved by the Government and reviewed by IDA. Evidence to date: Achieved. The district rationalization plans were developed and approved by the Ministry of Education on October 17, 2005. The rationalization plans were implemented in the pilot districts but not on a full scale. As part of the reform, 41 schools were rehabilitated and administrative measures were carried out to promote rationalization of human resources including adoption of regulations for competence-based selection of teachers and principals. The process did not result in the closure of inefficient schools. No significant efficiency gains have been reported apart from modest increase in the student- teacher ratio in the pilot districts, as reported elsewhere. 63. Trigger 5: Completion of a draft new organizational and staffing plan for the Ministry of Education (including the Institute for Education Problems). Evidence to date: Achieved. Three new departments (Monitoring and Assessment Unit, Education Management Information System Unit, Strategic Analysis and Planning Unit) were established in the MOE. Structural changes were made in the Institute for Education Problems (IEP), new regulations were approved and newly established structural divisions (Curriculum Center, and Center for Applied Education Research) were staffed through a competitive recruitment process. The organizational and staffing plan for MOE and IEP was finalized in 2007 although originally planned for 2005. 64. Trigger 6: Completion of a pilot sample-based reliable assessment of grade 4 student achievement in Mathematics and Azeri Language. Evidence to date: Achieved. The sample-based assessments in mother tongue and mathematics in grades 4 and 9 were conducted in April-May 2006. Analysis of the assessment results was made available in the spring of 2007. Results were widely discussed with the MOE and stakeholders. 65. In terms of the nine outcome indicators listed in Annex 1 of the PAD, as described below, the Project was able to fully achieve five of its nine outcome indicators and to partially achieve the reminder four outcome indicators. Most of these indicators, as pointed out below, are directly related to the six indicators discussed above. 14 66. Outcome Indicator 1: Curriculum development capacity established in the restructured Institute for Education Problems by 2006. (Related to indicator 1 above: National curriculum framework for grades 1-11 developed, standards developed and approved for grades 1-4 by the MoE; new syllabi developed at least for two grades and ready for piloting) Evidence to date: Achieved. A Curriculum Center was established within the restructured Institute for Education Problems (IEP) in December 2004. Adequate capacity, both in terms of human and physical resources, has been built in the Curriculum Center as evidenced by the fact that the Center was able to successfully develop the National Curriculum Framework and launch phased nationwide introduction of the new curriculum in Azeri schools in 2008/2009 academic year. 67. Outcome Indicator 2: Management and planning capacity of the MOE and local governments strengthened and performing their functions efficiently and effectively. (Related to indicator 5 above: Completion of draft organizational and staffing plan for the Ministry of Education, including the Institute of Educational Problems.) Evidence to date: Achieved. The activities supported in the course of the ESDP implementation have resulted in the strengthened management and planning capacity at the MOE as evidenced by the (i) introduction of planning and reporting system at MOE prompting each department to develop a three-year strategic plans; (ii) the establishment of key new units in the MOE (such as Strategic Analysis and Planning Unit (SAPU), Monitoring and Assessment Unit (MAU), Education Management Information Unit (EMISU), which are critical for improved management and planning of the system. These new units have played an important role in pursing education reforms in the country. No research based evidence is available that would evaluate management and planning capacity of the local government. Nonetheless, as part of ESDP, management trainings were provided to the rayon education department administrators. Representatives of the district education departments were also involved in the implementation of the School Grants Program. These activities have likely contributed to capacity building at the local government level as well. 68. Outcome Indicator 3: Education statistics, including learning outcomes, are readily available at the national level and pilot districts and published in the form of "Education Indicators" for policy makers and the public by 2006. (Related to indicator 6 above: Completion of a pilot sample-based assessment of grade 4 student achievement in Math and Azeri language.) Evidence to date: Achieved. Despite significant initial delays, the new EMIS was able to generate and make publicly available (through the MOE website) a school-by-school database. The first MOE statistical report was issued in 2008. Results of the examinations as well as data on student assessments are also available on the Ministry's website. 69. Outcome Indicator 4: Policy makers and education stakeholders use the results of learning outcomes and reliable statistics in their policy analysis and strategy development. Evidence to date: Achieved. ESDP produced excellent progress in terms of making available reliable school statistics and data on student learning outcomes. Both the education statistics generated by the EMIS and data on student learning outcomes based on 15 national sample based assessments, centralized graduation examinations in 9 regions of Azerbaijan and international assessment (PISA 2006) is available on the MOE's website. However, the evidence to judge the actual application of education data to inform policy making by the MOE leadership is not available. 70. Outcome Indicator 5: 25 percent of primary teachers (about 9,000) are familiar with the new curriculum, teaching methods, and assessment targets and begin to reflect those changes in their teaching practices during the 2007-2008 school year. Evidence to date: Achieved. About 18,000 primary school teachers have received training in modern teaching methodology and assessment tools required for the implementation of the new curriculum. According to the "Textbooks and Curriculum Implementation Survey" carried out in 2009, 40 percent of 1st grade teachers are satisfied with the quality of trainings received, 80 percent of the same teacher group indicate the need for more trainings. 71. Outcome Indicator 6: Present disparities in access to core textbooks (based on new curricula) in basic education grades (grades 1-9) eliminated by 2006. (Related to indicator 2 above: At least 90 percent of children in grades 1-5 have free of charge textbooks in core subjects.) Evidence to date: Achieved. The "New Textbook Policy in General Education System," entailing free distribution of textbooks in core subjects, was developed and approved in March 2006. The ESDP evaluation survey (2007) revealed that over 95 percent of children in grades 1-5 received free of charge textbooks. No disparities in access to core textbooks for basic education are reported by the "Textbooks and Curriculum Implementation Survey" carried out in May 2009. ESDP also provided supplementary reading materials to all school libraries. 72. Outcome Indicator 7: Student-teacher ratio in general education increased from 10:1 in 2002 to 12:1 in 2007 in the three pilot districts. Evidence to date: Partially achieved. As a result of rationalization activities as part of the ESDP, only moderate increase in STR was achieved based on EMIS data.2 2002 2006 Ismaili 8.9/1 10.9/1 Ujar 10.7/1 11.8/1 Ali Bayramli 10.9/1 12/1 73. Outcome Indicator 8: Efficiency and finance reforms, including district rationalization plans, prepared and implemented successfully in the three pilot districts by 2007. (Related to indicators 3 and 4 above: New funding formula and mechanism prepared and implemented on an experimental basis in three pilot districts. District rationalization plans prepared in three pilot districts and approved by the Government and IDA.) 2 According to the information provided by the Ministry of Education, Student to Teacher Ratio was 9 to 1 in the pilot districts in 2007. The difference is due to different methodology used to calculate STR. 16 Evidence to date: Partially Achieved. Under the efficiency and finance reforms, ESDP supported two key interventions in the three pilot districts: design and implementation of reforms in the financing and budgeting of education (introduction of the new financing formula and resource allocation mechanism, and the implementation of a system of school accountability) and school rationalization and improvement (school district rationalization plans; rehabilitation and provision of furniture, basic equipment and teaching aids to selected schools). These interventions were targeted at increasing the efficiencies in the use of financial, human and physical resources in the education sector. 74. Despite important progress in the development and initiation of activities under the efficiency and finance component of ESDP, implementation faced serious shortcomings due to existing legal constraints in budgeting and financing of education. At the time of ESDP preparation, there was a clear gap in the existing legal framework that constituted a real obstacle to the successful implementation of the anticipated reforms in financing and budgeting. Therefore, one of the objectives of the Project was to promote changes in budgeting and financing system in order to achieve the much needed flexibility, efficiency and local accountability. However, as explained in further detail below, throughout ESDP implementation, there was insufficient support within the relevant Government entities for adjusting the framework and removing the existing obstacles. 75. A per student financing formula was designed under the Project. The Government approved the piloting of a new funding formula for 59 schools in the three pilot sites (Shirvan city, Ismailli and Ujar districts) in early 2007. Since 2005, these pilot schools had operated their own treasury accounts and employed school accountants. However, the implementation of the new funding scheme in the pilot schools experienced serious shortcomings.. An initial evaluation of pilot experience in the 59 schools by the MOE has revealed several key weaknesses in the implementation of the new funding system. Namely, the following issues were raised: (i) although the per capita formula was used to calculate budget of the pilot schools, several cases were reported when the school budget was afterwards randomly changed (either increased or decreased) by the district finance departments without reapplying the formula; and (ii) the MOF approved the pilot school budgets not as a lump-sum amount but as a line-item, thus preventing schools to shift between line item expenditures. The latter has limited autonomy of schools to prioritize spending at their discretion. These issues stemmed from the existing legal and regulatory gaps in the sector. The Budget Law of Azerbaijan prohibits making changes to line-item budgets allocated to schools. Although the decree of the Cabinet of Ministers had exempted 59 pilot schools from this requirement in order to provide them with a certain degree of financial autonomy, from the legal perspective the law prevails over the decision of the Cabinet of Ministers. Moreover, familiarity with the objectives of the new funding system was low among finance departments and school councils of the pilot districts; this, coupled with the limited technical capacity and readiness of the school administrators to manage under the new system, contributed significantly to the weaknesses mentioned above 76. The district rationalization plans were developed, approved (Ministerial order #715 as of October 2005) and implemented in the three pilot districts. The plans envisaged closure of small and inefficient schools, rehabilitation of needy schools, and increase in Student Teacher Ratio through rationalization of human resources. As part of the reform, a total of 41 schools were rehabilitated (23 in Ismailli, 3 in Ali Bayramli and 15 in Udjar) and provided with furniture and ICT equipment. Administrative measures were also carried 17 out to promote rationalization of human resources, but no significant efficiency gains have been reported. Moderate increase in STR was achieved through the implementation of the district rationalization plans in the pilot districts, as reported above. However, as part of implementing district rationalization plans, optimization of small and inefficient schools did not take place. 77. The implementation of the two key interventions supported by the efficiency and finance component did not translate into significant efficiency gains due to the implementation shortcomings explained earlier. The Bank does not consider that the reforms were fully and/or successfully implemented. For this reason, this indicator is considered partially achieved, while the related indicators, which are defined in a narrow manner, are considered achieved. A comprehensive evaluation of the efficiency and finance reforms carried out under ESDP may provide solid basis for continuing dialogue with the Government and expanding finance reforms under the second stage of the program. 78. Outcome Indicator 9: Society at large express support for education reform by 2007. Evidence to date: Partially achieved. A survey of parents, teachers and school officials carried out in 2007 revealed that 58 percent of school principals (52 percent in baseline), 54 percent of parents (50 percent in baseline) and 53 percent of teachers (51 percent in baseline) were satisfied with the educational system, which shows a moderate increase from the baseline survey conducted in 2005. In the pilot districts, satisfaction was 4 percent to 9 percent higher among teachers, parents and school directors. However, the satisfaction of rural population with education reform decreased from 42 percent at baseline to 37 percent in follow-on survey. 3.3 Efficiency 79. Given that this is the first phase of an APL, one would not expect significant efficiency gains until the completion of the Program, particularly as the full-scale implementation of many reforms launched under this project, and results from these interventions, would most likely be measurable toward the end of the Program. For this reason, an economic analysis of the Project was not carried out for this ICR. 80. An important future economic benefit of the Project, and the Program at large, is that it is expected to contribute to laying an important foundation to promote the efficient use of the education budget, and ensuring that students benefit from quality education and acquire the necessary skills to eventually join Azerbaijan's labor market. For example, the large economic benefits of the Project's investments in curriculum implementation will only be reaped 10-15 years from now when these same students learning under the improved curriculum enter the labor market. 81. At the same time, the Bank team recognizes that the contribution of the interventions supported under the Project to make the education system more internally efficient, by piloting a new funding formula and district school network rationalization plans, was modest. Nonetheless, as a result of the implementation of the efficiency and finance component, the average student teacher ratio increased up to 11 in 3 pilot regions in 2009 as reported earlier. 18 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 82. The Project's overall outcome rating is Moderately Satisfactory given that the Project was able to achieve many but not all of its expected outcomes. The first phase of the Program has built an excellent basis for improving the education system in Azerbaijan over the Program's remaining life-span. As a result of Project interventions, management and planning capacity of the MOE and local governments was significantly strengthened. In this regard, good progress was achieved in the reorganization of the Ministry's structure and establishment of several key new units, such as Strategic Analysis, Planning and Personnel Unit (SAPU), Monitoring and Assessment Unit (MAU) and Education Management Information System Unit (EMISU). The establishment of the Education Management Information System (EMIS) and introduction of national assessments of student learning achievement have provided adequate tools for the Ministry to strengthen capacity for planning and monitoring education reforms. At the same time, trainings in management and planning techniques were provided to the rayon education department administrators. The Project produced some excellent results in the design and introduction of new curriculum, free provision of textbooks to students in core textbooks, and, on a more limited basis, in teacher training and development. At the same time, not all activities were implemented to the full scale envisioned during project design and have demonstrated mixed results. This is particularly true of the efficiency and finance component envisaged under the Project as previously explained. The component accounted for a significant share of project costs and the fact that it did not succeed in achieving the intended outcomes to the full extent results in a Moderately Satisfactory rating. 83. The Project's relevance was high both with respect to the Government's overall strategy for education and the Bank's 2006 CPS. Quality and efficiency reforms supported by the Project still remain as priorities for further development of the education system in Azerbaijan. The majority of interventions under this Project laid solid foundation for further education reform to be pursued under the second phase of the Program. 4. Assessment of Risk to Development Outcome Rating: Moderate 84. There is strong evidence that the Government remains committed to the majority of project priorities as evidenced by the launch of the follow-up operation focusing on scaling up the implementation of the new curriculum, teachers' professional development, student assessment, and education reform management capacity. 85. One area of concern that poses risk to the full achievement of the Program Development Objective is that there is yet no final strategy in place to expand pilot activities supported under the finance and efficiency component of the Project to the national level. Taking finance reform to the next level will be critical given the remaining needs to improve efficiency in the use of resources in the general education sector through increased student to teacher ratio and greater increase in the share of non-salary items in the general education budget. 19 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 86. Project preparation was supported by analytical work and financial support extended as part of the PHRD grant, which was approved in 2001 to assist the Government to prepare the ESDP. The main strength of project design and preparation lay in the ability to accurately identify and prioritize the reform needs of the country in the framework of the Government's Education Reform Program of 1999 and the draft Ten Year Education Reform Strategy. Given the Government's long-term agenda and commitment for education reforms, the choice of the Adaptable Program Loan (APL) instrument was appropriate for ensuring achievement of the operation's development objectives and sustainability. Quality at entry was also strengthened through intensive consultation with stakeholders as part of the participatory stakeholder workshops undertaken throughout project preparation (2002- 2003) as well as through close dialogue and partnership with the existing donors. 87. The quality at entry was reviewed by a World Bank internal Quality Enhancement Review (QER) Panel in September, 2002. The QER panel highlighted positive aspects of the project design, namely the relevance of the project design to the client's needs and preferences, the Bank's experiences with the first education project (LIL) in Azerbaijan and the political realities of the country that would affect implementation possibilities. The QER panel also stressed the areas which needed improvements. These were: to simplify and constrain the project design considering weak implementation capacity of the Ministry and to build on the successes under the LIL; to limit the scope of the project to components that the MOE had the power to implement and that were proper functions of the central ministry since the sector had significant problems that ideally should but could not be tackled because they were outside of the control of the ministry. The one shortcoming in the Bank's performance during the lending phase is that implementation capacity was overestimated. As a result there were significant implementation delays in the early stages of implementation. Moreover, the team had underestimated lack of support for advancing efficiency and finance reforms under the Project. Mitigation measures envisaged during preparation were not sufficient to ensure successful implementation of those components that required support and resources from different levels of government apart from the line Ministry. (b) Quality of Supervision Rating: Satisfactory 88. The Bank invested ­ in close cooperation with the Government ­ an enormous time and effort into very close supervision of project activities in order to proactively apply reality checks to the effective implementation of the Project and address issues on a timely basis. 89. Given the slow implementation progress during the earlier implementation years, the final achievements of the Project are impressive. Throughout implementation, the Bank was proactive in supporting the main counterparts in implementing the Project. Bi-annual 20 supervision missions (a total of 13) and a mid-term review helped the counterparts with the implementation, and problem areas were identified and addressed promptly, as well as recorded in Aide Memoires and Implementation Status and Results (ISRs) Reports. Given the initial procurement problems resulting from low capacity, as described earlier, the Bank's team was very proactive in identifying the bottle-necks and addressing issues as they arose. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 90. The Bank's performance during project preparation and supervision is rated Moderately Satisfactory as discussed above. 21 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 91. The Government remained committed to project goals throughout the entire period. The Government's Ten-Year Education Reform Strategy laid a strong foundation for the project design. A Special Working group established under the MOE played an effective role at the project preparation stage. 92. The performance of the Government varied between the earlier and later stages of project implementation. There were shortcomings in the Government's performance in the early stages (2003-late 2005) related to micro-management and heavy bureaucratic procedures. Initial slow disbursements and serious implementation delays resulted from the extensive and bureaucratic requirements for most of the procurement processes that involved required approvals by the Ministry of Finance and placed an unnecessary burden on the PCU staff during the initial years. Following the MTR, the Government's performance improved substantially and government officials worked closely with the Bank's team on a continual basis and were proactive in resolving the start-up delays. The Government demonstrated its full commitment by accelerating project implementation and producing the desired results. At the same time, lack of support for the financing and efficiency reforms meant that this part of the Project did not achieve its full potential. (b) Implementing Agency or Agencies Performance Rating: Satisfactory 93. One of the strong aspects of the Project was that implementation was well integrated into the Ministry of Education since the responsibility for the execution of project activities under each and every component and subcomponent was clearly assigned to the managers heading organizational units within the Ministry and not to the PCU. This feature has strongly contributed to the strong ownership and commitment towards ESDP. The high level of commitment of all MOE departments to the overall goals of the Government's educational reform program and to the implementation of the Project made it possible to largely achieve the ESDP goals despite serious initial delays. 94. The PCU team had a high degree of commitment to achieving the development objectives and it had adequate skills for fulfilling this commitment. The several staffing constraints of the PCU that arose at different points in the course of ESDP implementation were effectively addressed. The PCU managed the Project well and provided effective communication between the Bank and the MES staff responsible for the project components. All representatives of MES with whom the ICR team consulted expressed great appreciation of the hard work and professionalism of the implementing agency and its team. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 95. Overall, this ICR's assessment of the Borrower performance is rated as Moderately Satisfactory. 22 6. Lessons Learned 96. Reflecting on the overall Project, the key lessons that have been learned and/or reinforced during implementation are as follows: The choice of APL instrument was appropriate given the Government's long-term vision and comprehensive reform agenda reflected in the Ten-Year Education Reform Strategy. The APL provided an overall framework for the achievement of the program's development objectives over time and ensuring continuity and sustainability of sector policies. Given the complexity and range of needed reforms, carrying out systematic sector assessments during project preparation is essential and contributes to better project design. Throughout implementation, the MoES carried out studies in the following key areas: national standards and curriculum reform; teacher education; education economics and finance; education management; education research and policy; EMIS and school mapping; information technology; instructional technology; monitoring, evaluation and assessment; and textbook development. Integrating project management into the overall operation of the line Ministry triggers greater ownership and responsibility for the reform process. ESDP was designed as such that while the PCU was responsible for procurement, financial management, and project monitoring and evaluation, it was the heads of the relevant departments/units of the Ministry who were directly responsible for the implementation of project activities associated within their mandate. Such implementation arrangements gradually triggered stronger ownership for the Project and at the same time contributed to building adequate reform implementation capacity at the MOE level. The implementation of comprehensive finance reforms involving changes in the structure and management of the system requires political support as well as strong leadership of the respective line Ministry in order to stimulate the needed inter- ministerial collaboration and sufficient support for the change at all levels of the Government. This factor coupled with technical capacity constraints prevented scaling up efficiency and finance reforms under the ESDP. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 97. The Borrower's completion report was provided by the Ministry of Education and was incorporated in annex 7. The assessment of the ESDP implementation progress and achievements presented in the Borrower's report is consistent with the findings of this ICR. Comments on the ICR were received from the Cabinet of Ministers, including feedback from the Ministry of Education, Ministry of Economic Development and Ministry of Finance. Government comments have been reflected in the ICR. An unofficial translation of all Government comments is enclosed in Annex 7. (b) Co-financiers 23 98. As part of this ICR, the Bank team collected feedback from OSI-Azerbaijan, which was a project co-financier. The comments received from OSI are reflected in annex 8 of this ICR. (c) Other partners and stakeholders (e.g. NGOs/private sector/civil society) 99. Annex 6 includes comments from a limited set of stakeholders interviewed by phone about the impact of the Project. 24 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions) 1. QUALITY AND RELEVANCE OF GENERAL 7.05 4.97 0.70 EDUCATION 2. EFFICIENCY AND FINANCE 8.86 12.77 144.13 REFORMS 3. UPGRADING SCHOOLING QUALITY IN LESS 0.87 0.80 91.95 ADVANTAGED RAYONS: SCHOOL GRANT PROGRAM 4. MANAGEMENT, PLANNING 2.34 2.58 110.26 AND MONITORING CAPACITY 5. PROJECT COORDINATION 1.87 1.49 79.68 Total Baseline Cost 20.99 22.61 107.76 Physical Contingencies 0.00 0.00 0.00 Price Contingencies 0.00 0.00 0.00 Total Project Costs 20.99 22.61 Front-end fee PPF 0.00 0.00 .00 Front-end fee IBRD 0.00 0.00 .00 Total Financing Required 20.99 22.613 (b) Financing Appraisal Actual/Latest Type of Co- Estimate Estimate Percentage of Source of Funds financing (USD (USD Appraisal millions) millions) Borrower 2.80 2.92 104.29 International Development 18.00 19.69 109.38 Association (IDA) 117.70 SOROS Foundation 0.19 0.22 Total Project Costs 20.99 22.83 108.76 3 Due to exchange rate differentials, the total Credit amount increased by about US$2 million equivalent. 25 Annex 2. Outputs by Component Component 1: Quality and Relevance of General Education Curriculum Reform By Project completion, this component was considered Satisfactory as it had achieved most of its outputs. The National Curriculum Framework, including content standards, has been developed and approved in 2006. The new curriculum is outcomes oriented and student centered, designed to promote active teaching and learning environment as opposed to rote learning and mechanical transfer of knowledge. The National Curriculum provided a framework for the development of subject syllabi for primary grades (1-6), which were adopted in 2007. The new curriculum was successfully piloted in grade 1 in 2007-2008 academic year in the selected schools. The nationwide introduction of the new curriculum in grade 1 was launched in 2008-2009 academic year. Teacher Development The new policy document, namely the Strategy for Teacher Professional Development, was approved by the government. New subject "Foundations of Education", encompassing pedagogy, psychology, teaching methods and skills development through interactive learning methodology, was introduced in pre-service teacher training of primary school teachers. Introduction of new syllabi for pre-service training of primary school teachers was launched on a pilot basis in the State Pedagogical University, Azerbaijan Teachers Institute and Ganja State University. The staff of these institutions was trained to acquire knowledge and skills required for the implementation of new pre-service training curricula. The development of in-service teacher training programs based on new syllabi was supported by the Curriculum Center. To support these trainings, all affiliates of Azerbaijan Teachers Institute were provided with ICT, furniture and other equipment. In addition, two branches (Mingachevir and Jalilabad) were renovated. Approximately 18000 primary school teachers ware provided with in-service teacher trainings under this subcomponent. Textbooks and Reading Materials The achievements of this subcomponent have been significant. The results indicate considerable successes in improving access to textbooks and the availability of supplementary teaching and learning materials in schools. The end of ESDP evaluation survey conducted in 2007 revealed that 97 percent of students in grades 1-5 have free of charge textbooks. Survey results also indicate that by 2007 the MOE has improved provision of free of charge textbooks in other grades of general education as well. As part of the ESDP, all schools were provided with child encyclopedia (18 titles) and maps (19 titles). The subcomponent also supported the development of the new textbook policy. The new document "Textbook Policy in General Education System was developed and approved as part of the ESDP. Independent Textbook Evaluation Board was established in order to 26 ensure transparency in textbook development and procurement. Based on the new policy document, all textbooks are required to include teacher's guide and student's workbook. Textbook policy development under the project was supported by the OSI-Azerbaijan through the provision of textbook consultancy and a number of trainings, seminars and roundtables for policy makers, textbook publishers, evaluators and authors. Component 2: Efficiency and Finance Reforms By Project completion, this component was able to only partially achieve its original outputs. This component was rated as Moderately Satisfactory. Finance and Budgetary Reforms Although important steps were made in designing and implementing on a pilot basis the efficiency and finance reforms under this subcomponent, the implementation process faced significant shortcomings and the government was not ready to scale up these reforms as part of the phase one of the program. The government approved the piloting of a new funding formula for 59 schools in the three pilot sites in early 2007. The reform involved an introduction of per-capita financing and the lump-sum budgets for the pilot schools. Since 2005, these pilot schools had operated their own treasury accounts and employed school accountants. However, the implementation of the new funding scheme in the pilot schools experienced serious shortcomings as it relates to granting adequate level of school financial autonomy to schools, including for reallocation of funds from salary to non-salary budget lines. The difficulties, as explained in details earlier, were related both to legal gaps and lack of buy- in and adequate support in this part of the reforms by the Ministry of Finance at the time of implementation. However, the comprehensive evaluation of the efficiency and finance reforms carried out under ESDP is envisaged under the phase two of the program and may provide solid basis for continuing dialogue with the government and expanding finance reforms under the second stage of the program. Rationalization and School Improvement The district rationalization plans were developed, approved (Ministerial order #715 as of October 2005) and implemented in the three pilot districts. The plans envisaged closure of small and inefficient schools, rehabilitation of needy schools, increase in STR through rationalization of human resources. As part of the reform, Total of 41 (25 in 2006 and 16 in 2007) schools were rehabilitated (23 in Ismailli, 3 in Ali Bayramli and 15 in Udjar) and provided with furniture and ICT equipment. Administrative measures were also carried out to promote rationalization of human resources, but no significant efficiency gains have been reported. Moderate increase in STR was achieved through the implementation of the district rationalization plans in the pilot districts, as reported elsewhere. The activities under this subcomponent were also supported by OSI-Azerbaijan, as a result of which trainings were provided to school teachers in the pilot district in the application of Information Communication Technologies (ICT). Moreover, framework on ICT 27 application policy was developed and educational multimedia resources were piloted and disseminated in secondary schools. Component 3: Upgrading Schooling in Less Advantage Rayons: School Grant Program The component was rated Satisfactory as it had achieved its intended outputs. The objective of the School Grant Program (SGP) was to supplement the limited resources of schools and to promote equity and access to quality education in order to improve enrollment and attendance rates. Implementation of SGP was also accompanied with comprehensive training programs in partnership with the Open Society Institute of Azerbaijan. OSI facilitated networking among school communities from non-pilot and pilot regions and supported the development of manual on school proposal design, implementation, monitoring and evaluation. Total of 173 schools received grants throughout the three phases of SGP implementation. In addition, given the success of this component in terms of making non-salary resources accessible to schools, the MOE decided to expand this activity country wide and administer school grants program annually. By the Presidential decree dated Sept. 2007, grants were awarded to the best 50 schools to improve the quality of education. Component 4: Management, Planning and Monitoring Capacity The component was rated Satisfactory, as it had achieved most of its intended outputs. The component supported carrying out the national assessment of Mathematics and Azeri language in the 4th and 9th grades, and school-leaving graduation examinations at the end of the 11th grade, developing assessment standards for the new curricula to be used in the first four grades, elaborating a Concept Paper on the National System of Student Assessment, participating in PISA 2006. The National Center for Student Assessment was established to become a national agency for the evaluation of quality of education. Furthermore, the APL 1 has helped the Ministry of Education to establish a new EMIS Department. The department has already finalized the development of School Mapping, Human Resources, Financial, and Archive modules. In addition, APL 1 supported the MOE in drafting a new organizational and staffing plan, including job descriptions and job requirements. Component 5: Project Coordination and Monitoring By project completion, the achievements of this component were considered Satisfactory. The management of ESDP was streamlined into the overall operations of the MOE since project components were managed by the respective MOE officials (department heads). The PCU was fully incorporated in the MOE and successfully carried out project coordination, procurement, financial management, monitoring and reporting. The following table provides detailed description of outputs by components. 28 Project Development Objective: To strengthen capacity to plan, manage and monitor the reform program, and initiate the priority investments in selected districts. 1. Quality and Relevance of General Education. Curriculum Reform New Structure and operational Significant structural changes took place in IEP throughout roles of the Institute on Education 2005 including approval of new regulations and recruitment Problems (IEP) reviewed, revised of the staff on a competitive basis. Two new units, Center and streamlined by 2004. for Applied Education Research (CAER) and Curriculum Center were established in IEP and are fully functional. The organization and staffing plan of IEP was completed and approved by MOE in December 2004. The Curriculum Development Curriculum Center was established within IEP in December Department (CD) established, 2004 and is now fully operational. staffed and functioning in the restructured IEP by March 2004. National standards developed and Despite significant initial delays, National Curriculum adopted by the GOA by 2005. Framework for general education (grades 1-11) was developed by the MOE and approved by the Cabinet of Ministers in October, 2006. Syllabi for grades 1-4 revised and Subject curricula for primary grades (1-4) were approved in adopted by the MOE by July July, 2007. 2006. Syllabi for grades 5-11 prepared The draft syllabi for grades 5-11 have been finalized. and adopted by the MOE by July 2007. Syllabi for integrative subjects Subject curricula of primary grades approved on July 23, (e.g. science, social sciences) 2007 included curriculum for "life skills", which is newly prepared and adopted by the introduced integrated subject. MOE by July 2007. Syllabi for grade 1 Despite the delays, the subject curricula were piloted in piloted/implemented during the grade 1 in 23 schools of Baku in the 2007-2008 academic 2007-2008 school year. year. Nationwide introduction of Grade 1 subject curricula was launched in the 2008-2009 academic year. Teacher Development Teacher education strategy The Concept and Strategy for Teacher Professional prepared by 2005. Development was developed and widely discussed and consulted with the public. The document was adopted by the Cabinet of Ministers in June 2007. Primary teacher education Outcomes-based curriculum framework as well as subject curricula revised by 2007. syllabi for pre-service training of primary school teachers was developed and approved by the MOE in 2008. About 1,000 teacher education About 1106 Teacher education faculty from the State faculty trained in new teacher Pedagogical University, Azerbaijan Teachers Institute and development strategy, curriculum Gannja State University were trained to equip them with and new teaching methods by skills and knowledge on the implementation of new pre- 2007. service training curricula. Curricula and main textbook for Completed the course on the "Foundation of Education" developed by 2007. Around 100 teacher trainers 100 instructors at Pedagogic Universities and Colleges in trained in the delivery of the new Baku and Genja, trained by UNICEF, have started to teach course on the Foundation of their first-year students in primary education teaching 29 Education by September 2007. program in respective institutions from 2008-2009 academic year based on the newly developed subject curricula. Fourteen in-service teacher All fourteen in-service teacher training institutions received training institutions (ATI and 10 equipment. affiliates, Genje, Baku and Nakchevan) strengthened through the provision of equipment and learning materials by 2006. Four selected teacher training Only two affiliates of Azerbaijan TTIs (Jalilabad and institutes (TTIs) rehabilitated and Mingechevir) were rehabilitated and refurbished due to the refurbished by 2006. lack of funds. About 300 teacher trainers in Teacher trainers from TTIs have been involved in all school TTIs acquire the capacity to curriculum trainings provided under the project. deliver new training modules by 2006. About 800 teacher trainers for The training of the 320 Grade 1 master trainers was primary grade curricula trained in conducted by the Curriculum Center in 2007. new curricula and teaching methods by May 2007. About 9, 000 primary grade About 18,000 primary school teachers were trained in the teachers trained in new curricula delivery of new curricula in February 2008. and teaching methods by September 2007. Around 100% of trained teachers For the time of this ICR, no data is available to evaluate use new curriculum and teaching achievement of this indicator. The observational studies methods in their classroom during with regard to observation of teachers' behavior in the the 2007-2008 school year. classrooms are planned in the second phase of the program. Textbooks and Reading Materials All general school libraries (4, All general school libraries were provided with reading 513) provided the initial set of materials (Children's Encyclopedia set including 846 library reading materials (e.g. thousand copies of books of 18 titles) early 2005. Children's Encyclopedias) by Throughout 2007 17 different types of maps were 2005; and supplementary reading distributed to schools. materials provided by 2006. Percentage of teachers and The evaluation survey of 2007 revealed a decline in the use students using library resources to of library resources from 21% of teachers in 2005 (baseline) stimulate students' reading habits to 15% of teachers in 2007 and from 34% of the observation and to support core curricula to students in 2005 to 24% of students in 2007. Such decline increase independent thinking may be in part attributed to a 5% drop in the number of and research skills show school reporting having libraries and a plunge in the number substantial increase between 2004 of school reading halls. However, on a more positive side, and 2007. in pilot districts library use increased from the 17% of sampled teachers in 2005 to 27% in 2007 and from 23% of student in 2005 to 31% of students in 2007. A new textbook development The new "Textbook Policy in General Education System" concept (including policy and was developed and finalized through wide consultations standards) for the development with the stakeholders. The document was adopted by the and provision of Government- Cabinet of Ministers in March 2006. In addition, the MOE funded textbooks and reading approved the regulations of the Textbook Evaluation Board. materials prepared and adopted by 2006. 30 2. Efficiency and Finance Reforms Financing and Budgetary Reforms Education finance system Per capita financing formula was developed and approved reviewed and a new funding by the Cabinet of Ministers in May 2007 following which formula developed to allocate the new funding scheme was implemented on an public resources and experimental basis in 59 schools in the 3 pilot districts (20 implemented in the three districts schools in Ujar, 14 schools in Ali Bayramli, and 25 schools by 2006. in Ismailli) in accordance to the Decree #36 of the Cabinet of Ministers regarding the "Implementation of Finance and Budgeting Reforms Sub-component of ESDP" dated Febrary 22, 2007. The 59 schools were provided with treasury accounts and have been provided with accountants. A school accountability The implementation of this part of the ESDP experienced mechanism for the decentralized most difficulties and respectively this part of the reform is management and financing least advanced. system established and evaluated in the three pilot districts by Initial evaluation of pilot experience in the 59 schools by 2006. the MOE has revealed several key weaknesses in the implementation of the new funding system. Namely, the following issues were raised: (i) although per capita formula was used to calculate budget of the pilot schools, several occasions of interference were reported with the budget by the district finance departments; (ii) MoF approved the pilot school budgets not as a lump-sum amount but as a line-item budget therefore limiting autonomy of schools to reallocate between the line items and prioritize spending at their discretion; (iii) both the awareness and support for the objectives of the new funding system was low among finance departments and school councils of the pilot districts. This is further exacerbated by the fact that there is no permission from MoF to shift budget across line items. Difficulties are related to legal and regulatory gaps. The Budget Law of Azerbaijan prohibits making changes to line-item budgets allocated to schools. Although the decree of the Cabinet of Ministers have exempted 59 pilot schools from this requirement in order to provide them with certain degree of financial autonomy, from the legal perspective the law prevails over the decision of the Cabinet of Ministers. Human resource strategy to This activity was shifted under the Management and improve efficiency and attract Planning Capacity Sub-component as documented above. and sustain qualified teachers, school directors and teacher trainers prepared by the MOE by 2006. Rationalization and School Improvement Standards for school construction The draft on "Designing guidelines for effective learning and furniture prepared by 2007. environment in Azerbaijan" was prepared by the national and international consultants. Modern standards and norms for school construction are part of the guidelines providing opportunities for efficient learning premises. A nationwide school mapping for School mapping activity is implemented under the EMIS as general education completed by documented above. 2006. District rationalization plans The district rationalization plans for the 3 pilot prepared in the three pilot districts were approved by the Ministerial order districts by September 2007. #715 (October 2005). The plans envisaged closure 31 About 50 schools rehabilitated of small and inefficient schools, rehabilitation of and refurbished by 2007; student- needy schools, increase in STR through teacher ratio increased from 10:1 rationalization of human resources. However, the in 2003 to 12:1 in 2006. plans were implemented on a limited scale only. Although certain administrative activities were carried out to promote rationalization of human resources, no significant efficiency gains have been reported. No school closures took place since provision of school buses was not envisaged under the project. Total of 41 (25 in 2006 and 16 in 2007) schools were rehabilitated (23 in Ismailli, 3 in Ali Bayramli and 15 in Udjar) due to the lack of funds resulting from escalating construction costs in Azerbaijan. These schools were provided with furniture and ICT equipment. Moderate increase in STR was achieved based on EMIS data between 2002 and 2006: from 8.9/1 in Ismaili to 10.9/1; from 10.7/1 in Ujar to 11.8/; and from 10.9/1 in Ali Bayramli to 12/1; National ICT strategy developed The National ICT Strategy for 2003-2012 was approved by by 2004 and evaluation of pilot the Presidential decree in February, 2003. The concept was ICT carried out by 2007. elaborated in August 2004. ICT curriculum reviewed and Subject curricula of primary grades approve on July 23, revised by July 2007. 2007 included ICT from grade 1, which is the new policy in Azerbaijan primary education. 15 selected schools in the three All 41 schools under the project were provided with pilot districts equipped with computers and a/v systems. In addition, the Government computers and a/v systems by launched country-wide effort to provide all schools with 2006. computers. 70 teachers are trained in the use The training of primary school teachers (18000 teachers0 on of ICT in the school curriculum. new curriculum included ICT component. In addition, 30 teachers in the pilot districts; country-wised training has been undertaken on application 40 teachers in the 20 pilot schools of ICT in teaching under the State ICT program. supported under the Education LIL. 3. Upgrading Schooling in Less Advantage Rayons: School Grant Program 150 schools in the three pilot The School Grants Program was implemented in three districts apply for school grant phases and 141 schools applied for school grants (out of the based on the specified criteria by actual 147 schools in the pilot districts). The remaining 6 2007. schools were very small and lacked the human capacity and adequate physical environment to prepare and implement grant proposals. 150 school grants awarded by Exceeded. Total of 173 schools received grants throughout 2007. the three phases: Phase 1: 46 schools (26 in Ismaili, 4 in Ali Bayramli and 16 in Ujar); Phase 2: 73 schools (44 in Ismaili, 9 in Ali Bayramli and 20 in Ujar; Phase 3: 54 schools (100 in Ismaili, 20 in Ali Bayramli and 53 in Ujar). In addition, given the success of this component in terms of making non-salary resources accessible to schools, the MOE decided to expand this activity country wide and administer school grants program annually. By the 32 Presidential decree dated Sept. 2007, grants were awarded to the best 50 schools to improve the quality of education. 4. Management, Planning and Monitoring Capacity Establishment of New System of Student Assessment The National Center for Student Student Assessment Unit (SAU) was established within Assessment established, staffed, MOE in February, 2004. The Center was fully staffed and and functioning by March 2004. equipped. Capacity building of the new unit was supported though international consultant from CITO. SAU staff members also had opportunity to receive the necessary trainings in Netherlands. The first pilot National Student The first national assessments of Mathematics and Azeri Achievement Test for grades 4 Language for grades 4 and 9 were carried out in April ­May and 9 students for selected 2006. The analysis of results were made available in 2007 subjects prepared and and discussed with the Ministry and stakeholders. The implemented by May 2006; and Center also implemented the centralized Secondary results analyzed and disseminated Graduation examination in 9 regions of Azerbaijan in June by September 2007. 2008 and made the results of the examination available through the Ministry's website. Training on measurement and Several efforts were made to build assessment capacity in assessment (including school- schools: Trainings in school-based and national based assessment) provided for assessments was carried out in January 2006 and Teacher's 2,000 teachers and school guides for school-based assessment were published and directors by 2007. distributed to schools in 2006. Training of schools in national sample-based assessments in pilot districts was carried out in April 2006. The seminar on national assessment and examinations was carried out in 2006. Trainings on school-based assessment were provided for 1,000 teachers from teacher training institutions and pilot district schools (Ali Bayramli, Ujar and Ismaili) in May 2007. Conference on school based assessment was also held in 2007. Azerbaijan participated in the For the first time Azerbaijan participated in PISA in 2006. first international student The results were discussed and communicated to assessment (e.g. OECD PISA) by stakeholders in 2007. 2007. Establishment of EMIS EMIS Unit established, staffed EMIS unit was established in the MOE in 2004 and is now and functioning by 2004 in the fully operational. Trainings in the concept of EMIS and data central MOE; staff trained in the requirements for school mapping/human resources were three selected districts, Sumgait provided to 14 MOE staff and 6 staff in the three pilot City and Baku City by March districts. The necessary equipment and software was 2004. provided to Central EMIS station and 16 pilot EMIS stations. Four MOE staff and 16 district Both MOE staff and district staff received adequate staff responsible for EMIS trained training. in the techniques of data collection, simulation, analysis and reporting by 2005. EMIS developed and new school Four key EMIS modules were developed: School Mapping survey instruments designed in Subsystem (SMS), Human Resources Subsystem (HRS), consultation with different Electronic Document Archives Subsystem (EDAS), and stakeholders by 2004. Finance Subsystem (FS). Survey data for SMS in the pilot EMIS stations has been collected using the questionnaires. At a later stage, a shift was made to electronically collect the data to expand the scope of data collected through EMIS. Starting from 2008/2009 school year all district education department are able to submit school data in 33 electronic format to the EMIS unit. SMS is able to generate national date on the number of schools, students, teachers and student to teacher ratios. HRM sub-system module was installed in the MOE and districts in July 2006 and data on teachers' profile entered. The EDAS has also been installed and baseline documents collected. Less progress was made on the FS limited to preparatory work for training of accountants in the financially independent pilot schools. First annual Education Statistics The EMIS unit has made publicly available (on the MES Report published by 2007. website) the school-by-school data for years 2005/06 and 2006/07 contained in the EMIS database. The first education statistics report was issued in 2008. Management and Policy Development Management review and Structural changes were introduced both in the MOE and performance audit of MOE and IEP. IEP was fully restructured, new regulations were IEP completed by March 2005. approved and staff recruited on a competitive basis. New departments (SAU, EMISU, and SAPU) have been established in the MOE, which are fully functional and have played an important role in pursuing the reforms under the project. The organization and staffing Plan for MOE and IEP was approved in June 2007. Policy Analysis and Planning Strategic Analysis and Planning Unit (SAPU) was Unit (PAPU) in the MOE established in 2004 and fully staffed on competitive basis. established, staffed and With the support of technical assistance extended as part of functioning by March 2004. the project, the strategic planning for three years for each MOE department was prepared and implementation launched in 2006. Development of strategic plans for all 59 pilot schools was launched in 2007. SAPU also developed training materials in strategy planning, decentralization and school accountability for school principals and district education departments. Five staff responsible for PAPU Two staff from SAPU received training abroad in education have been determined and trained policy and analysis, management techniques, education in management and planning planning and decentralization. In addition, SAPU staff techniques. received several trainings locally in the course of project implementation. 166 rayon education department Although with a year delay, trainings for 145 rayon administrators trained in education department administrators took place in January management and planning 2006. New policy papers on decentralization and techniques by 2005. accountability have also been developed. 10 Project component 9 project sub-component coordinators and 15-20 senior and coordinators and 20 other senior mid-level managers from MOE were trained in management and mid-level managers from and training techniques in April-October 2005. MOE trained in management and planning techniques by 2006. 150 school directors (from pilot Initial trainings in education management and strategic districts) trained in school-based planning were delivered to 147 school principals (total management, financing, and number of schools in the pilot districts) in January 2006. In planning techniques by addition, (i) a 3 day-seminar in pilot implementation of September 2005. school accountability system for selected schools principals (50) involved in the implementation of decentralization and school accountability system was implemented in 2007 and (ii) a two-day workshop on performance-based management was delivered for SAPU staff, ESDP sub-component coordinators, officials of MOE and school principals (147 people) from pilot rayons in May 2007. 34 A competence-based teacher and Regulations for competence-based selection of teachers and director appointment policy principals were developed and endorsed. School principals designed and adopted by MOE by were selected competitively based on these regulations. 2006. However, recruitment of teachers competitively is yet to be enforced. 5. Project Management and Evaluation PCU is fully staffed, equipped PCU was fully staffed and functional. and operational by September 2003. PCU staff complete relevant staff In the course of the project implementation, PCU staff development in project received relevant trainings and participated in study tours. management, monitoring and International training programs were attended by the PCU evaluation, procurement, and Procurement Manager and the Chief Accountant. financial management by January 2004. Log Frame based Monitoring and After initial delays, this activity was completed in May Evaluation system established for 2005. The PCU was able to carry out the following implementation of progress activities as part of the monitoring and evaluation: review; Reporting System is installed and operational by Monitoring and Evaluation Plan for the project was January 2004. developed with the help of the international consultant; The detailed M & reports on project status with regard to the impact and output indicators was produced using this framework; Baseline and Evaluation surveys were conducted in 2005 and 2007. Quarterly progress reports were produced throughout the course of project implementation. Detailed M & E Report was produced on School Grants Program; Comparative analysis was conducted on treasury account schools vs. schools receiving funds directly from Rayon Education Departments; Project financial goods, works PCU implemented procurement in line with the and services procured without implementation schedule for phase one of the program and delays in accordance with World in compliance with the Bank procurement procedures. Bank procurement guidelines. PCU prepares quarterly progress Quarterly progress reports were regularly furnished to the reports, and submits them to IDA Bank. for review on time, starting PYI end of the project. Baseline survey commissioned Baseline survey was commissioned by the PCU in 2005. and completed. Education Reform Strategy Progress for this activity has been documented under the updated by 2007. component "Management, Planning and Monitoring Capacity". PCU has sufficient capacity in PCU has built adequate capacity and has successfully terms of materials and human supported the Ministry in the preparation of the phase two resources to adequately assist of the program. MOE in the preparation and implementation of APL II. Communication Strategy The Plan for Public Awareness Campaign has been 35 (including project related developed and agreed with the Bank. Brochures on ESDP information published in the activities were developed, published and distributed to Ministry web page, national relevant stakeholders. media and local media) designed by December 2003 and implemented by September 2007. 80% of the population in the pilot The comparison of the 2005 baseline and 2007 evaluation districts are exposed to education surveys revealed that percentage of general population reforms and innovations (school principals, teachers and parents) of the pilot districts supported under the project by with an overall understanding of education reforms 2005. supported under ESDP increased from 49% to 71%. 36 Annex 3. Economic and Financial Analysis Not applicable. 37 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Education Economist/Task Team Halil Dundar ECSHD Leader Lead Economist, Program Team Michael Mills ECSHD Leader for Azerbaijan Antonio Lim Operations Officer ECSHD Kamil Malikov Operations Officer ECSHD Carmen Laurente Program Assistant ECSHD Maureen McLaughlin Education Sector Manager ECSHD Junko Funahashi Senior Council LEGEC Hannah Koilpillai Finance Officer LOAG1 Ida Muhoho Financial Management Specialist ECSPS Helen Shahriari Senior Social Scientist ECSHD Procurement and Implementation Leonardo Concepcion ECSHD Specialist Former Education Sector James Socknat Manager/Education Consultant Education Specialist/QER Chair and Sue Berryman Peer Reviewer Lead Education Specialist, Peer Ernesto Cuadra HDNED Reviewer Project Implementation Specialist, Sverirr Sigurdsson Consultant, Peer Reviewer Supervision/ICR Elvira Anadolu Health Specialist ECSHD Olena Fadyeyeva Operations Officer SECPO Carmen F. Laurente Senior Program Assistant ECSHD Caroline Mascarell Sr Social Protection Specialis ECSHD Juan Manuel Moreno Sr Education Spec. ECSHD Olmedilla Karina Mostipan Senior Procurement Specialist ECSPS Turgut Mustafayev E T Consultant ECSHD Karl Skansing Consultant ECSPS Lars M. Sondergaard Economist ECSHD Rosita Maria Van Meel Sr Education Spec. ECSHD Operations Analyst/ICR Team Nino Kutateladze ECSHD Leader 38 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY02 18 80.58 FY03 55 263.25 Total: 73 343.83 Supervision/ICR FY02 0.00 FY03 4 14.70 FY04 33 115.08 FY05 28 99.98 FY06 35 125.21 FY07 29 111.55 FY08 18 82.18 FY09 28 63.92 Total: 192 612.62 39 Annex 5. Beneficiary Survey Results Not Applicable. 40 Annex 6. Stakeholder Workshop Report and Results To solicit comments and assessments around the key interventions implemented as part of ESDP, the Bank education team organized phone interviews with six project stakeholders (teachers, school principals, head of district education department). Main findings and observations are summarized below: Curriculum Development Teachers and principals provided positive assessment of the new curriculum framework developed as part of ESDP. The new curriculum was thought to provide more freedom to teachers in selecting teaching materials and resources as well as better opportunities for increasing student motivation: New curriculum is implemented in grades 1 and 2. It allows the teachers to choose the teaching method they prefer, encourages them to use materials from the Internet and other resources in addition to textbooks. It also prompts students to search for information, which enables them to grasp the materials better and more deeply. Teacher Development The respondents indicated that workshops and teacher professional development opportunities offered as part of ESDP have equipped teachers with new knowledge and skills to improve the teaching and learning process. However, some concerns were raised about the duration of the training sessions. I have regularly taken part in the workshops organized by the Ministry of Education under the World Bank supported project. These workshops have helped me to learn and apply new teaching methodologies. Thanks to these workshops, I learned how to organize my classes more effectively and integrate new approaches. Around 90% of our teachers have taken part in in-service teacher training programs. The results are positive. In 2006, 3 of the 15 students enrolled in grade 11 were able to receive high scores and enter good universities. Since the project implementation 60-65 percent of our graduates enter universities. Our School also became one of the winners in best school competition in 2008. Many of our teachers have been trained in 2008-2009. Also, out primary teachers were trained in the implementation of new curriculum. The trainings have contributed to teachers' development. Unfortunately, these trainings are a bit short (one week long). 41 Textbooks and Learning Materials All respondents were satisfied with the availability of core textbooks and other teaching and learning resources available in their schools. We receive textbooks and learning materials from the Ministry of Education and Heydar Aliyev Foundation. Also, our library is connected to the Internet. Moreover, our school is subscribed to the Curriculum Journal published on a quarterly basis by the Ministry of Education. We received 18 titles of books and encyclopedias under ESDP. We receive books on time and there is no shortage of them. School Grants Program The respondents have reported that the school grants program has enabled to improve learning environment in schools. We took part in three projects: new library for innovative learning in school, implementing innovations in chemistry classes, and implementing innovations in math classes. Schools in my rayon education department have taken part in various school grant projects. Implementation of New Funding Formula and District Rationalization Plans Overall familiarity of the respondents on this part of the reform was relatively low as opposed to other parts of the reform. Those respondents who were knowledgeable about the efficiency and finance reform have reported implementation shortcomings both with respect to the introduction of the new funding formula and district rationalization plans. Unfortunately, the Finance Department does not grant us autonomy to distribute the funds we receive according to our needs. New funding formula is not fully implemented. 25 of schools have received autonomy to use budgets according to their needs. The system has not been efficient so far. Rationalization plan is not implemented even though there is a draft plan, which has been developed several years ago. 42 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Ministry of Education of the Republic of Azerbaijan July 27, 2009 Education Sector Development Project I. EDUCATION CONTEXT IN AZERBAIJAN PRECEDING THE ESDP 1. At independence in 1991, Azerbaijan, like other countries in the Commonwealth of Independent States (CIS), inherited a relatively well-developed educational system from former Soviet Union. However, since independence, the educational system has been deprived of resources, and the general standard of education has visibly declined. The situation has further deteriorated by the emigration of many highly educated professionals. In addition, the Nagorno Karabakh conflict, which led to the displacement of over one million people, also resulted in the loss of all schools in that territory, amounting to about one-fifth of Azerbaijan's total number of schools. Azerbaijan has been struggling to retain the positive achievements of the past education system while introducing education sector reforms appropriate to the needs of an independent democratic state and an emerging market economy. Though some reform efforts were made after independence the real education reforms in Azerbaijan started with a comprehensive Education Reform Program approved by the President and the National Parliament in 1999. The program aimed at (i) improving the quality and relevance of student learning; and (ii) strengthening its capacity to manage its resources effectively, efficiently and equitably. The Education Reform Program stressed the need to improve the quality of services to meet the needs of the market economy and targeted: (1) curriculum reform and teacher training; (2) provision of educational materials including textbooks; and (3) educational management and finance. On the wave of this Program the Government of Azerbaijan implemented the first Education Reform Project financed by Learning and Innovation Loan (LIL) of the World Bank during 1999-2004 (Note: Implementation of LIL was rated "satisfactory" in the Completion Report prepared by World Bank hired consultant). Since the approval of Education Reform Program, Azerbaijan made some progress in initiating reforms in the education sector, with particular focus on curriculum reform and in-service teacher training, and the development of new textbooks and materials for general education. In 2003, the Government of Azerbaijan in cooperation with the World Bank prepared the Ten-Year Education Reform Strategy with reflection of the following main areas of concerns: (i) improvement of the quality of teaching and learning; (ii) the establishment of a student learning assessment system; (iii) teacher development (pre-service and in- service teacher training); (iv) strengthening of management and governance; (v) improvement in the efficiency in the use of resources; (vi) upgrading curriculum for general education; (vii) provision of new equipment and learning materials; (viii) improvement in access to quality education; (ix) improvement of the availability and the quality of textbooks in general education; and (x) improvement of the quality of higher education. Adaptable Program Lending (APL) instrument was chosen to finance the Ten-Year Education Reform Strategy in three phases. Education Sector Development Project (ESDP) has been designed to cover the first phase of APL in 2003-2008. 43 II. ACHIEVEMENTS OF ESDP Component 1: Quality and Relevance of General Education Sub-Component 1: Curriculum Reform 2. Institutional building for curriculum development: Major curriculum reform and implementation related problems in general education of Azerbaijan were inadequate institutional capacity to conceptualize, design, develop and implement the new school curricula, including: (i) shortage of qualified and experienced curricula developers; (ii) limited knowledge and expertise among teacher trainers, administrators, supervisors, and methodology personnel on curricular issues in particular outcome based curriculum model; (iii) insufficient resources that restrict the curriculum development institution to perform its basic functions (e.g., lack of financial resources for staff development and basic education equipment and materials, etc.); (iv) limited consultation with teachers, parents, civil society, NGOs and regional representatives in the curriculum development process; and (v) lack of a systematic approach to curriculum development with the result that the curriculum development process appears very fragmented. With the aim to overcome these problems a Curriculum Center at the Institute of Education Problems has been established, staffed and provided with necessary facilities and equipment to play the role of key organization responsible for curriculum development and reform process. Implementation of staff development activities for the Center such as local and international trainings, study visits etc. provided valuable support to building capacity for sustainable curriculum development. Contracting international and local consultancies has even empowered the Center and ensured necessary expertise required for curriculum reform process during ESDP life. The Center has got knowledge and skills to engage, train and manage curriculum developers and produce relevant curricula for general education in Azerbaijan. 3. Curriculum development and introduction: Outdated curricula were a major concern for the improvement of quality and relevance of general education in Azerbaijan. Curricula for general education were outdated and overloaded with large percentage of the information irrelevant for the needs of the emerging market-based economy and democratic society. The whole curriculum in its form of "Educational Programs" lacked coherence and didn't not provide the opportunity for the systematic development of knowledge and skills by the progressive movement from simple to complex and by the integration of overlapping subject topics. Also, the knowledge and skills taught through these subjects focused on facts, rigid procedures, the ability to recall information and to carry out computations etc. Many of the subjects lacked content that could stimulate children's thinking and reasoning abilities. In this regard the main achievement under ESDP has been the development of National Curriculum for general education and its approval in 2006 by the Cabinet of Ministers. Learning outcomes and content standards, subjects taught at each level of general education, organization of pedagogical process, basic principals of evaluation and monitoring of learning outcomes are presented in National Curriculum. National Curriculum has provided a framework for development of the subject syllabi for primary grades of general schools and their approval by the Ministry of Education in 2007. Learning outcomes per grades, content and assessment standards as well as relevant learning strategies are covered in subject curricula. The didactic bases of these curricula are demand-driven, outcome- based and student-oriented principles. Outcome-based principle is the predetermination 44 of outcomes; demand-driven principle is the responsiveness of learning goals for the needs of society, state and individual; student-oriented principle is to teach the learners the life relevant skills and knowledge. Introduction of new curricula in first grade of general schools started in 2008-2009 school year all over the country. Introduction of new curricula up to 2009-2010 school year have been supported under ESDP by: implementation of public communication campaign on new curricula including dissemination seminars throughout the country and printing and dissemination of curriculum sensitization materials including school calendars etc. development of "Curriculum Internet Portal", establishment of "Curriculum" journal. Sub-Component 2: Teacher Development 4. Teacher development concept: The overall quality of both pre-service teacher education and in-service teacher training in Azerbaijan was deteriorated considerably during the transition period after independence. Though teachers in general education have been required to undertake in-service education every five years, resource constraints have restricted the provision of in-service teacher training in Azerbaijan. Despite this requirement a significant percent of teachers in rural areas never attended in-service teacher training programs. Baku In-service Education Institute and Azerbaijan Teachers Institute with its affiliates around the country are the main recognized providers of in-service teacher training in Azerbaijan. Most of these teacher training institutions were poorly staffed and equipped to deliver effective teacher training. The teacher training process has been rather mechanical and more supply- than demand-driven. Pre-service teacher training is offered in both specialized secondary institutions (technicums) for primary school (grades 1-4) teachers and in universities for teachers in grades 5-11. The system of pre-service teacher education in Azerbaijan has also been facing numerous challenges, including: (i) a lack of professional linkages between teacher training institutions and schools (as well as between the teacher training curricula and the school curricula); (ii) emphasis on narrow specialization, which needs serious revision in order to reduce the number of subjects studied; (iii) limited awareness of the range of new and innovative teaching and learning methodologies and the use of teaching materials and equipment in the preparation and professional development of teachers; (iv) limited qualifications and experience of teacher training staff and limited staff development opportunities; (v) lack of basic equipment and materials for effective teaching and learning in teacher education institutions. In order to address these problems comprehensively, the Strategy and Concept Document for Continuous Teacher Development has been developed and approved in 2007 by the Cabinet of Ministers. As seen from the title, all aspects of teacher development are integrated as a conception in this document. The document focuses on strategic development priorities, such as, the changes of the content of teacher development (pre- and in-service training and retraining), structure and management of the system of teacher recruitment, teacher development, and establishment of quality assurance system. 5. Pre-service teacher training: The main achievement in changing the content of teacher development has been the development of Curriculum Framework as well as Subject Syllabi for Pre-service Training of Primary School Teachers and their approval by the MoE in 2008. The process of curricula development included involvement of 45 international and local consultancies as well as local and international trainings and study visits. The special peculiarity of these curricula is that they are outcome-based. Initially competencies for teachers were identified in new curricula, then competencies were transferred into learning outcomes and content lines were drafted in a way that will enable to achieve the learning outcomes. The other special feature of curricula is that Education Foundations subject, comprising of pedagogy, psychology, teaching methods, and principally, skills development through active/interactive learning technology was included in new curricula. Teaching this subject will play an important role in upgrading the quality of teacher development and in moving from theory to practice. Introduction of new syllabi for pre-service training of primary school teachers started on a pilot basis at the State Pedagogical University, Azerbaijan Teachers Institute and Ganja State University in 2008. Teaching staff of these institutions have been trained to get knowledge and skills on implementation of new pre-service training curricula. A new textbook has been developed to support "Foundations of Education" subject newly introduced in pre-service teacher training of primary school teachers. 6. In-service teacher training: As was described in previous paragraphs in-service teacher training in Azerbaijan required a new content and regulation framework, which could be developed only after approval of Strategy and Concept Document for Continuous Teacher Development. Since approval of Strategy document could be late for introduction of new curricula in general schools the MoE had to offer a new in-service training for school teachers based on new school curricula. These in-service teacher training programs have been developed with the help of Curriculum Center. All affiliates of Azerbaijan Teachers Institute were provided with ICT, furniture and other office equipment as well as two branches (Mingachevir and Jalilabad branches) have been renovated to support these trainings. These trainings covered approximately 18000 primary school teachers during the ESDP implementation period. After approval of Strategy and Concept Document for Continuous Teacher Development the MoE with engagement of international expert and broad consultation process started developing new regulation framework for in-service training. New regulation has been based on the following main principles: shifting from an input-based and supply driven to an output-based (demand driven) in-service training; opening of in-service training system to multiple public and private training providers and accordingly to multiple training contents and thus creating a competitive environment in-service training system. Implementation of new in-service training regulations is envisaged under Second ESDP. Sub-Component 3: Textbooks and Reading Materials 7. Textbook policy: The Government of Azerbaijan made significant efforts to provide textbooks for grades 1-4 during years preceding the ESDP. For grades 5-11, textbooks have been normally purchased by households, but some parents of children in rural areas were not able to afford them because of income constraint. According to Government's Poverty Reduction Strategy the textbooks became free for all students in general education. The ESDP being part of the Poverty Reduction Strategy has monitored implementation of this committment throughout the project life. "End-of- project evaluation survey" of ESDP conducted in 2007 provided that 97 percent of pupils in grades 1-5 of general schools have free of charge textbooks. Survey results confirmed that by 2007 the MoE has improved provision of free of charge textbooks in 46 other grades of general education. In order to approach textbook development and provision in a comprehensive manner, a new policy document entitled "Textbook Policy in General Education System" has been drafted under ESDP. New textbook policy included the following sub-documents: (i); Textbook content (ii) Textbook evaluation criteria; (iii) Rules for development of new textbooks; and (iv) Technical and hygienic standards for textbooks. New textbook policy was approved after being agreed with relevant Government bodies and Cabinet of Ministers. Independent Textbook Evaluation Board was established in order to ensure transparency in textbook development and procurement. Since 2008 the textbooks developed on the basis of new school curricula have been printed and delivered to schools after being evaluated by this board. According to new policy the textbooks are now required to be accompanied by the teacher's guide and student's workbook. This set of books shall enrich the content of general education, and efficiently serve achieving learning outcomes of new curriculum. 8. Learning materials: The quality of learning environment in general schools was low in the period preceding the ESDP. This was due to the lack of access to teaching and learning materials. In the early 1990s, Azerbaijan shifted from Cyrillic to Latin script. As a result of this shift, there was a severe shortage of reading materials in Latin-script in the schools. To stimulate reading habits, support core curricula and encourage independent study and thinking, the ESDP financed procurement and supply of 18 titles of child encyclopedia (846 thousand copies) to all general school libraries in Azerbaijan. In order to improve learning environment in the schools the ESDP also financed provision of 19 titles of different maps to all schools. Component 2: Efficiency and Finance Reforms Sub-Component 1: Finance and Budgetary Reforms 9. The education budget in Azerbaijan, like in the other sectors, is a line-item budget based mainly on the Soviet era planning norms, and spending cannot be shifted across line items or between budget years. The rigid nature of this budgeting has undermined the ability of local education administrations to prioritize expenditures and to allocate resources according to local needs. To start addressing these obstacles 59 schools of Shirvan city, Ismailli and Ujar districts (pilot regions) have got financial independence under ESDP. Per-capita funding formula has also been developed and applied on pilot basis in these schools by the decree of Cabinet of Ministers. The development and application of financial autonomy and per-capita funding formula has been accompanied by international and local consultancies as well as local and international trainings and study visits. The new financing mechanisms have aimed to eliminate inequality in funding, involving school councils represented by the parents in budget planning, and stimulating the efficient use of funds. There were difficulties encountered during the implementation of the new mechanisms. Thus, during the monitoring conducted by the Working Group of the MoE it has been found out that there are violations of 3 basic principles of new financing mechanisms, which are as follows: Allocation of school budget on the basis of per-capita formula: Though the school budget was calculated pursuant to the per-capita formula, afterwards there were some cases of interference with the budget. The occurred cases of interference (either with increasing or decreasing the school budget) contradicted the principle 47 of per-capita funding. Though the school budget is calculated pursuant to the formula, afterwards finance departments interfered and randomly increased and/or decreased the funds (in most cases they were decreased). If the school budget calculated on the basis of the norms had not reflected the reality, it should not have been amended randomly but rather recalculated after having revised the relevant norms of the formula. Lump-sum allocation of school budget: The school budget has been submitted to the Ministry of Finance by the district education and finance departments not as lump-sum amount but broken down into expenditure line-items. Therefore, the Ministry of Finance approved the school budget not as a lump-sum amount but by expenditure line-items. However, normally the funds should have been allocated as lump-sum amount, and the school should have developed its annual breakdown of expenditures through distributing that amount among line-items. On the contrary, finance departments allocated funds by expenditure line-items, which limit autonomy of schools. Shifting between expenditure line-items by schools: This principle has not been understood by school councils and is not accepted by the finance departments of pilot districts. The reason is said to be the fact that the funds allocated by the Ministry of Finance are in reality by expenditure line-items (not lump-sum) and there is no permission from the Ministry of Finance to shift between items. Legal and regulatory gaps are the basis of these problems encountered during implementation. Thus, the Law of the Republic of Azerbaijan on Budget System adopted in 2006 prohibits making any changes in any line-items of the budget allocated to a school which, in fact, decreases financial autonomy of schools. In accordance with the Decision of the Cabinet of Ministers, schools involved in the pilot have been permitted to make such changes to ensure that schools exercise financial autonomy and spend allocated funds for their needs. But because from the legal perspective the law prevails over the Decision of the Cabinet of Ministers, schools have not been able to fully benefit from the granted autonomy. Such legislative gaps allowed for mentioned interference with the development of school budgets and expenditure of funds. However it would be worth mentioning that all 59 schools which were financially independent managed their own grant funds themselves. This gives evidence that schools with relevant instructive and reporting arrangements can efficiently manage their own resources. Sub-Component 2: Rationalization and School Improvement 10. School network rationalization: The high wage bill reflects over-staffing in the educational sector of Azerbaijan. If Azerbaijan were to achieve the average staffing ratio of the OECD countries for each of the three levels of education, it would employ 36 percent fewer teachers. The small number of students per teacher in general education is the result of a complex chain of events. First, the highest ratios of students to teachers are found in the smallest rather than the largest schools. Second, low workloads are a major contributor to high staffing needs in general education. Third, the policy of permitting teachers to teach only in their area of specialization plays a substantial role in raising the number of teachers employed and in determining the low ratio of students to teachers. Taking all these factors into consideration the school network rationalization has been launched in the same 3 pilot regions (Shirvan city, 48 Ismailli and Ujar districts). Separate rationalization plan for each pilot region has been developed with the support of international consulting company and national consultants. The mentioned rationalization plans has been based on the analyses and social assessment carried out in these districts. The plans included closure of small and inefficient schools, transportation of students to other ones by buses, rehabilitation of needy schools, increase of student teacher ratio as a result of reduction of personnel resources and thereby rationalization of school network to considerable extent. But these plans, in account of objective reasons, have been carried out to a limited extent. The main reason is that the project didn't envisage procurement of transportation means (buses). However there have been following administrative activities which resulted in rationalization to some extent: identification of minimum class size with regard to establishing additional parallel classes in selected schools; pensioning off the teachers who have reached the pension age by taking into consideration level of their labor productivity; consideration of the issue of whether or not to employ the teachers with secondary education degree employed by schools; consideration of the issue of whether or not to employ the teachers with incomplete higher education who failed to proceed with their education in higher education institutions and thereby expelled from these institutions; adjustment of number of teachers, and especially of the master teachers in schools in accordance with workload (number of classes); suspension of recruitment of new teachers who are not necessary for schools; fixing a teacher's minimum workload at 12 hours. In support of rationalization activities 41 schools were rehabilitated and provided with furniture and ICT equipment. As a result of rationalization activities the average student teacher ratio increased up to 11 in 3 pilot regions in 2009. This increase has been achieved in spite of a sharp decrease of the number of pupils in Ismailli and Ujar districts from year to year. In these districts decrease of the number of pupils is especially related with demographic problems, migration, and with the pupils continuing their education in vocational and special secondary schools after completing 9th grade. It is very important to note that while in the pilot regions the student-teacher ratio has increased (mainly because of ESDP intervention), the ratio has decreased nationally. 11. School building norms: The majority of school buildings in Azerbaijan were built according to old/Soviet standards for educational infrastructure, which are rigid in many respects and inefficient compared to generally recognized/international norms for education facilities. Most school infrastructures were outdated and largely inappropriate for modern/effective teaching methods. The existing school building norms, which were inherited from the former Soviet Union, have not been revised. On the other hand, planned transition to new national curriculum in the general education necessitated the change and efficiency of learning premises. An improved school building should create a condition for applying new teaching methods in new learning premises of 21 century. To achieve that a new "School Building Design Guidelines" has been drafted under ESDP using the international experiences and expertise of international and local consultants. Modern standards and norms for school construction are part of Guidelines which create great opportunities for efficient learning premises. The Guidelines have been developed for designing new type school buildings in Azerbaijan and is meant for 49 piloting during first years. These Guidelines, having been tested and improved every time may well contribute to the development of new norms and rules for construction nationwide. Requirements, proposals and recommendations for designing school buildings, considering local natural condition and climate, modern construction structure, new security and fire safety rules are provided in the Guidelines in order to apply new organizational forms of teaching. Principles of design are given in the Guidelines in a way that classrooms present inter-disciplinary, project-based, collaborative learning studio or flexible learning center and support 18 different learning modules. The objective of the Guidelines is to provide the compliance of the quality and character of the environment to be established with the educational goals of school, turn the schools to be built in the future into a place which is relevant for intellectual, creative, physical, and social activities, and at the same time a lively and hospitable place owned by pupils. Component 3: Upgrading Schooling in Less Advantaged Rayons: School Grant Program 12. The School Grant Program (SGP) has been designed to supplement the limited resources of schools and to promote more fully equity and access to quality and innovative education to improve enrollment and attendance rates. To implement the SGP a Manual has been developed and approved by the MoE. The Manual, later on, has been updated according to experiences from the preparation and implementation of the SGP. Based on the design, institutional arrangements and the selected criteria in approved Manual, the SGP has been implemented, monitored and evaluated for three phases in the three pilot regions (Shirvan city, Ismailli and Ujar districts). Implementation of each phase has always started with comprehensive training programs supported by the Open Society Institute of Azerbaijan, which has been a partner of the MoE for this component. Grants were provided to 46 winning projects in 3 pilot regions in the 1st stage of the SGP for an aggregate utilized amount of USD 175000. In the 2nd stage grants were rewarded to 73 winning projects for which USD 351000 has been spent. Finally, 54 winning grant projects have implemented in the 3rd stage for an amount of USD 187000. Given the success of this component in terms of making non-salary resources available to schools, the MoE initiated allocating resources for such grants countrywide. In September 2007, the President of Azerbaijan Republic issued a Decree to award grants to the 50 best schools to further improve the quality of education and motivate the teachers. The regulation for the competition has been elaborated by the MoE and approved by the Cabinet of Ministers. Such allocation of grants became an annual practice of the MoE. Component 4: Management, Planning, and Monitoring Capacity Subcomponent 1: Establishment of a New System of Student Assessment 13. Student assessment concept: The peculiarity of assessment rules of Soviet period was that learners' assessment and assessment standards was never considered as a component of curriculum. This resulted in assessment being always based on the subjective observation of teachers which pushed pupils to study only for the sake of getting mark. Along with education standards, the assessment standards have also been integrated into the content of general education curricula. In order to comprehensively approach student assessment in Azerbaijan a document entitled "National Assessment 50 Concept for General Education" has been developed and approved by the Government. School-based, national and international student assessments are the main three types of assessment defined in the Concept to be implemented in Azerbaijan. As a first step in institutional arrangement for student assessment the ESDP supported establishment of a Student Assessment Unit in the MoE. This unit has been provided with building and all necessary equipment for administration of the assessments. The staff of the unit has been trained locally and internationally to better cope with the challenges of the unit. 14. National assessment: ESDP financed national assessment for Mother tongue and Math subjects in grades IV-IX in 2006. An internationally reputable organisation (CITO, Netherlands) has been involved in administration of this national assessment. The main finding of this assessment has been a difference in performance between rural and urban areas in both grades and subjects. Indeed, Baku (capital city) population scored highest. The availability of better equipment, materials and better qualified staff could be part of the explanation for the higher performance in urban areas. It is worth noting that students in both grades have been better at reading fictional texts than so-called every-day texts that any reader will come across in newspapers, magazines etc. 15. International assessment: International assessment in Azerbaijan started with the joining the PISA in 2006 under ESDP financing. The level of skills in mother tongue, math and sciences of 6500 15-year-old pupils in 72 general schools has been assessed under PISA in 2006. The performance of our pupils in math was very positive. They held the 4th place for theoretical skills among 57 countries in the rating chart. But as for the practical skills the performance of our pupils has been unsatisfactory. In order to track performance of Azerbaijan education in the world arena and get the best evidences of implications of education reforms in Azerbaijan the MoE during project life continued particpation in PISA 2009 and joined two more international student assessment programs namely PIRLS and TIMSS. The results of latter assessments are not known yet and therefore not part of the report. 16. School-based assessment: As a first step of changing school-based assessment, Ministry of Education has started to apply centralized general school leaving exams since 2005/2006 school year. The response cards in new exams are checked by optical mark readers and the results are disclosed in a short term. Such a way of organizing the examination minimizes the subjectivity factors. The Monitoring and Evaluation Unit of the MoE produced a first analytical report on the results of the secondary school graduation exam in 2008. This report deals with "quality indicators" by school and by subject and is certainly a significant contribution in terms of data availability for evidence-based policy decision-making in MoE. Subcomponent 2: Establishment of Education Management Information System (EMIS) 17. At the start of the ESDP the MoE had the following deficiencies with regard to efficient operations and management: labor-intensive, slow for data aggregation, difficult for data analysis, time consuming and limited information. There was no centralized policy with regard to information (data) gathered (and reported) by the databases. At the MoE (and in the districts) there was no expertise or experience for EMIS implementation and management; and no responsible department or unit. Therefore, the ESDP supported establishment of EMIS unit at the MoE. This support has been provided by engagement 51 of international and local consultancies as well as financing of hardware and software for EMIS installation in MoE and district education departments including provision of necessary training. The EMIS unit started implementation of EMIS throughout the MoE and local education departments. Today, the EMIS database comprises information on general schools including comprehensive information on students and teachers, school principals and other staff, physical facilities etc. The EMIS Unit made publicly available a school-by-school database at the MoE official website. The Unit is also able to electronically receive and handle education statistics from the Rayon Education Departments. Although the EMIS database is rich of information about general schools and is available on MoE LAN there are not many users of EMIS nowadays. The main user of the EMIS is the Monitoring and Evaluation unit of the MoE for purposes related to implementation of different types of assessments including exams, national and international assessments. The main reason for limited use of EMIS is the lack of culture and skills to use computerised information database. Subcomponent 3: Management and Planning Capacity 18. In 2003 the education sector was inadequately organized and managed for the delivery of education services. The MOE had a weak management and planning capacity to support tasks such as policy development and long term planning, and an effective monitoring and evaluation system that will assess the performance of the education system and the reform process. The functions required for implementations of challenging education reforms had not been appropriately reflected in the structure of the MoE. To address these problems first an Organization Plan for MoE and Institutte of Education Problems (IEP) has been developed and approved by the Minister of Education. Strategic planning and reporting has been introduced in MoE that implied introduction of 3-year planning by departments of MoE. By reorganizing the structure of the Ministry of Education the following new structural units have been established: (1) Strategic Analysis and Planning unit (SAPU); (2) Monitoring and Assessment unit (MAU); (3) Education Management Information System unit; (4) Preschool Education unit. These units made great contribution in reforms and development activities under ESDP. In order to support policy decisions with applied researches a Center for Applied Education Researchs (CAER) has been established in IEP. Taking the advantage of the wealth of information collected in the EMIS the SAPU has been able to produce MoE's first annual "Education Indicators Report". Such report is an important step in building the capacity within the MoE to analyze and make sense of large data sets. The most important achievement under this sub-component is the MoE's capacity built throughout the project implementation for designing, initiation and implementation of ambitious reforms in education sector. Component 5: Project Coordination and Monitoring 19. Following a lesson learned from previous experience, management of ESDP has been integrated into the overall operations of the MoE. The project implementation scheme was quite complex and well managed and coordinated by the MoE staff (sub- components coordinators are civil servants employed in the MoE). Project Coordination Unit (PCU) has been fully incorporated in the MoE with the following functions: assisting MoE in coordination of project activities; monitoring and reporting on implementation progress, procurement and financial management. 52 20. Coordination of ESDP activities: The PCU has made every effort to ensure that ESDP is coordinated at all levels as follows: Coordination at the Government level: The example of coordination at the Government level could be coordination of school financing reforms within the Government especially with the Ministry of Finance. Coordination at the MoE level: During implementation of ESDP the MoE has been implementing over 14 other state financed projects. These projects included activities related to the ESDP scope, which might lead to duplication or at least confusion if not coordinated in time. Coordination at the Project level: This included coordination between components and sub-components as well as with the partners of MoE in the ESDP such as Open Society Institute in Azerbaijan. The main instruments utilized here have been regular meetings of MoE leadership with ESDP implementers dedicated mainly to reviewing and analyzing reports by ESDP (sub) component coordinators. Coordination with the World Bank: This has been probably the most systematic and routine coordination and included all aspects of ESDP implementation. The PCU has been in day-to-day contact with the World Bank for every project issue. In addition there have been at least two supervision missions a year on ESDP implementation. Coordination with relevant activities financed from other sources: The best example of such coordination has been collaboration with UNICEF and Eurasia Foundation. Due to such collaboration, mainly achieved through ad-hoc meetings, the UNICEF financed international review of school and teacher development curricula prepared under the ESDP. Another result of such coordination has been the support of Eurasia Foundation to financially autonomous schools on topics related to school management. 21. Monitoring and reporting on implementation progress: Monitoring and Evaluation Plan of ESDP has been developed and agreed with the World Bank. Reporting system for ESDP has been introduced to be on quarterly basis. These quarterly progress reports have been prepared and submitted to the World Bank on time. The detailed M&E Report for the Second and Third Phases of School Grant Program (SGP), and an Overall Report of all the three phases of SGP has been prepared and submitted to the World Bank also on time. Based on Monitoring and Evaluation Plan, ESDP performance has been monitored closely through the list of key performance indicators. Progress towards key performance indicators has been reported regularly during the implementation of the ESDP. Baseline survey of ESDP was conducted in May 2005, mid-term evaluation in 2006 and end-of-project evaluation in 2007 with involvement of international M&E consultant and local consulting company. The comparative analysis report of baseline, midterm and end-of-project evaluation has been submitted in implementation progress report on September 30, 2007. Compliance with some of them has been given a function of "triggers" for authorization of the second phase of Program. The below table provides information on fulfillment of triggers during ESDP. Triggers Implementation of triggers 1. National curriculum framework for National Curriculum of General Education was grades 1-11 developed, standards developed by the Ministry of Education and developed and approved for grades 1-4 approved by the Cabinet of Ministers on October 53 by the MoE 30, 2006 2. New syllabi at least for two grades All subject curricula of primary grades was developed and ready for approved on July 23, 2007 implementation on a pilot or nationwide basis 3. At least 90 percent of children in "End-of-project evaluation survey" of ESDP grades 1-5 have free of charge conducted in 2007 provided that 97 percent of textbooks in "core subjects pupils in grades 1-5 of general schools have free of charge textbooks. 4. New funding formula and mechanism Financial independence was granted to 59 prepared and implemented on an general schools of Shirvan city, Ismailli and Ujar experimental basis in the three pilot districts. Per capita funding formula was applied districts in these 59 schools by the decree dated February 22, 2007 of the Cabinet Ministers. 5. District rationalization plans prepared Rationalization plans were developed in Shirvan in the three pilot districts, under terms city, Ismailli and Ujar districts selected as pilot of reference satisfactory to IDA, and regions and approved by the Ministry of approved by the Government and Education on October 17, 2005. reviewed by IDA 6. Completion of a new organizational Structural changes were made in Education and staffing plan for the Ministry of Problems Institute and this trigger was met. New Education (including the Institute for Regulations was approved and newly established Education Problems) structural divisions (Curriculum Center, and Center for Applied Educational Researches) were staffed through competition. 3 new units were already established in the structure of the Ministry of Education (Monitoring and Assessment Unit, Education Management Information System Unit, Strategic Analysis and Planning Unit) and were staffed through competition. 7. Completion of a pilot sample-based Census-based assessment in the mother tongue reliable assessment of grade 4 student and mathematics subject of 4th and 9th grade achievement in Mathematics and Azeri students were carried out in April-May 2006. language Analysis of the outcomes was ready in the spring of 2007. 22. Procurement management: Procurement plans have been developed and updated always on time and procurement process followed implementation schedule of ESDP. Goods, works and services for the ESDP have been procured in accordance with the Credit Agreement and World Bank procurement procedures were diligently followed. Training and guidance to pilot rayon education departments on procurement procedures under School Grant Program has been implemented on time. Almost 285 procurement packages have been processed, contracts signed and completed by the closing date of the ESDP. 23. Financial management: Project disbursements under ESDP amounted to total USD 21.83 million. Implementation of these disbursements including (i) implementation and recording of daily financial transactions and tax payments, (ii) preparation and 54 processing withdrawal applications for the World Bank and Government funds, (iii) preparation of different types of financial reports as well as (iv) arrangements for annual audits has always been on time. III. RATING OF KEY PERFORMANCE INDICATORS The development objective of ESDP has been to strengthen the capacity to plan, manage and monitor the reform program more effectively and efficiently and initiate the highest priority investments in selected districts. This development objective is achieved by the project as explained below by the status of outcome/impact indicators. Outcome/impact indicators: 1. Curriculum development capacity established in the restructured Institute for Education Problems by 2006. Satisfactory: A new Organization Plan for Institutte of Education Problems (IEP) has been developed and approved by the Minister of Education in December 2004. Curriculum Center has been established at IEP, staffed and provided with necessary facilities and equipment to play the role of key organization responsible for curriculum development and reform process. 2. Management and planning capacity of the MoE and local governments strengthened and performs its functions efficiently and effectively Satisfactory: Strategic planning and reporting has been introduced in MoE that implied introduction of 3-year planning by departments of MoE. By reorganizing the structure of the Ministry of Education the following new structural units have been established: (1) Strategic Analysis and Planning unit (SAPU); (2) Monitoring and Assessment unit (MAU); (3) Education Management Information System unit (EMISU). The staff of these units has been trained by provision of local and oversees trainings. These units made great contribution in reforms and development activities under ESDP. 3. Education statistics, including learning outcomes, are readily available at the national level and pilot districts and published in the form of "Education Indicators" for policy makers and the public by 2006. Satisfactory: Since 2006 the EMIS database comprises information on general schools including comprehensive information by grade level, gender and location (rural and urban) on students and teachers, school principals and other staff, physical facilities etc. The EMIS unit made publicly available a school-by-school database at the MoE official website (www.edu.gov.az). Results of the examinations as well as reports on international and national student assessments are also posted on MoE website. Using these data the MoE issued first education statistics report in 2008. 4. Policy makers and education stakeholders use the results of learning outcomes and reliable statistics in their policy analysis and strategy development. Satisfactory: Policy analysis and strategy development is usually initiated by the high level leadership of the MoE to support and justify policy interventions from MoE. 55 Except SAPU the MoE units don't use extensively information about learning outcomes and education statistics in their work. 5. 25 percent of primary teachers (approximately 9,000) are familiar with new curriculum, teaching methods, and assessment targets and begin to reflect those changes in their teaching practices during the 2007-2008 school year. Satisfactory: Approximately 18000 primary school teachers have been trained on new curriculum including teaching methods and assessment tools during the ESDP implementation period. 6. Present disparities in access to core textbooks (based on new curricula) in basic education grades (grades 1-9) eliminated by 2006. Satisfactory: Evaluation survey conducted in 2007 (ARTIM, 2007) provided that availability of free of charge textbooks (based on old curriculum) in basic education grades (grades 1-9) does not show disparity in access to textbooks between pilot and non-pilot rayons. "Textbooks and Curriculum Implementation Survey" conducted in May 2009 (Sigma, 2009) confirmed lack of disparities in access to textbooks based on new curriculum. 7. Student-teacher ratio in general education increased from 10:1 in 2002 to 12:1 in 2007 in the three pilot districts. Moderately satisfactory: As a result of rationalization activities under ESDP the student teacher ratio in general education has been reported by Economics unit of MoE to be as follows in three pilot districts in 2007: 12.00 in Shirvan city (former Ali Bayramli); 10.90 in Ismailli district; 11.80 in Ujar district. 8. Efficiency and finance reforms, including district rationalization plans, prepared and implemented successfully in the three pilot districts by 2007. Moderately satisfactory: Efficiency and finance reforms including rationalisation plans have been really implemented in the three pilot districts. However difficulties encountered in implementation process and explained under Component 2 of Section II. Achievements of ESDP makes it difficult to say that these reforms have been successful. 9. Society at large express support for education reform by 2007. Satisfactory: Main finding of Evaluation survey conducted in 2007 is that "Society at large (school directors, parents, teachers) express support for education reform across all the items in 2007" (ARTIM, 2007). 56 IV. LESSONS LEARNED FROM ESDP IMPLEMENTATION EDSP demonstrates that in spite of large interventions and investments in education from the state budget the projects implemented with the Bank are the most important tools in initiating major reforms in education sector. The major lessons learned from ESDP are as follows: Good sustainability vision for project activities is very crucial right from the beginning of the project. A reform to be successful needs to have a dedicated leader within MoE. Lack of such leader was the main reason for problems under Component 2; Unfortunately, inter-ministerial collaboration in Azerbaijan succeeds not much beyond formal correspondence between the ministries; A `Quality Monitoring Framework' should be put in place by MoE and staff capacity is enhanced for this purpose. Results framework for projects should be well understood by MoE and M&E plan should be effective right from the beginning of the project; Comprehensive public awareness and information campaign is critical to avoid unnecessary slow down of public embrace of the education reforms and their impact. In order to put more emphasis on this very important function and in order to make it sustainable throughout the project operation it should integrated into MoE and not seen as PCU function. Formal performance appraisal linked to appropriate incentives is needed to motivate and reward project staff for the high-pressure work environment of PCU. V. PERFORMANCE OF THE BORROWER The Government of Azerbaijan has since the initiation of the ESDP, remained committed to the project objectives. The total estimated Government contribution to the Project implementation has amounted to USD 2,894,251. Initially, there was lack of coordination between different agencies involved with the project and the project received `unsatisfactory' rating from the Bank Supervision Mission for a few components. The situation vastly improved from early 2005. PCU has been very active in coordinating the project. It has a highly motivated staff. This aspect, along with the increased cooperation from the different departments of MoE, has lead to successful implementation of the project and achievement of Project Development Objective. VI. PERFORMANCE OF THE BANK The Bank worked closely with the Borrower to support its efforts to successfully implement the project. Despite change in the Task Managers, the dialogue during the implementation process was smooth and both parties sought and found solutions to implementation bottlenecks (civil works procurement and financial aspects). The bank supervised the project regularly (twice a year) and this intensive supervision was one of the major contributing factors in achieving the targets. The Bank has agreed to the request for project extension from March 31, 2008 to March 31, 2009. The Bank' performance has been very positive especially for: 57 Good supervision and extension of the life span of the project; Flexible response to the needs of the country and the education sector. Good team work and communications with the Government including per-capita financing reforms topics. Field visits and meetings with the stakeholders and the beneficiaries. Strong support in procurement and finance. All these efforts created a positive relationship between the Bank and the Borrower that led to a successful implementation of the project. VII. PLANS FOR SUSTAINABILITY OF ESDP ACHIEVEMENTS The general plan for sustainability of the achievements of ESDP is the initiation of the Second Education Sector Development Project (SESDP). The SESDP includes continuation of the majority of achievements under ESDP. Those achievements and interventions which are not included in the SESDP are addressed by the Government of Azerbaijan with the following arrangements: (i) Pre-service teacher training: The Higher Education Unit of the MoE has taken an ownership over the achievements and interventions of ESDP in pre-service teacher training area. It is believed that this unit will continue a decent work towards improvement or quality of pre-service teacher training especially in the view of the two facts described below: The Higher Education Unit is no longer responsible for in-service teacher training, which releases its staff from many related duties and the staff may and is actually more concentrating on pre-service teacher training; MoE is in the process of preparation of Higher Education Project, which shall partially address pre-service training it takes place in higher education institutions. (ii) Textbook policy: Textbook and Publishing Unit of the MoE in collaboration with Textbook Approval Board has already taken ownership over systematic implementation of new Textbook Policy. The work that these unit and board completed during last and this year provides evidence of sustainability for implementation of textbook policy. (iii) The School Grant Program: Annual competitions of the Government of Azerbaijan for allocation of grants to the 100 best teachers as well as 50 best schools is an evidence of sustainable implementation of the ideas promoted by the School Grant Program implemented under ESDP. 58 Unofficial Translation of Borrower's comments to the ICR THE AZERBAIJAN REPUBLIC DEPUTY PRIME MINISTER Ref #: 15/5-120 06 May 2010 To: Gregory Jedrzejczak Country Manager of the World Bank in the Azerbaijan Republic Dear Mr. Jedrzejczak, Further to your letter dated 9 March 2010, please see attached for your consideration the recommendations and comments of the relevant entities with regard to the "Report on Completion of Implementation and Results of the Education Sector Development Project". Yours sincerely, Elchin Afandiyev 59 THE AZERBAIJAN REPUBLIC MINISTRY OF EDUCATION Ref #: 40-12-157/14 To: Mr. Elchin Afandiyev 08.04.2010 Deputy Prime-Minister of the Azerbaijan Republic Dear Elchin , We have reviewed the "Report on Completion of Implementation and Results of the Project" with respect to the Education Sector Development Project (IIA Credit 3769AZ) submitted to the Ministry of Education together with the letter to Mr. Gregory Yedjechak with your endorsement No. 17/110--15/11 dated 5th April 2010. Please be advised that the document was developed together with the representatives of the World Bank in due course and all our comments and recommendations were considered during the preparation period. Therefore, we do not have any additional comments or recommendations with regard to this document. Please consider the above. Yours sincerely, MISIR MARDANOV Minister 60 THE AZERBAIJAN REPUBLIC MINISTRY OF ECONOMIC DEVELOPMENT To: Mr. Elchin Afandiyev AZ1000, Baku city, 40, U. Hajibayov Street, City Hall Deputy Prime-Minister of the Azerbaijan Republic Tel: (12) 493 88 67, Fax: (12) 492 5895 email: office@economy.gov.az Date: 04 May 2010 Ref #: 20/01 - 2/130 ­ 2/427 (In reply to: ref. 17/1102-15/11 Dated 05 April 2010) Opinion about the Report on Completion and Results of the completed first phase of the "Azerbaijan Education Sector Reform" Dear Mr. Elchin Afandiyev, Further to your instructions the Ministry of Economic Development has reviewed the letter requesting to express the opinion about the Report on Completion and Results of the completed first phase of the "Azerbaijan Education Sector Reform" program of the World Bank. As it is indicated in the said report, the project considers the measures such as the creation of the basis for preparation of the curriculum in the reorganised Institute of Educational Problems by 2006, strengthening of the management and planning potential of the Ministry of Education and local education authorities, development and public disclosure of the education statistics, the training of primary school teachers in connection with the implementation of the new curriculum, development of the new school financing formula and its implementation in three pilot districts and increase in the number of students. The implemented measures resulted in establishment of the Curriculum Centre in the Institute of Educational Problems, preparation and public disclosure of the first statistical report of the Ministry of Education in 2008, the creation of the three new departments (Monitoring and Evaluation, Information System and Strategic Analysis of the Education Management, Planning and Human Resources) in the Ministry of Education and the Institute of Educational Problems was reorganised, financing of the renovation of a selected number of schools in the three pilot districts was implemented, participation of approximately 18 000 primary school teachers in trainings on modern education methodology and evaluation methods required for implementation of the new curriculum and in minor increase in student-teacher ratio in three pilot districts. Considering the above, we would like to note that we have no comments or recommendations to the Report on Completion and Results of the said project. Please consider the above. Yours sincerely, Shahin Mustafayev Minister 61 THE AZERBAIJAN REPUBLIC MINISTRY OF FINANCE To: Cabinet of Ministers of the Azerbaijan Republic AZ1022, Baku city 83, S. Vurgun Street Tel.: 404 46 99, 404 47 15 Fax: 404 47 20, 404 47 21 Email: office@maliyye.gov.az 28 April 2010 No 08/02 ­ 44-447 To: 17/1102-15/11 dated 5 April 2010 Regarding the World Bank Report on the final results of the first phase of the education sector development The Ministry of Finance having reviewed the World Bank Report on the final results of the first phase of the education sector development notes the following: With regard to the increase of the student-teacher ratio in three pilot districts from 10:1 in 2002 to 12:1 in 2007 (section "Efficiency and Financial Reforms", paragraph 3 of the main performance indicators of the report), we would like to note that as per the analysis performed in general education system of the pilot schools the student-teacher ratio was 9:3 instead of 12:1. It must be noted that the level of the teacher-student ratio directly depends on the weekly lesson load of the teachers. Although the weekly lesson load of the teachers in general education schools in most countries is 18-20 hours, the weekly lesson load of the teachers in Azerbaijan is 12 hours and this fact does not allow the significant changes (increase) in student-teacher rate in general education schools. The expert opinion expressed in article 2.2, paragraph 3 "lack of agreement between the Ministry of Finance and the Ministry of Education on the responsibility mechanisms of budget management schools" is absolutely groundless. Thus, during the reform period the budget of pilot schools was developed together with the project group of the Ministry of Education as per the formula based on the expenses per student and the Ministry of Finance took all the required measures in this direction supporting this component together with all other phases of the reform. In connection with the 2nd Component: Efficiency and financing reforms" paragraph of the 8th section "Results indicator", i.e. "allocation of lumpsum budgets to pilot schools", we must note that in accordance with the requirements of the statutory documents regulating the budget legislation the budgets of the budget entities are approved as per the expense articles for the given year. Therefore inclusion of these ideas into the report conflicts with the legislation. The opinion of the experts expressed in the third and fourth paragraph of section 8 of the Report 62 "Results indicators" - "the gaps in legislation and lack of Ministry of Finance support during the implementation", as well as "the lack of self-sufficiency of pilot schools, limitation of substitutability of wage and non-wage items" is not clear. Thus, in accordance with Article 18.4 of Azerbaijan Republic Law "On budget system" if needed, certain changes between the expenditure items may be made". During this year there were no limitations in financing of all required spheres using the saved expense items and this process was performed by local financial authorities and Ministry of Finance. Thus, after the Decree No 36 of the Cabinet of Ministers of Azerbaijan Republic on execution of the "Financing and budget reforms" subcomponent of the "Education sector Development Project" dated 22nd February 2007, there was the necessity of the diversification between expense items in some of the pilot schools (Ismailly region), which was supported by the Ministry of Finance and 47.6 thousand manats difference between the schools was approved on 12th December 2007. Therefore we cannot agree with the expression of "limited self-sufficiency" of pilot schools. In addition, it must be noted that in accordance with the requirements of the Law of Azerbaijan Republic "On budget system" the reform program does not consider the requirements of the Unified Budged Classification developed together with the representatives of the International Financial Fund and approved by the Government. Thus, the budget expenses or each budget funded entity for the given year is approved by the expense items as per the requirements of the Unified Budget Classification and the cash execution of the budget for the year is performed as per statutory documents of the treasury. However the reform program does not consider these requirements of the budget legislation and demands the forecasting of secondary school budgets on the basis of the unified expense principle and full independence of the schools in usage of the expenses. Moreover, the Ministry of Finance recommended to include Baku schools with better infrastructure rather than the schools in far rural regions into the pilot program in order to better direction of the reform programs in the secondary school. Notwithstanding the numerous discussions of the representatives of Ministry of Education and the World Bank held in the Ministry of Finance about the conflict of these requirements with the country legislation, the recommendations, comments and suggestions of the ministry were not considered in the development of the reform project. With respect of the last sentence of the 4th paragraph of the Report ­ "the education and support with respect of the new financing system between financial departments and pilot regions was weak" ­ we must note that the numerous discussions with local financial authorities have revealed the formal nature of the boards' activity, they were not familiar with the reforms in the schools and neither boards, nor the local financial authorities participated in decisions made at schools. At the same time, the local education authorities were reorganised in order to accommodate the new financial issues in pilot schools and each pilot school has employed accountants for preparation of financial documents. However, the insufficient economical knowledge of the accountants created serious difficulties in their work with the financial documents. In most cases the financial documents of the pilot schools were prepared with the assistance of the specialists of financial authorities. 63 Section 9 of the Report - "Result indicator" - indicates the support level of this reform in the society and the status of the acceptance of this reform by the country citizens. Thus, in pilot schools 42% of principals, 47% of teachers and 46% of teachers were unsatisfied with the reforms. However, the satisfaction level of the village dwellers has reduced from 42% during the initial poll to 37% at later stages. "3.3. Efficiency" section of the Report only covers the economical part of the reform and prioritises the financial independence of the pilot schools for success of the reform. We think that the efficiency of the reforms in the schools is measured not only with the financial independence, but also with the level of the teaching and getting the education, with the achieved results in education since the start of the reform, or with the information about the level of admission of the students of the pilot schools to the colleges or about the aspects of the general knowledge levels. However the report did not reflect these issues and did not give any conclusions with this regard S. Sharifov Minister 64 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders Open Society Institute (OSI) Azerbaijan provided co-financing for the selected number of activities under the following three subcomponents of ESDP: Textbook Policy Development, Improving Equity and Access of the Poor to Education and Rationalization and School Improvement. The detailed description of OSI contribution to the Project is reflected in Annex 2. In addition, the Bank team collected feedback from OSI staff, Ms. Elmina Kazimzade, with regard to the collaboration as part of ESDP. The following comments were received: "This partnership between OSI, WB and MOE within ESDP Program in Azerbaijan was a first experience of such cooperation between government, international finance agency and non-governmental organization. The cooperation allowed to apply OSI expertise in education in reforming process and created an opportunity to shift from grassroots activities to policy impact initiatives. The cooperation might be suggested as a good example of consolidation model of joint efforts of different donors, state and civil society institutions. There are list of outcomes of the cooperation that can be considered as success indicators of this cooperation: - Empowerment of Civil Society Institutions as influential actors in Education Reform. Example: Establishing Textbook Approval Board - Strengthening of the education quality through partnership between state and non-governmental institutions. Example. OSI spin off Education Center for Innovations in Education was engaged in second phase of ESDP". 65 Annex 9. List of Supporting Documents Azerbaijan Country Assistance Strategy. Report No. 25790-Az. April 2003. Azerbaijan Country Assistance Strategy, FY07-10. Report Nu. 27812. November 2006. Ministry of Education of the Republic of Azerbaijan. Quarterly Progress Reports (Project Coordination Unit). 2003-2009. Ministry of Education of the Republic of Azerbaijan. Evaluation Survey to Measure Key Performance Indicators under the Education Sector Development Project (Artim MMC). October 2007. The World Bank. Aide Memoires of Supervision Missions (13 documents). 2003-2009. The World Bank. Development Credit Agreement. Credit No. 3769-Az. September 2003. The World Bank. Project Appraisal Document. Education Sector Development Project (APL I). Report No. 25284. April 2003. The World Bank. Project Appraisal Document. Second Education Sector Development Project. Report No. 40973. March 2008. The World Bank. Azerbaijan. The Challenges of Knowledge Economy: The State of the Education System. Report No. 40808-Az. September 2007. 66 45°E 46°E 47°E 48°E 49°E To RUSSIA N Makhachkala GE OR GI A Balaken G FEDERAT IO N To To Tbilisi BALAKEN Zagatala rea Tbilisi ZAGATALA te KHACHMAZ r GUSAR C Gakh au Gusar Khachmaz AZERBAIJAN To AGSTAFA GAKH ca Guba Vanadzor GAZAKH su Devechi To Sheki s GUBA Vanadzor Agstafa SHEKI M DEVECHI TOVUZ Oghuz o 41°N Gazakh Ku ra Mingechevir u Siyazan 41°N Tovuz Tovuz OGHUZ SIYAZAN n Le SAMUKH Reservoir Gabala ttt a a a ss SHEMKIR Khizi iii n GABALA er Shemkir Nabiagaly MINGECHEVIR s s s Ismailli SHAMAKHY Mingechevir ISMAILLI KHIZI Ganja To Agdash SUMGAYIT SUMGAYIT Caspian Sea C C C GANJA Yevlach Yevlach Goychay Shamakhy Vanadzor a Gedabay Geranboy AGDASH Sumgayit u Khanlar GOYCHAY Akhsu Maraza YEVLACH c c c KHANLAR Udjar Khyrdalan a a a GEDABAY GERANBOY AKHSU su su su Dashkesan GOBUSTAN GOBUSTAN Barda UDJAR s DASHKESAN Lake Tartar artar Tartar Kyurdamir ABSHERON BAKU BAKU M BARDA AR M E N I A Sevan ART TARTAR TARTAR K K K o ZARDAB KYURDAMIR u u u Zardab u u u HAGIKABUL n r r r To ttt a a a a Sevan AGDAM AGDZHEBEDI Kelbadzhar Kura SABIRABAD Gazi-Mammad i in i Agdzhebedi 40°N L 40°N To KELBADZHAR Sabirabad o o o Yerevan Agdam s s s KHODZHALY KHODZHALY IMISHLI l a Ali Bayramli w TURKEY Khodzhaly Saatly n ALI BAYRAMLI This map was produced by the Map Design Unit of The World Bank. Imishli d The boundaries, colors, denominations and any other information LACHIN Khodzhaveno BEILAGAN SAATLY SAATLY SAATLY shown on this map do not imply, on the part of The World Bank O Shusha SALYAN SALY SALYAN Group, any judgment on the legal status of any territory, or any N SADARAK Beilagan Araz VE Sadarak SHUSHA endorsement or acceptance of such boundaries. HA Lachin DZ Fizuli 50°E 51°E O Sharur FIZULI BILASUVAR Salyan KH SHARUR SHAKHBUZ Bilasuvar Gubadly Shakhbuz Dzebrail Neftchala Ar BABEK A ZE RBA IJ AN NEFTCHALA az GUBADLY DZEBRAIL GUBADLY JALILABAD NAKHCHIVAN NAKHCHIVAN Jalilabad DZHULFA Le Babek Zangilan e e Masally nk ORDUBAD ZANGILAN 39°N To RAYON CAPITALS Dzhulfa ar ar ar 39°N Salmas MASALLY MASALLY Ordubad an CAPITAL OF AUTONOMOUS REPUBLIC n n Yardymly Yardymly Lo YARDYMLY ARDYMLY YARDYMLY NATIONAL CAPITAL LENKORAN I SLA MIC REPUBLIC Lerik LERIK wl an Lenkoran RIVERS To To Tabriz O F IRA N d d d Tabriz MAIN ROADS 0 10 20 30 40 50 Kilometers ASTARA ASTARA RAILROADS Astara SEPTEMBER 2004 To IBRD 33365 Tabriz RAYON BOUNDARIES 0 10 20 30 40 50 Miles To INTERNATIONAL BOUNDARIES 45°E 46°E 47°E 48°E Rasht 49°E