KYRGYZ REPUBLIC Rural Education Project Report No. 114047 JUNE 27, 2017 © 2017 International Bank for Reconstruction This work is a product of the staff of The World RIGHTS AND PERMISSIONS and Development / The World Bank Bank with external contributions. The findings, The material in this work is subject to copyright. 1818 H Street NW interpretations, and conclusions expressed in Because The World Bank encourages Washington DC 20433 this work do not necessarily reflect the views of dissemination of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they represent. noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: The World Bank does not guarantee the World Bank. 2017. Project Performance accuracy of the data included in this work. The Any queries on rights and licenses, including Assessment Report: Kyrgyz Republic—Rural boundaries, colors, denominations, and other subsidiary rights, should be addressed to Education Project. Washington, DC: World information shown on any map in this work do World Bank Publications, The World Bank Bank. not imply any judgment on the part of The Group, 1818 H Street NW, Washington, DC World Bank concerning the legal status of any 20433, USA; fax: 202-522-2625; e-mail: territory or the endorsement or acceptance of pubrights@worldbank.org. such boundaries. Report No.: 114047 PROJECT PERFORMANCE ASSESSMENT REPORT Kyrgyz Republic RURAL EDUCATION PROJECT (IDA–HI370) June 27, 2017 Human Development and Economic Management Independent Evaluation Group ii Currency Equivalents (annual averages) Currency Unit = som 2000 US$1.00 som 47.7 2001 US$1.00 som 48.4 2002 US$1.00 som 46.9 2003 US$1.00 som 43.6 2004 US$1.00 som 42.6 2005 US$1.00 som 41.0 2006 US$1.00 som 40.2 2007 US$1.00 som 37.3 2008 US$1.00 som 36.6 2009 US$1.00 som 45.0 2010 US$1.00 som 45.8 2011 US$1.00 som 50.0 All dollar amounts are U.S. dollars unless otherwise indicated. Abbreviations and Acronyms ADB Asian Development Bank CEATM Center for Educational Assessment and Teaching Methods FY fiscal year GDP gross domestic product ICR Implementation Completion and Results report KAE Kyrgyz Academy of Education KPI key performance indicator MLA Monitoring Learning Achievements MoES Ministry of Education and Science NSBA National Sample-Based Assessment OECD Organisation for Economic Co-operation and Development PAD project appraisal document PDO project development objective PISA Programme for International Student Assessment PIU project implementation unit PPAR Project Performance Assessment Report REP Rural Education Project SIL specific investment loan UNICEF United Nations Children’s Fund Fiscal Year Government: January 1 – December 31 Director-General, Independent Evaluation: Ms. Caroline Heider Director, Human Development and Economic Management: Mr. Auguste Tano Kouame Manager, Corporate and Human Development: Ms. Marie Gaarder Task Manager: Ms. Susan Caceres Contents Principal Ratings ................................................................................................................. v Key Staff Responsible......................................................................................................... v Preface............................................................................................................................... vii Summary .......................................................................................................................... viii Rural Education Project ............................................................................................... viii Lessons ............................................................................................................................ x 1. Background and Context................................................................................................. 1 General Background and Context ................................................................................... 1 The Education Sector in the Kyrgyz Republic ............................................................... 2 2. Objectives, Design, and Their Relevance ....................................................................... 4 Project Development Objective ...................................................................................... 4 Relevance of Objective ................................................................................................... 5 Project Design ................................................................................................................. 6 Relevance of Design ....................................................................................................... 9 3. Implementation ............................................................................................................. 11 4. Achievement of the Objectives ..................................................................................... 13 Objective 1: Improving Learning Conditions in Primary and Secondary Schools, with Priority Attention to Rural Areas .................................................................................. 13 Objective 2: Improving Learning in Primary and Secondary Schools, with Priority Attention to Rural Areas. .............................................................................................. 16 5. Efficiency ...................................................................................................................... 21 6. Ratings .......................................................................................................................... 22 Project Outcome............................................................................................................ 22 Risk to Development Outcome ..................................................................................... 22 World Bank Performance ............................................................................................. 23 Borrower Performance .................................................................................................. 24 Monitoring and Evaluation ........................................................................................... 25 7. Lessons .......................................................................................................................... 26 Bibliography ..................................................................................................................... 27 Appendix A. Basic Data Sheet.......................................................................................... 33 Appendix B. Other Data ................................................................................................... 36 This report was prepared by Anthony Martin Tyrrell who assessed the project in January 2017. The report was peer reviewed by Ann Flanagan (Economist, IEG), and panel reviewed by Judyth L. Twigg, Consultant. Marie Charles and Aline Dukuze provided administrative support. iv Appendix C. List of Persons Met ...................................................................................... 41 Figures Figure 1. Poverty Distribution at the Regional Level, 2009 ............................................... 5 Figure 2. Distribution of All Grade 4 students Participating in the NSBA 2007, 2009, and 2014................................................................................................................................... 18 Tables Table 1. School Enrollment Rate, Kyrgyz Republic, 2007–11........................................... 3 Table 2. Ratio of Female-to-Male Enrollment in Education, Kyrgyz Republic ................. 3 Table 3. Rural Education Project Planned Components, Subcomponents, and Actions .... 7 Table 4. Appraisal and Actual Cost of the Rural Education Project by Component ........ 11 Table 5. Comparison of Mean Test Subject Score for Grade 4 Students in Issyk-Kul and Talas .................................................................................................................................. 19 Table 6. Results of the NSBA for Grade 8 Students in Issyk-Kul and Talas ................... 20 Table 7. Kyrgyz Republic Scores Compared with OECD Average PISA ....................... 21 Table 8. Indicators for the REP......................................................................................... 26 v Principal Ratings ICR* ICR Review* PPAR Outcome Satisfactory Satisfactory Moderately unsatisfactory Risk to Moderate Moderate Moderate Development Outcome Bank Performance Moderately satisfactory Moderately satisfactory Moderately unsatisfactory Borrower Moderately satisfactory Moderately satisfactory Moderately satisfactory Performance * The Implementation Completion and Results (ICR) report is a self-evaluation by the responsible World Bank global practice. The ICR Review is an intermediate Independent Evaluation Group (IEG) product that seeks to independently validate the findings of the ICR. Key Staff Responsible Task Manager or Project Leader Sector Director Country Director Appraisal Michael Mertaugh Maureen McLaughlin Dennis de Tray Completion Dingyong Hou Alberto Rodriguez Motoo Konishi vi IEG Mission: Improving World Bank Group development results through excellence in independent evaluation. About this Report The Independent Evaluation Group (IEG) assesses the programs and activities of the World Bank for two purposes: first, to ensure the integrity of the World Bank’s self-evaluation process and to verify that the World Bank’s work is producing the expected results, and second, to help develop improved directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, IEG annually assesses 20–25 percent of the World Bank’s lending operations through fieldwork. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which Executive Directors or World Bank management have requested assessments; and those that are likely to generate important lessons. To prepare a Project Performance Assessment Report (PPAR), IEG staff examine project files and other documents, visit the borrowing country to discuss the operation with the government and other in-country stakeholders, interview World Bank staff and other donor agency staff both at headquarters and in local offices as appropriate, and apply other evaluative methods as needed. Each PPAR is subject to technical peer review, internal IEG panel review, and management approval. Once cleared internally, the PPAR is commented on by the responsible World Bank country management unit. The PPAR is also sent to the borrower for review. IEG incorporates both World Bank and borrower comments as appropriate, and the borrowers’ comments are attached to the document that is sent to the World Bank’s Board of Executive Directors. After an assessment report has been sent to the Board, it is disclosed to the public. About the IEG Rating System for Public Sector Evaluations IEG’s use of multiple evaluation methods offers both rigor and a necessary level of flexibility to adapt to lending instrument, project design, or sectoral approach. IEG evaluators all apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (additional information is available on the IEG website: http://ieg.worldWorld Bankgroup.org). Outcome: The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently. The rating has three dimensions: relevance, efficacy, and efficiency. Relevance includes relevance of objectives and relevance of design. Relevance of objectives is the extent to which the project’s objectives are consistent with the country’s current development priorities and with current World Bank country and sectoral assistance strategies and corporate goals (expressed in poverty reduction strategy papers, Country Assistance Strategies, sector strategy papers, and operational policies). Relevance of design is the extent to which the project’s design is consistent with the stated objectives. Efficacy is the extent to which the project’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Efficiency is the extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least-cost compared with alternatives. The efficiency dimension is not applied to development policy operations, which provide general budget support. Possible ratings for outcome: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, highly unsatisfactory. Risk to Development Outcome: The risk, at the time of evaluation, that development outcomes (or expected outcomes) will not be maintained (or realized). Possible ratings for risk to development outcome: high, significant, moderate, negligible to low, and not evaluable. World Bank Performance: The extent to which services provided by the World Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of supported activities after loan or credit closing, toward the achievement of development outcomes). The rating has two dimensions: quality at entry and quality of supervision. Possible ratings for World Bank performance: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory. Borrower Performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development outcomes. The rating has two dimensions: government performance and implementing agency(ies) performance. Possible ratings for borrower performance: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory. vii Preface This is the Project Performance Assessment Report (PPAR) for the Rural Education Project in the Kyrgyz Republic, which was supported by a single investment loan (SIL- IDA H1370) of $15 million equivalent (special drawing rights 10.0 million). The Rural Education Project was approved on December 14, 2004, and fully disbursed on closing, March 31, 2011. The project did not undergo formal restructuring. This report was prepared by Anthony Tyrrell, senior consultant, under the guidance of Susan Caceres, senior education specialist. The findings are largely based on an eight-day mission to the Kyrgyz Republic (January 12–20, 2017), conducted by Anthony Tyrrell, as well as extensive desk review of available data and documentation. The mission met with representatives of the Ministry of Education and Science in the capital, Bishkek—in particular, representatives of the Kyrgyz Academy of Education, and the National Testing Center, both located at the ministry. While in Bishkek, the mission also met with representatives of the project implementation unit, other donors working in the sector (Asian Development Bank, the United States Agency for International Development, and nongovernmental organizations). The mission visited schools in the Issyk-Kul oblast region, one of the two pilot regions for the project, and met with school principals and other personnel involved in leadership, evaluation, and other capacities in the rollout of the REP. The mission met with regional, and local representatives of the Ministry of Education and Science in the Issyk-Kul oblast. Finally, the mission also met with key staff of the World Bank in Washington and Bishkek. The list of persons met is in appendix C. Preparation for the mission and report involved examination of (i) World Bank project files, (ii) project-related reporting documents and evaluations, and (iii) education studies with data from the government and other development partners, as well as the relevant research literature. The Independent Evaluation Group team gratefully acknowledges the logistical assistance and other support from Gulmira Sultanova (operations officer) and Meerim Sagynbaeva in the Bishkek Office of the World Bank, and provided by a local consultant, Aikynai Yusupova. Following standard Independent Evaluation Group procedures, a copy of the draft report will be sent to relevant government officials and agencies for their review and feedback. No comments were received from the Borrower. viii Summary General education in the Kyrgyz Republic comprises 11 grades, one less than in most developed countries. The system is robust in both enrollment rates and access from a gender equality perspective. The primary to lower secondary transition rate is universal, and the net enrollment rate in lower secondary is about 90 percent, and in higher secondary is about 72 percent. General education is taught in Kyrgyz or Russian in 90 percent of schools and in Uzbek in 9.4 percent of schools. Teaching is available in more than one language in about 25 percent of schools, often leading to parallel classes being taught in different languages within the same school, resulting in increased cost of education provision. Governance in the education sector was, and continues to be weak; the Ministry of Education and Science is nominally responsible for setting education policy and providing oversight in the education sector. However, aspects of the system operate largely independent of its oversight, leading to a disconnect between funding and policy execution with associated inefficiencies. At project design, the broader system, with particular reference to schools in rural areas in which 65 percent of the population reside, also faced many other challenges— such as outdated curriculum, poorly trained teachers, lack of textbooks, and lack of finance— such that project documentation described many of the existing primary and secondary schools as “barely alive” (World Bank 2004, 3). The general stagnation associated with relative policy neglect and lack of funding that occurred post-independence in a system designed to fit the Soviet model resulted in a significant decline in education quality even as measured within the confines of what was, in effect, an anachronistic curriculum and assessment system. Rural Education Project The project development objective for the $15.00 million Rural Education Project (REP, 2005–11) as stated in the development grant agreement and in the project appraisal document was “to assist the Recipient in improving learning and learning conditions in primary and secondary schools, with priority attention to rural areas.” Aspects of the project, for example, support for the development and provision of textbooks and national-level assessment, had a national focus. Other aspects, such as development of a formative assessment approach, were focused on two pilot oblasts, or regions (Issyk-Kul and Talas). The objective was supported by six components, as follows: improved teacher incentives; school subgrants for improved learning; textbooks and learning materials; student assessment; education budgeting and strategic planning; and project management. The components were realized, in turn, through many activities, including development of a performance management system and associated training of school management and education administrators at local levels; provision of support to schools for learning materials and resources (for example, maps, charts); implementation of learning improvement plans in schools in the pilot oblasts; development and production of textbooks; introduction of a formative assessment approach in the targeted oblasts; and, the development and nationwide dissemination of learning support materials, support for international, and national summative assessments, and support for the development of a per capita financing approach ix that sought to enhance the budgeting process and promote equity in financial support to schools and students across the country. As such, design and activity was diffuse and unfocused in this, the World Bank’s first entry into the education sector in the Kyrgyz Republic, noting the prior and ongoing involvement of other donors, including the Asian Development Bank, the United States Agency for International Development, and the United Nations International Children’s Emergency Fund. The relevance of objectives is high. In 2002, the World Bank was approached by the then- Minister of Education, who requested assistance in the education sector, with a particular focus on schools in rural areas. The World Bank’s country assistance strategy for fiscal year (FY)03–06 noted that poverty surveys had identified the deteriorating quality of basic education as a key concern of the rural poor, and the 2003 National Poverty Reduction Strategy adopted the goal of stemming the deterioration in education quality. The objective continues to be relevant. The Kyrgyz education strategy for 2012–20 states that the educational system in 2020 will be the main tool for the promotion of social and political development and will ensure the country’s competitiveness in regional and international processes. The World Bank Group’s country partnership strategy FY14–17 notes that, despite significant increases in public spending on education, outcomes continue to be unsatisfactory (World Bank 2013b, 13). Relevance of design is modest, with particular reference to a lack of prudent selectivity given the known capacity limitations within the system and the range and relative success (or otherwise) of interventions supported by other donors. Also noted is the project’s packed agenda and the fact that the project represented the World Bank’s first entry into the education sector in the Kyrgyz Republic. Although there is insufficient reliable evidence to support the attainment of improved learning, the project realized various, albeit lesser-order, outcomes that contributed to enhancing the conditions under which improved learning may be realized. Key among these were the raising of consciousness, awareness, and public debate regarding the extent of the challenges facing the education sector (through support for international- and national-level assessments that presented clear evidence to that effect). These efforts had the effect of generating greater consensus, or policy clarity, regarding what needed to be done. The piloting of a per capita financing approach provided valuable input toward the development of a national scheme for a more equitable approach to funding of schools that will enhance equity (in financing arrangements) once implemented in full (partially implemented to date). The project also supported inputs such as textbooks, and learning materials, as well as well- implemented pilots (if not ultimately systematically implemented at national level) on school planning, formative assessment, and teacher incentives. That said, fundamental challenges persist with, for example, reference to governance and financing, teacher training, education content, standards, and assessment. Project outcome is rated moderately unsatisfactory. That rating is based on high relevance of the objectives, modest relevance of the design, modest achievement in improving learning, and substantial achievement in improving learning conditions. Efficiency was rated modest. Although no cost-benefit analysis was undertaken, the project did deliver a significant quantum of outputs within a modest budget. Budget reallocations, including the reallocation x to support a second round of the Programme for International Student Assessment (PISA) in 2009, were effective. The risk to development outcome is moderate, given fiscal constraints, continuing weaknesses in governance, and the possibility of political volatility associated with the 2017 presidential election. The performance of the World Bank is rated moderately unsatisfactory at entry, moderately satisfactory during supervision, and moderately unsatisfactory overall. The borrower’s performance is rated moderately satisfactory. The government remained committed to improving learning and learning conditions, but there were significant and frequent changes in the administration leading to inefficiencies and slowed progress. Both government and implementing agency performance were moderately satisfactory. Lessons The evaluation of the REP yields several key lessons. A selective, and realistic approach to design is likely to create a more closely articulated link (attribution) between World Bank interventions and outcomes. In this case, expert and thorough diagnosis of need and associated challenges was not accompanied by a realistic assessment of what was possible, taking into account constraints such as the political economy, commitment, capacity, resources (human and financial), and available time. Project design should be calibrated in line with these and other relevant considerations, noting that the REP was formed and implemented in a volatile political environment and within a system with clearly identified and acknowledged capacity limitations. Relatively inexpensive but strategically positioned interventions can have a significant effect. World Bank support for the implementation of the 2006 and 2009 PISA assessments, a minor cost in the context of the overall project, had such an effect. The PISA results shocked the system—shock being the term used by almost all those interviewed during the mission to describe the PISA effect. The results of the PISA assessments (as well as the home-based National Sample-Based Assessment) jolted the system into what is, basically, a process of culture (rather than simply system) change, forcing a national dialogue on the poor quality of outcomes from education quality and required remedies. Robust monitoring and evaluation is a fundamental component of project management and is integral to the articulation of, and sometimes to the realization of, desired outcome. In the case of the REP, monitoring and evaluation was poorly designed and implemented, thereby limiting the capacity to articulate the ultimate value of the project— and also representing a lost opportunity with reference to building much-needed capacity within the Ministry of Education and Science. It is critical to promote and nurture collaboration with other donors in pursuit of common, strategic goals. In this case, representing the World Bank’s first entry into the education sector in the Kyrgyz Republic, the team was well appraised of the work of other donors already engaged in the Kyrgyz education sector. As a newcomer to the sector under invitation from the then-Minister for Education, and given its broader convening role, the World Bank was well positioned to carefully build on or into existing efforts. However, certain of the World Bank–supported efforts added to the “patchwork” quality of donor xi efforts in the Kyrgyz education sector, with different players piloting different approaches to effectively the same challenge (for example, formative assessment or curricular reform) in different geographic locations, with little ultimate effect at the system level. Auguste Tano Kouame Director Human Development and Economic Management Department 1 1. Background and Context 1. This Project Performance Assessment Report (PPAR) reviews the experience and achievements of World Bank support for various measures across the education system in the Kyrgyz Republic through the Rural Education Project (REP). The REP was approved on December 14, 2004, and became effective on May 2, 2005. The REP was financed by a specific investment loan (SIL) of $15.0 million equivalent provided by the International Development Association. The Government of the Kyrgyz Republic provided match funding of $0.5 million equivalent. 2. The project was selected for a field-based assessment because of its focus on student assessment and improving the quality of education. Parallel field-based assessments were undertaken for projects in Brazil and the Lao People’s Democratic Republic that share this project’s (partial) focus on improving respective national-level assessment systems. In addition to the stand-alone PPARs in each instance, learning from all three project-level evaluations is being combined to feed into an Independent Evaluation Group (IEG) learning engagement being carried out in partnership with the Education Global Practice over the course of fiscal year (FY)17. General Background and Context 3. The Kyrgyz Republic is a landlocked, mountainous country in Central Asia neighbored by China, Kazakhstan, Tajikistan, and Uzbekistan. A parliamentary democracy with a multiethnic (Kyrgyz, Russian, and Uzbek) population of almost six million people, the country is one of a small number of lower-middle income countries in the Europe and Central Asia Region. In 2015, gross national income per capita was $1,170. The country’s poverty rate ($5 per day in 2005 Purchasing Power Parity terms) was 83.9 percent in 2014. 1 4. The Kyrgyz Republic gained independence from the former Soviet Union in 1991 and, in that year, joined the Commonwealth of Independent States. In 1992, the state joined the United Nations and the World Bank, as well as the Conference on Security and Co- operation in Europe, the predecessor of the Organization for Security and Co-operation in Europe. In 1993, economic reforms were launched leading to introduction of the Kyrgyz som as the unit of currency, replacing the Russian ruble. Initially, the country stood out among the newly independent Central Asian republics for its sound, multiparty democratic system. Post- 1991, the Kyrgyz Republic became relatively open, as evidenced in particular by an outspoken civil society and, relatively speaking, a thriving free press. However, by the mid- 1990s, the economy floundered, and many people moved to the neighboring resource-rich countries of Kazakhstan or the Russian Federation to seek employment, often with a view to sending remittances back home. 5. In March 2005 (just as the REP was due to become effective) then-President Askar Akayev was forced to step down following increasingly angry protests against perceived corruption and misgovernance. He was replaced by President Kurmanbek Bakiyev, who came to power with the stated agenda of reforming the country and ending corruption. However, over time, similar charges to those leveled against the former president were leveled against President Bakiyev. Journalists and opposition politicians were subject to 2 harassment and imprisonment, and discontent over the general state of affairs came to a head in April 2010 (the REP closed 31 March, 2011), resulting in violent protests and, eventually, the ousting of President Bakiyev. Roza Otunbayeva, a former foreign minister who led the opposition, stepped in to lead the country on an interim basis. She served until President Almazbek Atambayev was inaugurated in December 2011, marking the first peaceful transfer of presidential power in independent Kyrgyz Republic’s history. 6. The economy of the Kyrgyz Republic since independence has been as turbulent as its politics, often mirroring instability in that dimension. Between 1990 and 1995, the country’s gross domestic product (GDP) almost halved as it adjusted to the new post-Soviet reality. In 1996, the economy began to recover, driven by reforms in the agriculture sector and by gold production. The economy slowed again between 1998 and 1999, before returning to a more sustainable growth in 2000, with GDP growth rates in the range of 5–7 percent per year; however, technological problems faced by Kumtor Gold Mine (a large enterprise on which the economy relied heavily) and unfavorable climate conditions that affected agriculture production, resulted in very low rates of growth 2002–05. The economy grew again prior to the global economic crisis—GDP cumulatively increased by 16.8 percent in 2008–09. The immediate effect of the crisis on the Kyrgyz Republic was moderate, manifesting itself primarily through a decline in remittances and exports, although GDP remained positive. 2 7. The FY14–17 World Bank Group country partnership strategy for the Kyrgyz Republic notes the country’s economic performance over the preceding decade was volatile, reflecting several domestic and external shocks (World Bank 2013b). The country partnership strategy found that the economy was characterized by significant informality, reliance on a few sectors, and remittances. The prudent fiscal stance adopted until 2008 was replaced by significant expansion, initially in response to the global crisis and later to a domestic crisis resulting in budget deficits averaging 5.4 percent of GDP between 2010–12, one of the highest in the Europe and Central Asia Region. The Education Sector in the Kyrgyz Republic 8. General education in the Kyrgyz Republic comprises eleven grades, one less than in most developed countries. Children typically start primary education at the age of six or seven, and primary education lasts four years (grades 1–4), followed by five years of middle school and two years of high school; participation in school is compulsory up to the end of grade 9. Pupils in general education can attend primary schools (grades 1–9), basic schools (grades 1–11), or middle and high schools (grades 5–11). Currently, about 80 percent of schools are basic schools, enrolling pupils from grades 1 to 11. 3 9. Tables 1 and 2 show that the education system in the Kyrgyz Republic is robust in terms of (i) enrollment rates, and (ii) access from a gender equality perspective, both positive legacies of the former Soviet system. At the primary level from 2007 to 2011, school enrollment in the Kyrgyz Republic (98.9 percent gross) is not far off average rates of enrollment compared with other countries in the Europe and Central Asia Region (102.4 percent gross), and across the world (106.0 percent gross). At the secondary level, the Kyrgyz Republic performs less well (85.5 percent gross) than the Europe and Central Asia Region average (96.7 percent gross) but much better than the global average (51.0 percent 3 gross). The primary to lower secondary transition rate is universal. Table 1 shows effective gender parity in the ratio of males to females enrolled in primary and secondary education. General education is taught in Kyrgyz or Russian in 90 percent of schools and in Uzbek in just over 9 percent of schools (a small number of schools have classes in Tajik). Because of the level of linguistic diversity, teaching is available in more than one language in about 25 percent of schools. Parallel classes are often taught in different languages within the same school, and this increases the cost of education provision. Table 1. School Enrollment Rate, Kyrgyz Republic, 2007–11 SCHOOL ENROLLMENT RATE: KYRGYZ REPUBLIC 2007 2008 2009 2010 2011 School enrollment, primary (% 97.9 98.1 98.3 99.6 100.9 gross) School enrollment, secondary 86.8 85.6 84.4 84.0 88.2 (% gross) Source: World Bank 2013a, 2013b. Table 2. Ratio of Female-to-Male Enrollment in Education, Kyrgyz Republic School Level Year 1990 1995 2000 2005 2011 Primary 102.0 101.0 98.0 99.0 99.0 Secondary 99.0 - 103.0 100.0 100.0 Tertiary - 125.0 101.0 124.0 124.0 Source: World Bank 2013a, 2013b. 10. The governance system in the education sector is fragmented. The Ministry of Education and Science (MoES) is nominally responsible for setting education policy and providing oversight in the education sector, but aspects of the system operate largely independent of it. For example, in general secondary education, schools are typically managed by local authorities, with little central oversight. A 2014 World Bank Public Expenditure Review for the education sector notes that the separation of financing and school management responsibilities between the central and local levels, respectively, is not an issue per se (World Bank 2014). However, because (i) central funding is regulated by norms that are not consistent or enforced at the local levels and (ii) the availability and sharing of feedback information is scarce, the ensuing disconnect between funding and execution realizes inefficiencies and inadequate provision of real inputs in schools. 11. The 2004 project appraisal document (PAD) for the REP paints a bleak picture of the education sector at project inception. 4 Many of the challenges outlined in the PAD were associated with the post-1990 transition and subsequent economic and political volatility, which robbed what had been a stable and established (if outdated) system of resources. In fact, the PAD suggests that existing primary and secondary schools and higher education institutions were only “barely alive” and that the quality of education had declined seriously except in a small number of largely urban schools that benefit from significant parental and community contributions, or have received assistance from external donors (World Bank 2004, 3). 5 Due to budgetary constraints, the government reduced the duration of compulsory 4 education from 11 years to 9 years, and shifted much of the responsibility for financing public primary and secondary education to local governments and parents. The government also authorized schools to raise money from the rental of facilities and other activities and required students to purchase or rent textbooks that had formerly been provided free (Mertaugh 2004). The effect of this policy had particularly negative consequences for rural schools. Local authorities often lacked the capacity to make up the difference between the central contribution and the funding required to maintain a properly functioning school system because the limited tax base was already strained and parental contributions were meager given the concentration of poverty in rural areas (75 percent of the poor; World Bank 2004, 3). 12. The PAD outlined several other challenges. Sixty-five percent of the population lived in rural areas, presenting a challenge for education delivery and knock-on effects in terms of cost (for example, smaller class and school sizes). The rural concentration of the population and associated poverty also affected demand for education (e.g. affordability of textbooks), and reduced teacher incentives. In urban areas, teacher salaries and schools were supplemented by contributions from parents and local governments, whereas rural teachers received only their official salary from the central budget—less than 40 percent of average earnings in public administration. There was little community involvement and school accountability, particularly in rural areas where parents and local governments lacked the means to supplement the running costs of schools, which deprived them of leverage for school accountability. An outdated approach to teaching and learning characterized by rigid, teacher-centered methods and a pedagogy stressing the memorization of facts also held up progress. Finally, inefficient and ineffective process of textbook provision and poor content in existing textbooks—the Kyrgyz Academy of Education (KAE) dominated the process of authorship, selection, and production of textbooks, creating “an intrinsic conflict of interest”— presented a challenge. 6 13. The PAD for the ongoing support for the education reform project (P113350) that was approved in 2013 reflects on continued poor-quality education outcomes, despite high levels of public spending on education (World Bank 2013c). 7 That PAD echoes many of the issues and challenges outlined in the PAD for the REP, which was written nine years earlier. 8 2. Objectives, Design, and Their Relevance Project Development Objective 14. The project development objective (PDO) is consistently framed in both the development grant agreement (12) and the PAD as follows: “to assist the Recipient in improving learning and learning conditions in primary and secondary schools, with priority attention to rural areas” (World Bank 2005, 12; World Bank 2004, 8). 15. Within the PDO, there are two distinct objectives: one relating to the improvement of learning and the other to the improvement of learning conditions, noting that interventions designed to improve learning conditions must, ultimately, be linked to learning outcomes (that is, they are not an end in and of themselves). The sole key performance indicator (KPI) is squarely associated with improved learning outcomes (that is, improved average scores in 5 learning assessments in mathematics, science, and Russian / Kyrgyz / Uzbek for students in grades 4 and 8). 9 16. In pursuit of its objectives, the project operated at both national and oblast levels (see table 3 in the Project Design section for detail regarding distribution of components in that regard). National-level assessments of grade 5 students showed results in literacy and numeracy skills varied greatly by oblast, with the poorest results evident in rural areas. The poorest performing oblast was Batken, followed by Talas and Issyk-Kul. The government opted not to include Batken as a pilot oblast because there were other, ongoing donor initiatives there. Issyk-Kul and Talas were selected because they had, respectively, the second, and third-lowest levels of student performance (World Bank 2004, 8). It is notable, however, that although each of the selected oblasts had a higher-than-average share of poor households, they were not the poorest oblasts overall (figure 1, panels a and b). 10 Figure 1. Poverty Distribution at the Regional Level, 2009 a. Share of Oblast’s Population in b. Distribution of Poor Across All Poverty Oblasts 25 50 46 44 45 20 37 38 40 31 33 35 15 30 25 21 10 20 14 15 10 10 7 6 5 3 1 2 3 2 5 0 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 Absolute Extreme Absolute Extreme Source: World Bank 2011a. Note: 1. Bishkek; 2. Issyk-Kul; 3. Jalal-Abad; 4. Naryn; 5. Batken; 6. Osh; 7. Talas; 8. Chui. 17. Figure 1, panels a and b, shows that, whereas high concentrations of poor people live within the populations of each of the targeted oblasts (Issyk-Kul has the highest concentration at 46.1 percent, Talas the fifth highest at approximately 33.0 percent), the majority of poor people across the population as a whole reside in the Jalal-Abad and Osh oblasts (Figure 1, panel b). In fact, their 520,000 and 383,000 poor people, respectively, represent more than half of all poor people in the Kyrgyz Republic. Issyk-Kul has the third highest proportion of poor people (12 percent) whereas Talas has the lowest (4 percent). Relevance of Objective 18. The objective was relevant at the time of approval and remained relevant at both project closure (2011) and the time of the IEG mission for this evaluation (January 2017). The objective is supported by the sole KPI for the project. 6 19. The government, concerned about declining quality, approached the World Bank to request assistance in the education sector, and in October 2002, specifically requested a project to help improve the quality of rural education. Following the government’s request for support, the World Bank team consulted with other donors to explore how the proposed REP could best dovetail with and complement ongoing interventions and, in so doing, ensure added value and relevance (World Bank 2004, 6–7). The results of the second Monitoring Learning Achievements (MLA) survey in 2005 reinforced the challenge associated with education quality. It showed that the quality of learning outcomes (the core of the project’s focus) from primary education had actually gone down between 2001 and 2005 (see appendix B). 20. The country assistance strategy for FY03–06 notes that poverty surveys had identified the deteriorating quality of basic education as a key concern of the rural poor, as had the National Poverty Reduction Strategy, which adopted the goal of stemming the deterioration in education quality and increasing attendance in primary schools from 90 to 97 percent by 2005 (World Bank 2003a, 23, 2003b). The PAD also refers to the National Survey of Primary Education Quality carried by United Nations Educational, Scientific, and Cultural Organization out in November, 2000 that found that 8 percent of rural schools lacked electricity, 45 percent lacked adequate heating, and 78 percent lacked adequate furniture. In addition, the country assistance strategy also states that the Government of the Kyrgyz Republic, the World Bank, the Asian Development Bank (ADB), 11 and UNICEF (United Nations Children’s Fund) agreed to work closely to develop a shared strategy and approach focusing on critical needs such as teacher training and textbooks, and that the World Bank would prepare the REP with a particular focus on rural communities in the poorest provinces. 12 21. Official recognition of the relevance and importance of the project objective is, perhaps, even more explicitly understood in the post-project period. In that regard, the Education Development Strategy of the Kyrgyz Republic, 2012–2020, represents a significant step in the development and statement of consciousness regarding education quality, suggesting the educational system in 2020 will be the main tool for the promotion of social and political development (Kyrgyz Republic 2011). The country partnership strategy for FY14–17 notes that, despite an increase in public spending on education from 3.9 percent to 7.1 percent of GDP during 2001–11, outcomes in terms of quality have been unsatisfactory (World Bank 2013b, 13). National assessments show fully half of all grade 4 and 8 students score below basic competency levels in mathematics, reading, and sciences. 22. The relevance of objectives is rated High. Project Design 23. From a project design perspective, this evaluation emphasizes three overarching factors in relation to the REP, which was designed as an SIL. First, the Kyrgyz education system faced many high-level system and capacity challenges (for example, governance and strategic planning capacity) as well as more rudimentary challenges including, for example, the state of the physical infrastructure, lack of teaching materials, and few textbooks. Given the extent of need, there was a significant range in the set of interventions that would have 7 been notionally relevant; however, the strategic, development potential of any given intervention, or set thereof, would necessarily be determined within the context of the overall education system and its relationship to broader political and administrative systems. Second, the REP represented the World Bank’s first entry into the education sector in the Kyrgyz Republic. That alone would suggest the need for a cautious approach and close engagement with not only the client but other donors (ADB, USAID, UNICEF) who were already active in the sector and other key stakeholders on whom the success or otherwise of the project would depend (for example, local and regional structures in the education system, teachers, teacher training entities, curriculum development specialists). Finally, the available World Bank budget for the project in the face of the extensive challenges facing the education system in pursuit of improved learning quality was $15 million. That limitation would also suggest, from the outset, the need for a strategic and targeted approach well aligned with the work of other donors. COMPONENTS 24. The project was designed as an SIL comprising six components: improved teacher incentives, school subgrants for improved learning, provision of textbooks and learning materials, student assessment, education budgeting and strategic planning, and project management. Within the six components were 18 subcomponents (within which there were sub-activities) as set out in table 3. Certain components aimed to address relevant matters at the national level, whereas others focused more on two rural oblasts, Talas and Issyk-Kul. Given the challenges outlined in the PAD, as well as the focus and aim of the project, the packed project agenda is notable. Along with the national versus oblast concentrations, some components dealt with primary and some with secondary levels, and others looked at improving capacity at a central level (the MoES), whereas others concentrated on capacity at the local level. Table 3. Rural Education Project Planned Components, Subcomponents, and Actions Components, Targeting, Planned and Actual Cost Subcomponents and Actions Component 1: (i) Improve existing school-based self-appraisal procedures for teachers and Improved teacher school principals through the development of model performance incentives management arrangements to elaborate the new performance criteria (ii) Develop improved national guidelines and criteria for performance Region specific and evaluation; the new criteria would address student progress and improve national comparability of school performance (iii) Develop a performance incentive scheme structured to map onto the Planned: $2.5 million prevailing salary scale for teachers and principals, and a pilot implementation Actual: $2.8 million of the scheme in the pilot oblasts (iv) Finance rural teacher fellowships for three cohorts of 200 new teachers per year for a duration of three years (v) Provide technical assistance and train teachers, school principals, and education administrators in the implementation of the new performance management system 8 Components, Targeting, Planned and Actual Cost Subcomponents and Actions Component 2: (i) Learning improvement subgrants: Assistance would be provided to school School subgrants for teams comprising teachers, principals, and parents in the pilot oblasts to improved learning prepare learning improvement plans to describe most urgent learning needs and proposed action programs to address those needs Region specific (ii) Technical assistance: Supported orientation and training of school teams and rayon (district) and oblast staff, as well as hands-on support to schools Planned: $1.8 million and communities to prepare and implement viable learning improvement Actual: $2.4 million plans, including microprojects Component 3: (i) Textbook provision to fund (a) the development of a new generation of Textbooks and textbooks and teachers’ manuals; (b) reprints of existing textbooks and learning materials teachers’ manuals; and (c) the development and publication of new textbook manuscripts National (ii) Improvement of the textbook rental scheme to provide financing to improve the scheme to ensure adequate financing of textbook replacement Planned: 6.7 million and to establish a textbook rental fund as an autonomous body with a Actual: 6.1 m managing board (iii) Provision of learning materials (for example, library books, maps, globes) Component 4: (i) Improvement of school-based and classroom-based formative assessment Student assessment to promote “assessment for learning” through the development, field testing, and dissemination of teacher support materials National (ii) Improvement of existing examinations to assist the Minstry and the KAE to (a) develop and introduce more challenging questions in the national Planned: $1.6 million examinations that would allow students to demonstrate their ability to apply Actual: $2.2 million reasoning and independent thinking skills and (b) make national examinations a better gauge of student ability by abolishing the prevailing assessment practice of using only previously available test questions (iii) periodic sample-based national surveys of student achievement to (a) finance and support at least three follow-up periodic sample-based national assessments of the same skills, based on the same sampling frame and (b) support improvement of test items for these sample surveys (iv) Support participation in the 2006 round of the OECD PISA survey, including the routine national and international costs of PISA participation (v) Capacity building in modern practices and techniques in student assessment to strengthen the assessment function of the Ministry by building capacity and specialist expertise and by establishing a sustainable, identifiable professional and technical unit. Component 5: (i) Capacity building to support training technical assistance and office Education budgeting equipment and software to help establish a new Education Budgeting and and strategic Strategic Planning Unit in the Ministry planning (ii) Commissioned policy studies to support the preparation of four annual cycles of papers on priority topics of education budgeting and strategy National (central (iii) Monitoring and evaluation to support for (a) technical assistance, studies, government) and surveys to monitor and evaluate project outcomes and broader educational trends and (b) actions to provide additional information on the Planned: $1.8 million 9 Components, Targeting, Planned and Actual Cost Subcomponents and Actions Actual: $1.4 million outputs and outcomes of the project, as well as information on broader educational trends. Component 6: Finance (a) minor refurbishment of the PIU office space, (b) adequate office Project management equipment and supplies, and a project vehicle; (c) technical assistance and training for PIU staff in project management, procurement, and financial Planned: $0.9 million management; (d) annual external audits of the project; (e) a public Actual: $0.9 million information specialist support to the project; and (f) incremental operating cost for the PIU. Source: World Bank 2004. Note: a. The textbook rental scheme was dropped as a result of a parliamentary decree through which the government decided to finance new textbooks in accordance with the country’s legislation to provide all children with a free compulsory education. As such, the plan to fund the rental scheme was removed from the development grant agreement (June 2009 amendment). KAE = Kyrgyz Academy of Education; MoES = Ministry of Education and Science; OECD = Organisation for Economic Co-operation and Development; PISA = Programme for International Student Assessment; PIU = project implementation unit. Relevance of Design 25. The relationship between project components and activities in pursuit of project objectives was disjointed. There was, notionally, a link between project inputs and intended outcomes, but given the low capacity environment, design lacked strategic focus, prioritization, and effective sequencing of interventions. The project’s six components and 18 subcomponents spanned multiple areas of policy (for example, teacher incentives, textbook content and provision, teacher training, in-class assessment, national-level assessment) relevant to achieving the PDO; however, within the confines of available time and resources, and given the range and scope of challenges in the education sector, project design was unrealistic. 26. Analysis and engagement, as evidenced through analytical work, aide-mémoire, and the PAD, make it clear that the project team was well appraised of the scale of the challenges. The team was also aware of what other donor organizations (ADB and USAID) were doing, and what they had been able (and unable) to achieve. 13 And, finally, the team was aware of the financial and human resource base available to it over the limited life of the project. There was no lack of analysis or diagnosis. 14 27. To achieve the broad objective of improved learning (objective 1) would clearly have required improved learning conditions (objective 2). However, based on the available analysis of the education sector and its challenges, to achieve the first objective on a sustainable basis and ensure ongoing support and development of successful pilots in the second would have required significant intervention in the governance and policy-making domain to ensure a systematic and coherent pursuit of improved learning. The available analysis suggested the center of the education system did not possess the resources and capacity to build on, mainstream, and use learning on an ongoing basis; however, the project proposed to do very little to address this core constraint. There was also no explicit commitment by the client to mainstream elements of the project, regardless of how 10 successful they may have been. 15 Some capacity-building activities at central level were incorporated into project design (components 5 and 6), but these were limited in scope. 28. Certain interventions—for example, research into and the piloting of a per capita financing approach and the provision of support for a Program for International Student Assessment (PISA)—had strategic potential. The former could have enhanced systemwide budgeting and a more equitable distribution of limited resources, and the latter could have better defined the relative scale of the challenge and raised political and public awareness accordingly. Other, typically more expensive interventions, had limited strategic potential. 29. Given the lack of selectivity and focus, relevance of design is rated Modest. IMPLEMENTATION ARRANGEMENTS 30. Implementation arrangements were set out in the PAD, recognizing the MoES as the executing agency for the project (the minister was designated national coordinator), with strategic orientation and oversight to be provided by a National Council (World Bank 2004, 18–19). 16 The council would meet every six months and provide oversight to other donor activities in the sector with a view to ensuring coordination. Project Coordination Committees would also be set up in each of the two pilot oblasts to provide local-level oversight and coordination. Within the MoES, project management would be carried out by the newly established REP implementation unit (PIU), which was also implementing project preparation activities financed through a Policy and Human Resources Development Grant from Japan. 17 31. The PAD also recognizes the need for coordination with other donors (particularly with ADB and USAID, both already active in the sector) in preparing and implementing the REP through, for example, consultations during missions; joint missions (IEG could not source evidence of the conduct of joint missions); videoconferences; and sharing of mission terms of reference, and aide-mémoire. However, the level of envisaged coordination did not stretch to implementing a joint ADB/World Bank PIU as requested by the Ministry, noting both units were housed at the ministry and continued to operate independently of each other at the time of the mission. 32. Each project component would have a designated working group, established by ministerial order, to assist in project preparation. Lead departments within the ministry would be responsible for the respective components or subcomponents. For example, the Budget Department was responsible for the implementation of improved teacher incentives; the PIU for implementation of the school grants for improved teaching and learning; the Educational Materials Department for textbooks and learning materials; and the Student Assessment Unit for student assessment activities. The education budgeting and strategic planning component would report directly to the minister, and the PIU (with technical assistance support) would cover arrangements for project implementation, including procurement and financial management requirements. The PIU would also prepare the project operational manual and a separate operations manual for component 2 (school grants for improved teaching and learning). 11 3. Implementation 33. Key dates. Approved by the Board of Executive Directors on March 14, 2002, the project was expected to begin March 31, 2005. However, President Akayev was pushed out of power on March 17, 2005, delaying planned project start. The project became effective on May 2, 2005. The midterm review took place in October 2007. The project was not formally restructured, but its closing date was extended by one year (to March 31, 2011) to provide additional time for the completion of (i) fellowship payments to new teachers who had completed three years’ service in rural schools and (ii) to finance Kyrgyz participation in a second Organization for Economic Co-operation and Development (OECD) PISA. Planned versus actual costs, financing and disbursements. The total cost estimated at appraisal was $15.50 million equivalent. The actual cost was $15.80 million, which is $0.30 million higher, the extra becoming available through currency appreciation (World Bank 2011b, 32). During the project, funds were reallocated twice among expenditure categories. The first reallocation occurred in December 2007 and the second in September 2009. Resources were reallocated from component 5 (education budgeting and strategic planning) to component 1 (improved teacher incentives), component 2 (school subgrants for improved learning), and component 4 (student assessment). Resources were also moved between cost categories, from works and training to goods, equipment, or materials; sub-grants; consultants’ services; and teacher incentives. After the textbook rental scheme was dropped, resources were shifted (within the same component) to textbook provision and learning materials. 18 Table 4. Appraisal and Actual Cost of the Rural Education Project by Component Component and Appraisal Estimate Actual Cost Percentage of Contingency (US$, millions) (US$, millions) Appraisal Component 1: Improved 2.4 2.8 116.7 teacher incentives Component 2: School 1.8 2.4 133.3 subgrants for improved learning Component 3: 6.1 6.1 100.0 Textbooks and learning materials Component 4: Student 1.5 2.2 146.7 assessment Component 5: 1.7 1.4 82.4 Education budgeting and strategic planning Component 6: Project 0.9 0.9 100.0 management Total baseline Cost 14.5 15.8 109.0 Physical Contingencies 0.8 n/a 0 Price contingencies 0.2 n/a 0 Total project cost 15.5 15.8 101.9 12 FACTORS AFFECTING IMPLEMENTATION 34. Extraneous factors. The project was, more or less, book-ended by political crises. Project start was delayed due to a political crisis that flared in March 2005 resulting in the ousting of then-President Akayev, shortly after project approval in December 2004 (planned project effectiveness date was March 31, 2005). The project eventually got under way in May 2005. In April 2010, antigovernment political demonstrations took place against President Akayev’s successor, President Bakiyev, fueled by a widespread belief that corruption, especially nepotism, and misuse of public assets had increased. A new government eventually took office following presidential elections in October 2011. 35. Over the almost six years of project implementation, frequent changes in political leadership influenced priority setting, decision making, and efficiency; the Kyrgyz Republic had three presidents, six changes in the administration, and seven changes at the ministerial level for the education portfolio (World Bank 2011b, 15–16). However, activity pushed ahead due to the determined efforts of the PIU, component coordinators, and many others, including education officials at regional and local levels, trainers, school administrations, and individual teachers. 36. Factors within government control. The textbook rental scheme, to be financed under component 3, was dropped as a result of a parliamentary decree through which the government decided to finance new textbooks in accordance with the country’s legislation, that is, to provide all children with a free compulsory education. As such, the plan to fund the rental scheme was removed from the development grant agreement (June 2009 amendment). 37. Factors within the control of the REP. Initially, the poor capacity of the PIU resulted in delays and bottlenecks. The PIU experienced high staff turnover and lacked capacity in certain areas, such as procurement, which caused delays. 19 The PIU also had three directors over the course of the project. However, the PIU demonstrated improved performance over time, as attested to by interviewees for the evaluation mission (January 2017) who had high regard for the now very experienced PIU currently implementing the World Bank’s sector support project. The Implementation Completion and Results report (ICR) notes that there was no change in component coordinators during the entire time of the REP. These committed and competent midlevel professionals significantly contributed to the achievements of the project and to developing cross-referencing between the various project components, for example, linking teacher training in formative assessment and performance assessment (World Bank 2011b). 38. The plan to build on the MLA assessment series to track student performance (and thus substantiate the KPI) was undone when, subsequent to project start, the World Bank decided the methodology supporting the MLA was not adequate. This resulted in a new baseline being established to facilitate tracking through the National Sample-based Assessment (NSBA; implementation of which was supported by the project in 2007 and 2009). Midterm review was conducted in October 2007. The question of the sustainability of project interventions was the primary concern of the midterm review. In that regard, the midterm review raised issues such as (i) political pressures that may arise to give all teachers the same salary increases rather than implementing an incentive payment system that some 13 might perceive as unfair and (ii) the pressure that may develop during the teacher assessment process to categorize all teachers as high performers with attendant budgetary implications for ongoing salary payments. As elaborated below, the midterm review also concentrated on the development of monitoring and evaluation (M&E) arrangements for the project. 39. Safeguards compliance. Both an environmental and social assessment (the latter to better understand the education system) were carried out for this class B project. The only environmental safeguard policy triggered by the project was Environmental Assessment (OP/BP/GP 4.01). 20 The ICR confirms there were no significant deviations or waivers from World Bank safeguards during project implementation. 40. Fiduciary compliance. An assessment of the financial management arrangements— accounting, financial reporting, auditing, and internal controls of the PIU—was undertaken in October 2004. The World Bank advised the PIU on what it needed to put in place to complete an acceptable financial management manual, which was required prior to Board Submission. The internal control system at the PIU ultimately met World Bank requirements, and was assessed to be capable of providing timely information and reporting under the REP project. The ICR states that Financial Management Reports for the project were usually received on time and found to be acceptable, as were audit reports with unqualified (clean) opinion on financial statements. Procurement of all works, goods and technical services under the project followed the Procurement Guidelines “Procurement under [International Bank for Reconstruction and Development] Loans and IDA Credits”, and procurement planning was found to be adequate, and to be supported by approved procurement plans that were published annually on the Image Bank website. Contracts and payments were well managed, as was the management and maintenance of associated payment documents and records, acceptance protocols, and other supporting documentation. 4. Achievement of the Objectives 41. There are two objectives within the PDO. The first objective relates to the improvement of learning, and the second to the improvement of learning conditions. Because the pursuit of improved learning conditions effectively “serves” the achievement of improved learning (the ultimate purpose of improved learning conditions being improved learning outcomes), the presentation below will address that second objective, first. Also as stated above, there is only one KPI for the project as a whole, and it relates to improved learning— improved average scores in learning assessments in mathematics, science, and Russian/Kyrgyz/Uzbek for students in grades 4 and 8—noting that no specific target was set for this indicator. Objective 1: Improving Learning Conditions in Primary and Secondary Schools, with Priority Attention to Rural Areas 42. The project supported international and national assessments, the results of which demonstrated the low level of educational outcomes from the school system, and this, in turn, contributed to greater urgency in the attention paid to the education system. That urgency is demonstrated, for example, by the publication of Education Development Strategy of the 14 Kyrgyz Republic, 2012–2020, which acknowledges the role played by the assessments in prioritizing educational reform. In addition, the project supported the setting up of an Education Budgeting and Strategy Planning Unit that contributed to the 2012–20 strategy (Kyrgyz Republic 2011). Furthermore, the project piloted a per capita financing approach to support better and more equitable funding of education overall, and this ultimately influenced the development of a normative funding model to support the budgeting process. A review of the system for the development and production of textbooks was produced and has fed into ongoing approaches to system development, noting continuing challenges in 2017 regarding the achievement of alignment between textbook content, production, standard setting, and assessment. Funding to support the actual production of textbooks, the most expensive intervention, had limited effect. Other efforts supported by the project, such as the development and piloting of a formative assessment system in primary schools, support for the introduction of learning improvement plans, and the introduction of an incentive-driven performance management system had little systematic effect beyond the life of the project. OUTPUTS AND INTERMEDIATE OUTCOMES 43. The REP produced a wide range of basic outputs and intermediate outcomes relevant to this objective. In that regard, we note the significant logistical success of the project, that is, the project successfully implemented a diverse range of interventions that involved large numbers of people, and many organizations throughout the system. However, we also note limited strategic, medium- to longer-term relevance for a number of (more expensive) interventions. Key outputs are summarized below. 44. A teacher fellowship program was supported with a view to improving incentives for young teachers to opt for, and remain in, teaching in rural areas. A total of 257 fellowships were awarded, just short of the targeted 300, and retention among those awarded a fellowship was almost 97 percent over the life of the project. 21 The PIU confirmed that the program came to an end in November 2011 (Government Decree 730) and that teacher shortages persist at a rate of about 2,000 per year. Studies suggest that the issue of teacher retention is particularly complicated (Shamatov, 2013). The crux of the issue is that many newly qualified teachers tend not to teach at all or for only a very short period of time (OECD and World Bank 2010). 22 Other issues in this predominantly female and aging workforce include the practice of supplementing the cohort with retirees and having teachers teach subjects in which they have no qualifications. 45. Also in the realm of teacher incentives, 40 trainers and 35 external evaluators were trained to help implement a performance management initiative in the pilot oblasts. Overall, performance management training was delivered to 303 school teams and 2,500 participants (including head teachers, deputy directors, and specialists representing district-level in- service training institutes). 23 The performance management initiative sought to introduce teacher appraisal to support the implementation of an incentive scheme for teachers. Ultimately, 45 percent of the 11,000 teachers in the target oblasts were evaluated, and approximately 25 percent of them (2,972) successfully completed assessment and received incentive payments. As such, the concern expressed in the midterm review regarding the possibility of pressure to positively assess all participants was averted (World Bank 2009). Teacher assessment was also cross-referenced with the willingness of individual teachers to 15 engage in and apply the formative assessment approach, a positive innovation. 24 Performance management methodological manuals were published and distributed widely. 46. The IEG mission met with trainers and evaluators who had been involved in the implementation of the approach. They were enthusiastic about the project and pilot efforts; however, they also reported that the formal or structured project management initiative and incentive-driven approach effectively stopped with project closure. School principals interviewed reported the same, noting, however, that in certain instances, elements of discretionary funding achieved through savings have been used on an ongoing basis to incentivize teachers in certain schools. In addition, elements of the approach have been included in training for teachers, principals, and vice principals at the Issyk-Kul Regional In- Service Training Institute, and at the Talas Regional Methodical Center; however, the pilot approach was not systematically introduced to other oblasts across the country. 25 47. About 97 percent of schools took part in a whole-school incentive program in the pilot oblasts, and 30 high-performing schools were awarded incentives that allowed them access to funding for school improvement, such as fitting out methodological rooms or teacher resource centers, multimedia rooms, and sports halls. In addition, 576 microprojects (small grants) were supported to provide schools in the pilot oblasts (for example, between 2007 and 2010, 1,082 computers were provided in the two oblasts, as were sports and music equipment (World Bank 2011b, 61). Of the 576 supported microprojects, 300 were supplemented by contributions from the local communities and local government. In addition, 40 trainers were trained to provide support to school teams, of which 303 were established, comprising 1,600 persons. Some 303 parents or local community representatives involved in school teams were trained in school improvement planning, a good demonstration effort in terms of community empowerment. School management and teachers interviewed over the course of the mission said the experience and training had enhanced their capacity and that of others (including parents and citizens) to more broadly engage with local and regional authorities. This suggests possible knock-on effects or unintended consequences; however, these contentions cannot be further substantiated. Also, there were no ongoing, formal supports once the project closed. 48. Almost all schools in the pilot oblasts prepared learning improvement plans and, on the basis of these plans, completed School Report Cards for each year between 2007 and 2010. In 2009, following learning improvement plans performance evaluation results, 65 pilot schools received a funding incentive (total vale $135,000). Principals interviewed during mission said that the process of applying for inclusion in the training had enhanced their capacity to apply for funding and grants (from local government and other donors). Principals and education officials from other, non-pilot oblasts also received training. A course on how to develop a learning improvement plans was produced jointly by the MoES and the KAE, and 4,250 copies of a manual were distributed among schools and training institutions throughout the country. However, there was no systematic follow-up in the provision of training to support a standardized approach. 49. The project sought, on a one-off basis, to support the provision of learning materials (for example, supplementary reading materials, maps, globes, and laboratory equipment). All schools in the pilot oblasts received vouchers (the value of which was based on number of 16 students) that allowed them to select and order learning materials from trade fairs organized under the project. All nonpilot schools in rural areas received a “basket” of learning materials. The project also supported two rounds of PISA (2006 and 2009) and two rounds of the NSBA (2007 and 2009). 50. The textbook rental scheme proposed under the project was not realized, as the government decided to replace it with the provision of free books for all those in compulsory education. 26 Project funds were redirected to support this new approach, and the ICR reports a total 1,064,995 textbooks distributed nationwide (no target was set; World Bank 2011b). The largest single area of focus was in nine titles, in four languages, of textbooks and teachers’ methodological manuals with a total print run of 950,000; but, this support turned out to be, at best, a temporary fix, as evidenced by analysis outlined in the PAD for the sector support project, which notes that more than “half of students do not have access to a serviceable textbook, the result of significant chronic underfunding’ and that “most textbooks reflect defects of the curriculum” (World Bank 2004, 3). 27 OUTCOME 51. The Strategic and Analytic Unit supported by project technical assistance provided input into the development of the Education Development Strategy for the Kyrgyz Republic, 2012–2020, and into associated planning documentation. The strategy acknowledges World Bank (and other donor) support in testing and piloting initiatives designed to improve education quality, including in education management and education financing. The strategy commits to a per capita financing model, and the action plan incorporated lessons learned from the pilot. 28 At the time of the mission, PIU representatives reported that the process of moving toward the adoption and implementation of a version of the approach—now referred to as the normative financing model—is underway for the education system as a whole, with technical support provided by the sector support project. The process is now complete for the Naryn, Talas, Jalal-Abad, and Osh oblasts. In addition, the Education and Science Statistical Bulletin, the production of which was originally supported under the REP, continues to be published with support from UNICEF. 52. The objective to improve learning conditions was achieved Substantially. Objective 2: Improving Learning in Primary and Secondary Schools, with Priority Attention to Rural Areas. 53. There is some evidence of a minor, positive shift in learning outcomes for certain age cohorts; however, (i) the significance of these shifts is not established and (ii) the data do not allow for disaggregation of urban and rural achievement. Furthermore, postproject data from the 2014 NSBA, available within the same series for primary school students (grade 4), suggests that apparent gains between 2007 and 2009 did not hold in 2014. OUTPUTS AND INTERMEDIATE OUTCOMES 54. The REP funded two rounds of PISA (2006 and 2009; see more detail in the Outcome section). In addition to provision of funding to support the assessments, the REP supported 17 three presentations and forums in 2008on PISA 2006 findings for stakeholders in the capital, Bishkek, and in the pilot oblasts, along with172 training sessions for 5,000 teachers on new approaches to summative assessment, using sample PISA tests, in preparation for PISA 2009. 55. The REP funded two rounds of the NSBA (2007 and 2009; see more detail in the Outcome section), which played an important role in providing “home-based” confirmation of the trends in learning outcomes identified through the PISA and also helped to further highlight the need for enhanced assessment capacity throughout the system and for curricular and other changes in approach and practice. 56. The REP supported the development of a formative assessment model in the targeted oblasts. The model was designed to reorient pedagogical approaches toward a more child- centered mode that would shift the prevailing focus from rote learning to one that establishes and nurtures the child’s abilities. Teacher participation in the formative assessment training and implementation was voluntary, but teacher performance incentives were linked to engagement in formative assessment. Furthermore, school managers in the pilot oblasts were urged to embed formative assessment and performance management as part of their school- based learning improvement plans (supported under component 2). Interviewees at all levels of the system were enthusiastic about the formative assessment initiative; however, it is important to note that, although it was successfully implemented in the pilot oblasts and generated materials were widely circulated, the initiative effectively stopped once the project came to a close. In some schools (including those visited during the IEG mission), a form of the formative assessment approach continues to be practiced on a less formal basis, but this is not systematic, and no ongoing training or supports are available to teachers (including teachers new to the profession). It is also the case that the formative assessment approach supported under the REP was one of several approaches trialed in the Kyrgyz Republic; in fact, different donors and programs (for example, World Bank, USAID, and ADB) promoted different versions of formative assessment with different geographic targeting at different times. 29 57. The REP also sought to improve professional capacity within the MoES with reference to student assessment and supported a review to improve existing examinations. An Assessment Unit was established in 2006, and relevant staff received various training inputs. That unit was ultimately integrated into the KAE in 2010. Specialists from the unit took part as test developers and administrators in piloting a new school leaving examination model developed under the REP. The pursuit of improvement of existing examinations involved the development of new “examination tasks” piloted in the targeted oblasts and one additional oblast. The results of the pilot exercises were discussed and debated at a major conference in 2008. A final student attestation model was piloted in May 2010, and results were disseminated across the country. A report on the new model was presented to the MoES. OUTCOME 58. The sole outcome indicator for the REP, which is directly tied to this objective, is improved average scores in learning assessments in mathematics, science, and Russian / Kyrgyz / Uzbek for students in grades 4 and 8. There is no reliable evidence available with which to support such improvement. 30 In definitional terms, the indicator references 18 improvement in average performance at national level, not simply improved performance in the two oblasts that were the particular focus of certain components and actions (noting a certain amount of data is available at both levels as outlined below). 59. What is known regarding performance based on national assessment is limited to facts established through the NSBA of students in grades 4 and 8, which was implemented by Center for Educational Assessment and Teaching Methods (CEATM) in 2007 and 2009. A further round was supported under the sector support project in 2014 but, on that occasion, CEATM implemented the assessment for grade 4 students only. The National Testing Center was commissioned by the MoES to undertake a separate assessment of grade 8 students and, as such, the 2014 results for grade 8 students are not comparable. Analysis of the assessment of primary level grade 4 students by CEATM in 2014 over the three assessments is represented in figure 2. Figure 2. Distribution of All Grade 4 students Participating in the NSBA 2007, 2009, and 2014 a. Reading comprehension: b. Math: percent distribution c. Homeland studies: percent Percent distribution of all of all grade 4 students by distribution of all grade 4 grade 4 students by levels levels of academic students participating in the of academic achievement achievement NSBA by levels of academic achievement Advanced Advanced (Level 4) (Level 4) Advanced Above Basic Above Basic (Level 4) (Level 3) (Level 3) Above Basic (Level 3) Basic (Level Basic (Level Basic (Level 2) 2) 2) Below Basic Below Basic Below Basic (Level 1) (Level 1) (Level 1) 0 50 100 0 50 100 0 50 100 2007 2009 2014 2007 2009 2014 2007 2009 2014 Source: National Sample-Based Assessment 2014. 60. In relation to the achievements of grade 4 primary school students in reading and comprehension, figure 2, panel a, shows the following: • In 2007, 35.6 percent of participating students performed at basic level (level 2) and above, and 64.4 percent were not able to achieve the minimal acceptable level. About 8 percent of students performed above basic level (level 3), and only 3 percent achieved the advanced level (level 4). • Results for 2009 showed a decrease in students’ abilities to read and comprehend texts. The percentage of students who did not achieve the level 2 increased by 4.2 percent and reached the value of 68.6 percent. There was also a decrease of the percentage of students in the levels 3 and 4. • Results for 2014 showed a slight improvement in reading comprehension abilities compared with the 2009 round. The percentage of students in level 1 decreased by 3.3 19 percent and almost matched the results of the first round in 2007. Levels 3 and 4 did not change significantly (8.5 percent). In total, 34.7 percent of participating students achieved in the level 2 and above, still below 2007 results. 61. In relation to the achievements of grade 4 primary school students in math, figure 2, panel b, shows the following: • There was noticeable improvement in 2009 over 2007. Students scoring at level 1 decreased by 5.5 percent, the percentage of students scoring at level 3 doubled to approximately17 percent, and the percentage of student achieving level 4 increased by 2.7 percent. • In 2014, the results of the grade 4 students in math decreased again, reverting to slightly below 2007 levels. The decline was especially noticeable in levels 3 (6.3 percent) and 4 (1.7 percent). The percentage of students in level 1 (increased by 7.3 percent. 62. In relation to the achievements of grade 4 primary school students in homeland studies, figure 2, panel c, shows the following: 31 • The patterns of results under this subject heading is similar to that under math: improvements achieved in the 2009 round over the 2007 round were largely lost in 2014. The percentage of students in level 1 increased by 3.2 percent, accompanied by a decrease in the percentage of students in levels 3 and 4, in which there is already a low proportion of the sample students. Table 5. Comparison of Mean Test Subject Score for Grade 4 Students in Issyk-Kul and Talas Subject and Issyk-Kul Talas Year Mean Test Score National Rank Mean Test Score National Rank Reading 2007 501.7 3 488.4 4 2009 510.9 2 463.8 6 2014 515.2 4 498.3 5 Math 2007 498.1 3 490.4 7 2009 548.3 2 503.0 5 2014 521.5 2 497.3 7 Home Studies 2007 496.8 4 486.3 5 2009 536.6 2 500.7 5 2014 517.3 3 504.8 5 Source: CEATM 2014. 20 63. Focusing on the performance of the two targeted oblasts, table 5 compares the mean test score for each of the subject areas for fourth graders in the NSBA (2007, 2009, and 2014) and shows the ranking for each of the two regions compared with all other regions. The data suggests uneven progress. In the case of Issyk-Kul, reading scores have progressively improved, whereas performance in both math and homeland studies declined between 2009 and 2014, although still up on 2007. In the case of Talas, reading scores dipped between 2007 and 2009 and then improved between 2009 and 2014. Following some improvement between 2007 and 2009, math scores declined between 2009 and 2014. In homeland studies, there was minimal improvement between 2009 and 2014. However, it is important to note that the absence of an established control or comparison group makes it impossible to assess whether the project contributed to these achievements (as was also noted by the World Bank 2011b, 22). Attribution is further complicated by the fact that other initiatives were ongoing in various oblasts throughout the country. The lack of separate data on performance in urban and rural schools within the oblasts themselves makes it impossible to determine relative performance in that regard. Table 6 shows the existing data based on the 2007 and 2009 NSBA for secondary-level eighth-grade students in the two pilot oblasts: Table 6. Results of the NSBA for Grade 8 Students in Issyk-Kul and Talas Subject and 2007 Scores 2009 Scores Year (percent) (percent) Level 1 Level 2 Level 3 Level 4 Level 1 Level 2 Level 3 Level 4 Issyk-Kul Math 79 17 4 0 66 22 10 2 Reading and 68 20 9 4 59 15 17 9 Comprehension Talas Math 91 9 0 0 73 18 7 2 Reading and 85 12 2 1 70 16 11 4 Comprehension Source: National Sample-Based Assessment 2009. 64. On face value, the data suggest improvement between 2007 and 2009, particularly in the reduction of the proportion of the assessment candidates scoring at below-basic level (Level 1). However, as above, the ICR cautions against over-interpretation due to data limitation and the lack of established control groups for comparison purposes, and given the lack of disaggregation of data on rural schools and urban schools in the two pilot oblasts. The Director of CEATM cautioned in like manner during interview, noting the tests were not specifically set up, or designed to support an analysis of the efficacy of the REP and/or its broader efforts in the pilot oblasts. 65. The REP also funded two rounds (2006 and 2009) of PISA, which involves assessment of a random sample of 15-year-olds. The results show some improvement in each of the three areas assessed, although the significance of this improvement has not been analyzed or tested, and also noting the test is not designed to provide comparative 21 performance across regions within the country itself. The Kyrgyz Republic ranked last among participating nations, on both occasions, in math, science, and reading: Table 7. Kyrgyz Republic Scores Compared with OECD Average PISA Kyrgyz OECD Kyrgyz OECD Kyrgyz OECD Year of PISA Republic Average Republic Average Republic Average Assessment Science Reading Math 2006 322 500 285 492 311 498 2009 330 501 314 493 331 496 Source: Derived from https://www1.oecd.org/pisa/pisaproducts/39725224.PDF and http://www.oecd.org/pisa/pisaproducts/46619703.PDF. 66. Overall, the most recent data (2014 NSBA, Grade 4 Assessment) suggest little or no improvement in the performance of students at the primary level nationally. Nor is significant improvement discernible in the relative performance of grade 4 students in the two oblasts targeted by the REP, noting provisos regarding the reliability of the data with a view to supporting such a contention. Data sets generated through the REP suggest limited improvement in Grade 8 (NSBA 2007 and 2009) and among 15-year-olds (PISA 2006 and 2009). However, the significance of these face value improvements among different cohorts of secondary school students has not been tested. Once again, the provisos expressed above are relevant with reference to establishing relative performance of targeted oblasts or rural dwellers therein. 67. The objective to improve learning was achieved Modestly. 5. Efficiency 68. No cost-benefit analysis was undertaken for the PAD (although the project itself set out to address various identified areas of inefficiency in the Kyrgyz education system, for example, duplication of production of statistics on education and lack of incentives to encourage efficiencies in the school system). As part of project preparation, a benefit incidence analysis was conducted that confirms the pro-poor and equity effects of investment in basic education, but no follow-up analysis was conducted to demonstrate outcomes that may be attributed to the project in that regard. That said, the project did deliver a lot of measurable output within the confines of a relatively modest budget including, as noted in section 4, broad coverage (school learning plans, training in formative assessment, and performance management) of schools and teachers in the targeted oblasts. The project supported the production of textbooks, piloted a per capita financing model, and supported the provision of learning materials to all schools. The government also referred to efficiencies achieved in the unit pricing of learning materials, due to enhanced procurement. The project also funded two (rather than the planned one) PISA assessments within budget. It can also be argued, as acknowledged in the Education Development Strategy of the Kyrgyz Republic, 2012–2020, that the results of the PISA assessments were instrumental in promoting the accelerated pursuit of quality and effectiveness within the education system. 22 69. The efficiency of the project is rated Modest. 6. Ratings Project Outcome 70. The project’s objective is highly relevant to both country context and strategy and to the World Bank Group strategy for the Kyrgyz Republic. Project design is rated modest due, in particular, to its unrealistic scope, overloaded agenda, and poor M&E, despite good quality analysis and engagement. Given the absence of stated intent or associated measurement, it is only possible to assess objectives in relation to the country-level. The objective to improve learning, the core outcome aspired to, was only modestly realized. Available data suggest little or no improvement at the primary level, despite the efforts of the government, the World Bank, and other donors. The PDO was challenging given system limitations, and the extent of the challenges inherent, in context, in modernizing mind-set and building system wide assessment capacity. The objective to improve the conditions to support learning was substantially achieved. The REP contributed to building the momentum for change, not least through the low-cost medium of funding two PISA assessments, but also through piloting initiatives that are being used to modernize the sector. That said, other, relatively expensive interventions had little ongoing, systematic effect. Fundamental challenges persist, for example, with reference to governance and financing, teacher training, education content, standards, and assessment. Efficiency is rated modest as it is not possible to assess value-for- money, although the project piloted initiatives that may help realize the achievement of future efficiencies. 71. Overall, the project’s outcome rating is Moderately Unsatisfactory. Risk to Development Outcome 72. At the policy level, the risk is low as there is broad system and political commitment to improving education quality as articulated through the education strategy for 2012–20 and the accompanying action plan, both of which reference initiatives piloted through the REP. At the political level, risk is moderate. The years since the close of the REP have been relatively settled; however, given that a presidential election is scheduled for November 2017, the stakes are high, and effecting the transfer of power has introduced an element of volatility in the past. Institutional risk is moderate. Governance issues within the education sector have yet to be addressed. In particular, whereas the MoES has nominal policy responsibility, it continues to lack the power of budgetary oversight that would allow it more readily push the implementation of policy. Financial (budgetary) risk is moderate. Although the Kyrgyz economy has remained resilient, the main sources of risk relate to possible adverse developments in neighboring economies, principally related to oil prices and exchange rate dynamics that will require, in response, a mix of expenditure consolidation and revenue mobilization (World Bank Group 2016). The economy also continues to be heavily reliant on the Kumtor mine, which introduces volatility. 32 Given a very high level of investment in education as a proportion of overall public expenditure, the capacity to invest 23 further in education reform may be limited. (remunerations represented more than 70 percent of the education cost in 2011). 33,34 73. Risk to development outcome is rated Moderate. World Bank Performance QUALITY AT ENTRY 74. In 2002, the Government of the Kyrgyz Republic requested World Bank assistance to address declining education quality, particularly in rural areas. Aide-mémoire detail close and extensive preparatory engagement of the World Bank team at political and administrative levels, and also shed light on the evolving priorities that the project would ultimately address, including, for example: the focus on schools in rural areas; the need for textbooks to reflect the evolving curriculum; and the need for teacher training to reflect and connect with other donor-assisted projects in this area (for example, the ADB Education Sector Development Project; and the Participation, Education, and Knowledge Strengthening project, financed by USAID; see appendix B for other summary detail of early, preparatory missions, including reference to interaction with other donors. 75. The PAD reflects key lessons learned in similar World Bank–supported education projects in the Region, as well as lessons associated with other development interventions with similar objectives reflecting, for example, the importance of performance incentives for teachers in efforts to improve learning outcomes; realistic evaluation of the institutional capacity for managing reform and project activities, and an appropriate project scope to improve the chances for success (noting that the REP included technical assistance designed to improve institutional capacity); and, the importance of clearly defined implementation plans as an integral part of project preparation (World Bank 2004). 35 Project preparation was also supported through a Policy and Human Resources Development Grant (from Japan), which allowed the World Bank, inter alia, to secure support from experts with an international perspective and to support consultation with key stakeholders and beneficiaries. The project was backed up by economic and financial analysis that further elaborated the association between education completion and future earnings, a social assessment to allow for better tailoring of the project to local needs and context, and an environmental assessment. An assessment of the PIU financial management arrangements was undertaken in October 2004 to underpin and ensure fiduciary integrity. 76. However, despite the depth and level of analysis, engagement, and support, project design was both weak and unrealistic, particularly in the context of a stand-alone project (SIL). The World Bank team undertook thorough diagnosis but did less well in the next, critical step of designing a coherent and strategically focused set of interventions that, given the context, would be best placed, on a sequenced basis, to effect desired change. Furthermore, the lack of an effective M&E system (see the Monitoring and Evaluation section) meant there was a missed opportunity to build relevant capacity within the Ministry, and limited the opportunity to support attribution. 24 77. The results framework set out in the PAD was weak, and unspecific with little quantification of indicators. The original framework had one KPI, one intermediate results indicator per component, and a host of results indicators, many of which were not relevant to the indicators and none of which were specified with either baselines or targets. The framework was simplified and rationalized as part of the 2007 midterm to reflect one KPI and six intermediate results or outcome indicators. Finally, and critically, project design lacked an evaluation strategy that would support the achievement of the PDO as a result of project efforts. There is some discussion in the PAD of the merits of setting up an experimental design to allow for treatment and control group comparisons, but this was rejected on the basis of technical difficulty, political sensitivity, and implementation complications (although we note that participating oblasts formed ready-made, large scale treatment groups that could have been compared with nonparticipating oblasts—or a staggered approach to implementation may have allowed for measured comparison over time, and avoided ethical or other sensitivities; World Bank 2004, 18). Taking the above into account, quality at entry is rated Moderately Unsatisfactory. QUALITY OF SUPERVISION 78. The original team ensured tight preparation in terms of fiduciary, social, and environmental issues. In addition, considerable attention was paid to ensuring clarity in the Project Operating Manual to further ensure tight project implementation. Aide-mémoire, reflecting regular supervisory missions, are detailed and were used to alert the Government and the PIU to issues arising and to suggest how these could be addressed in line with World Bank procedures. 36 However, all 14 Implementation Status Reports rate implementation progress and progress toward achievement of the PDO as satisfactory. Whereas it is the case that the project set many activities in train, and produced a significant quantum of output toward bolstering learning conditions, it is less clear that significant, ongoing progress was being made toward the realization of improved learning. The midterm review carried out in October 2007 was used to bolster the approach to project monitoring, further establish the integrity of fiduciary controls, drive engagement with the client, and seek greater complementarity and integration among components (World Bank 2009). However, the fundamental lack of an evaluative framework to support the articulation of the achievement in relation to improved learning was not addressed, undermining the possibility of establishing attribution. The quality of World Bank supervision is rated Moderately Satisfactory. 79. Overall World Bank performance is rated Moderately Unsatisfactory. Borrower Performance GOVERNMENT PERFORMANCE 80. Despite frequent changes in the administration over the life of the project, the government was broadly committed to project objectives, even if such changes led to (i) inefficiencies as new personnel came to grips with their portfolio, including the REP; and (ii) elements of inconsistency, from one regime to another, in the extent to which the project was understood to be strategically important. However, the ICR reports that government officials 25 worked closely with the World Bank’s project team on an ongoing basis, and cooperated fully with the task team (World Bank 2011b). The government also contributed the planned $0.5 million as project funding. Government performance is rated Moderately Satisfactory. IMPLEMENTING AGENCY 81. By the end of the REP, the PIU was well respected for its professionalism. That view was confirmed through interviews held as part of the evaluation mission. A key achievement of the PIU (and other midlevel personnel in the system) involved keeping the many moving parts of the project in motion in a politically volatile climate. The ICR notes that, in the early stages of project implementation, the PIU’s performance was unsatisfactory, both in terms of commitment and leadership (World Bank 2011b). Aide-mémoire reflect advice provided by the World Bank team to the client regarding the importance of hiring well qualified personnel—there were three directors of the PIU over the course of the project. The ICR reports that, by 2008, performance improved considerably with particular reference to financial management (World Bank 2011b). The PIU was also instrumental in developing and supporting linkages between various of the project interventions, most notably, linking engagement with the formative assessment model with teacher assessment under the performance management system. Reporting arrangements. The PIU submitted all required quarterly and annual reports in a timely manner. These reports were informative and provided valuable feedback on how the project was progressing, covering all project activities. Implementing agency performance is rated Moderately Satisfactory. 82. Overall borrower performance is rated Moderately Satisfactory. Monitoring and Evaluation 83. Design. The results framework set out in the PAD was comprised of one PDO or outcome level indicator, and three intermediate results indicators, as well as 18, lower level results indicators (2004, 39–42). The PDO was not revised over the life of the project; however, in June 2007 the results indicators were revised on the basis that the original composition was too crowded and the indicators vague. The revised PDO indicator also locked the measure of project success into improvement in student scores in learning assessment (see table 8). 26 Table 8. Indicators for the REP Key Performance / Outcome Indicator Intermediate Results Indicators and Targets (Revised)* Improved average scores in Number of textbooks delivered to schools (No target) learning assessments in Percentage of schools with learning improvement plans in target oblasts (Target of 100 mathematics, science, and percent from a zero baseline). Russian / Kyrgyz / Uzbek Number of teachers benefitting from the rural fellowship (Target of 300 from a zero for students in grades 4 and baseline). 8 Number of annual yearbooks of education statistics published (No target). Number of commissioned studies on education sector priority topics (Target of 5). System for learning assessment at the primary level (No target). Note: The original outcome indicator (revised June 2007) was improved average scores in Monitoring Learning Achievements (MLA) surveys for students in grades 4 and 8. The World Bank decided that the methodology used for the MLA was not adequate for its purposes and decided, instead, to support the NSBA carried out by the CEATM, an independent testing entity set up with USAID funding that also executed the National Admission Test for tertiary level schooling. 84. The revised KPI (improved average scores in learning assessments in mathematics, science, and Russian / Kyrgyz / Uzbek for students in grades 4 and 8) was elaborated with baselines, but no targets. The first intermediate outcome indicator (baseline but no target) was a simple measure of “textbooks delivered to schools” with no reference to content or quality. The fourth and fifth were measures, respectively, of (i) the number of “yearbooks of education statistics published” and (ii) the number of “commissioned studies on education sector priority topics,” both of which had a baseline, the latter also defined by a target. Again, no measure of quality, utility, or application. Intermediate indicators 2, 3, and 6 focused more closely on the pilot oblasts. Intermediate indicator 2 measured the “percentage of schools with learning improvement plans in target oblasts,” and the third measured the number of “teachers benefitting from the rural fellowships program.” Both were defined with baselines and targets. The sixth measures “system for learning assessment at the primary level.” 85. Implementation. Until the midterm review (October 2007) the M&E system was ineffective. Basically, various data were collected unsystematically. As a result, an international consultant was contracted to review and remedy the situation. That review resulted in a revised, simplified results framework and a unified project M&E strategy. A trained local consultant was also hired to provide technical assistance to component coordinators and to respond to data requests. The new system resulted in a more effective, if limited system. 86. Utilization. The M&E system was patched in following the midterm review and was project specific that is, it was not embedded in MoES systems and practice, noting the MoES itself lacked M&E capacity. So, whereas data collected during implementation was used by the PIU to monitor progress and to make recommendations on budgetary and other issues, an opportunity was missed to enhance capacity at a more systematic level. 87. The quality of monitoring and evaluation is rated Modest. 7. Lessons 88. The evaluation of the REP yields a number of key lessons as follows: 27 89. A selective, and realistic approach to design is likely to create a more closely articulated link (attribution) between World Bank interventions and outcomes. In this case, expert and thorough diagnosis of need and associated challenges was not accompanied by a realistic assessment of what was possible, taking into account constraints such as the political economy, commitment, capacity, resources (human and financial), and available time. Project design should be calibrated in line with these and other relevant considerations, noting that the REP was formed and implemented in a volatile political environment and within a system with clearly identified and acknowledged capacity limitations. 90. Relatively inexpensive but strategically positioned interventions can have a significant effect. World Bank support for the implementation of the 2006 and 2009 PISA assessments, a minor cost in the context of the overall project, had such an effect. The PISA results shocked the system—shock being the term used by almost all those interviewed during the mission to describe the PISA effect. The results of the PISA assessments (as well as the home-based NSBA) jolted the system into what is, basically, a process of culture (rather than simply system) change, forcing a national dialogue on the poor quality of outcomes from education quality and required remedies. 91. Robust monitoring and evaluation is a fundamental component of project management and is integral to the articulation of, and sometimes to the realization of, desired outcome. In the case of the REP, monitoring and evaluation was poorly designed and implemented, thereby limiting the capacity to articulate the ultimate value of the project—and also representing a lost opportunity with reference to building much-needed capacity within the MoES. 92. It is critical to promote and nurture collaboration with other donors in pursuit of common, strategic goals. In this case, representing the World Bank’s first entry into the education sector in the Kyrgyz Republic, the team was well appraised of the work of other donors already engaged in the Kyrgyz education sector. As a newcomer to the sector under invitation from the then-Minister for Education, and given its broader convening role, the World Bank was well positioned to carefully build on or into existing efforts. However, certain of the World Bank–supported efforts added to the “patchwork” quality of donor efforts in the Kyrgyz education sector, with different players piloting different approaches to effectively the same challenge (for example, formative assessment or curricular reform) in different geographic locations, with little ultimate effect at the system level. Bibliography CEATM (Center for Educational Assessment and Teaching Methods). 2014. National Sample-Based Assessment (NSBA) 2014: Report on the Results of the Survey. Bishkek, Kyrgyz Republic: CEATM. Clarke, Marguerite. 2012. “What Matters Most for Student Assessment Systems: A Framework Paper.” Publication 68235, World Bank, Washington, DC. Economist, The. 2013. “Mining in Kyrgyz Republic: Gold in the Hills.” Economist.com, May 16. www.economist.com/news/asia/21573615-attitudes-toward-foreign-investors-hold-troubled-country- back-gold-hills. Kyrgyz Republic. 2011. Education Development Strategy for the Kyrgyz Republic, 2012–2020. Bishkek, Kyrgyz Republic: Government of the Kyrgyz Republic. 28 Hu, Dingyong. “Education Reform in the Kyrgyz Republic.” ECA Knowledge Brief, World Bank, Washington, DC. Mertaugh, Michael. 2004. “Education in Central Asia, With Particular Reference to the Kyrgyz Republic.” In The Challenge of Education in Central Asia, edited by Stephen P. Heyneman and Alan J. DeYoung. Greenwich, Conn: Information Age Publishing. National Sample-Based Assessment 2014: Report on the Results of the Survey. Bishkek, CEATM, 2014. OECD (Organisation for Economic Co-operation and Development). 2010. PISA Results 2009: Executive Summary. Paris: OECD. ———. 2010. Lessons from PISA: Kyrgyz Republic. Paris: OECD and World Bank. Shamatov, Duishon A. 2013. “Everyday Realities of a Young Teacher in Post-Soviet Kyrgyz Republic.” In Post-Soviet Kyrgyz Republic: Political and Social Challenges, edited by Pinar Akcali and Cennet E. Demir, New York, Routledge. ———. 2003. “Kyrgyz Republic—Country Assistance Strategy.” Report no. 25708-KG, Washington, D. C., World Bank, 22 Apr. 2003. ———. 2003. “National Poverty Reduction Strategy and Joint Assessment.” Report 25377-KG, World Bank, Washington, DC. ———. 2004. “Kyrgyz Republic— Rural Education Project.” Project Appraisal Document Report 29954-KG, World Bank, Washington, DC. ———. 2005. “Kyrgyz Republic— Rural Education Project.” Development Grant Agreement H137-KG, World Bank, Washington, DC. ———. 2007. Mid-term Review of the Rural Education Project. World Bank: Washington, DC. ———. 2011a. “Kyrgyz Republic—Profile and Dynamics of Poverty and Inequality, 2009.” Report 65068, World Bank, Washington, DC. ———. 2011b. “Kyrgyz Republic— Rural Education Project.” Implementation Completion and Results Report ICR00001927, World Bank, Washington, DC. ———. 2013a. “Kyrgyz Republic—Country Assistance Strategy and Interim Strategy Note Completion Report Review for the period FY2007-FY2012.” Independent Evaluation Group, Report 79522, World Bank, Washington, DC. ———. 2013b. “Kyrgyz Republic—Country Partnership Strategy FY14–17.” Report 78500-KG, World Bank, Washington, DC. ———. 2013c. “Kyrgyz Republic— Sector Support for Education Reform Project.” Project Appraisal Document Report 69576-KG, World Bank, Washington, DC. ———. 2014. “Kyrgyz Republic—Public Expenditure Review Policy Notes: Education.” Public Expenditure Review Report 88978, World Bank, Washington, DC. ———. 2015. “Poverty Global Practice, Europe and Central Asia. Kyrgyz Republic.” Poverty Profile 2013. Research Report 99772-KG, World Bank, Washington, DC. World Bank Group. 2015. Kyrgyz Republic—Partnership Program Snapshot. Washington, DC: World Bank Group. ———. 2016. Kyrgyz Republic Economic Update, Winter 2016: A Resilient Economy on a Slow Growth Trajectory. Washington, DC: World Bank Group. 1 See http://www.worldbank.org/en/country/kyrgyzrepublic/overview. 29 2 See http://www.economist.com/news/asia/21573615-attitudes-towards-foreign-investors-hold-troubled-country- back-gold-hills. 3 In 2011/12, about 400,000 students were enrolled in primary education and an additional 700,000 in secondary education. Ninety-eight percent of students attended public schools, and over two-thirds of students were in rural areas. Practically all of the 18,000 primary school teachers were female, as were 80 percent of the 41,000 teachers at secondary level. 4 Drawing on a National Survey of Primary Education Quality carried out in November, 2000, the PAD noted that 28 percent of rural schools lacked electricity, 45 percent lacked adequate heating, and 78 percent lacked adequate furniture. 5 Appendix B provides a summary list and chronology of donor support to education 1993–2016. The REP represents the World Bank’s first engagement in a sector in which other donors (such as the Asian Development Bank [ADB], UNICEF [United Nations Children’s Fund], and U.S. Agency for International Development [USAID]) had already been active; for example, ADB’s Education Sector Development Program (school rehabilitation, textbook and learning materials development, and modernizing learning assessment; ADB was also in the process of preparing a second project to run parallel to the World Bank project) and USAID’s Participation, Education, and Knowledge Strengthening Project, the main thrust of which was in-service teacher training, curriculum reform, parental and community involvement in schools, school rehabilitation, and education finance and management reform. 6 In 1998, the Asian Development Bank supported a textbook rental scheme under which schools were provided with an initial supply of textbooks to be rented to students at a fraction of real cost, with rental income used to replace textbooks. But in reality, schools were required to deposit textbook rental receipts in treasury accounts, and funds were often used to purchase new textbooks rather than to replace existing stock, resulting in a lack of textbooks once again. 7 The 2013 project appraisal document notes a boom in education spending, largely explained by a near doubling of compensation for teaching and nonteaching staff following strikes in 2011 (World Bank 2013c). 8 Challenges include, for example, an outdated, overly academic, and theoretical curriculum at the primary level; lack of textbooks as a result of chronic underfunding; teacher shortages in certain subjects (particularly in rural and remote areas); weak organizational structure; shortage of relevant skills and poor incentivizing of staff at the MoES; weak governance; and weak demand for data from the policy-making process (World Bank 2004). 9 The project appraisal document notes that the project will contribute to other, higher-level objectives that would help the country achieve better educational results with the limited available public resources (improve efficiency); help establish public expectations for accountability on the part of all public officials and public service providers; and, help demonstrate the benefits of competition (i.e., through reforming the production of the monopolized system for the production of textbooks; World Bank 2004, 7). 10 Over 64 percent of the population lives in rural areas, and the number of the poor in rural areas (1,297,000) is three times higher than in urban areas (422,000). As such, 75.4 percent of the poor reside in rural areas (World Bank 2011a). 11 Prior to World Bank involvement, the Asian Development Bank supported the largest education projects and also implemented a project (the $15.50 million Second Education Project, 2006–12) parallel to the REP. That project was designed to create a general education system responsive to the needs of a modern economy and focused on poorer areas through, for example, modernizing the curriculum and learning assessment; strengthening the teaching profession; and improving participation and quality in rural schools. 12 Noting that (i) although the pilot oblasts have significant concentrations of rural dwellers, they did not have the highest numbers of poor people; and (ii) Talas did not, relatively speaking, have a particularly high concentration of poor people within its own population base (see Figure 1 panels a and b). 13 Annex 2 of the project appraisal document provides a thorough comparison between proposed World Bank and the ongoing ADB and USAID interventions in the Kyrgyz Republic (World Bank 2004). 14 Available assessment data also showed that, after Batken oblast, the next-lowest-performing oblasts were Talas and Issyk-Kul. They became the targeted sites for much of the project activity. As such, available data confirmed educational underperformance in the targeted oblasts. 30 15 In fact, sustainability is a core concern of the 2007 Mid-term Review, which proposes a position paper on project interventions complete with a long range forecast of development. 16 The National Council would also provide oversight of other donor-supported initiatives and was to be chaired by the Vice Prime Minister for Social Affairs. It would comprise representatives of the MoES, the Ministry of Finance, the Ministry of Labor and Social Protection, the Comprehensive Development Framework Secretariat, the President’s Administration, the Prime Minister’s Office, the Ministry of Local Self-Government, and the Parliamentary Committee on Education. 17 The core PIU team was comprised of a Director, a Deputy Director, a Financial Manager, a Procurement Specialist, Administrative/Disbursement Assistants, and four full-time project component coordinators who would ensure coordination among the related implementation units within the MOE. Finally, the unit would also include an Oblast Coordinator and Accountant for each of the two pilot oblasts. 18 Detailed data are provided in appendix C, tables C.1, C.2, and C.3. 19 The project implementation unit also lacked capacity in monitoring and evaluation that meant, effectively, there was no monitoring and evaluation system in play until after the mid-term review. 20 The project appraisal document notes the World Bank will fund very minor rehabilitation (e.g., window replacement) rather than larger-scale civil works construction or rehabilitation of schools (World Bank 2004, 25). 21 Of the 96, 85, and 76 fellowships awarded in the academic years 2005/06, 2006/07, and 2007/08, respectively, 96, 85, and 67, respectively, stayed in the scheme, for an overall retention rate of 96.4 percent (World Bank 2011b, 25) 22 Between 2002 and 2005, only 17 percent of newly qualified teachers took a teaching position; 14 percent taught for one year and 11 percent for three years. 23 Training seminars on teacher performance assessment were also delivered to 920 trainees in Batken. 24 Subsequently, 2,780 teachers, principals, and rayon education managers were trained in the formative assessment approach between 2007 and 2010, and 25,000 copies of manual “Formative assessment” were widely distributed. 25 The scheme influenced a subsequent performance appraisal system (Teacher Performance Appraisal of 2011), which was later canceled. A new model of Teacher Performance Appraisal was developed in 2016 but it was not implemented, principally due to changes at Ministerial level. 26 The project had sought to address the depletion of the stock of textbooks through support for a textbook rental scheme, and to establish an autonomous Textbook Rental Fund. Under the Soviet system, textbooks had been provided free, but the post-transition collapse in public finances resulted in parents being required to purchase textbooks, and this had proved onerous for many. A textbook rental scheme introduced with ADB support in 1998 had failed. 27 In addition to the above: technical assistance was provided to help set up and equip a new Strategic and Analytic Unit (established October 2006); Annual yearbooks of education statistics were published for 2007, 2008, and 2009; Five studies were commissioned on education sector priority topics, meeting the target. The most significant of these were the Development of the Capitation Financing Introduction Plan (2007); and the Strategic Study on Textbooks Availability in the Kyrgyz Republic (2007); A system for learning assessment – a Formative Assessment (formative assessment) approach - was developed and introduced at primary level (see below); A per-capita financing approach was researched, and relevant training was provided to school principals/managers (186) and local education officials (57). The model was piloted in 400 schools across the country. 28 On May 30, 2013, the government adopted a regulation “on transition of education organizations to republican budget-based financing through territorial subordinated divisions of the Ministry of Education and Science of the Kyrgyz Republic from financing through local budget.” The major precondition for this action was a decision made by the Committee for Education, Science, Culture and Sport of the Kyrgyz Republic on September 25, 2012 № 904- i 29 Key outputs associated with formative assessment supported under the REP include 50 trainers selected and trained to introduce formative assessment in the pilot oblasts; consultations supported by trainers on new teaching and assessment methods for 200 principals in pilot oblasts; 245 training courses conducted by trainers for 8,500 teachers in the pilot oblasts; 1,080 representatives of school administration, rayon (district), and city education departments in pilot oblasts being provided with training to introduce them to the formative assessment approach; 31 10,000 copies (8,000 in Kyrgyz and 2,000 in Russian) of the teachers’ manual, Formative Assessment and Methods of Student-Centered Teaching and Learning, were published and distributed nationwide; the development and wide circulation of 15,000 copies of the manual Student Assessment. New Approaches and Methods (published in Kyrgyz and Russian); and four two-day training sessions on formative assessment were conducted for 330 rayon education managers, school principals, and resource centers or associations in Batken oblast (the oblast with the lowest level of performance in national assessments of educational outcomes). 30 As earlier noted, the Monitoring Learning Achievements (MLA) was to be the instrument through which learning outcomes would be tracked. However, the methodology of the MLA proved unacceptable to the World Bank, and instead the Rural Education Project funded an National Sample-based Assessment that was implemented by the independent entity, the Center for Educational Assessment and Teaching Methods (CEATM)—originally set up with USAID support to implement the National Admissions Tests to objectively determine access to tertiary-level education. 31 Homeland studies includes disciplines that prepare a child for life in the modern environment and provide basic skills and knowledge for further learning of the natural sciences in secondary and higher grades (chemistry, biology, physics, astronomy, and geography). 32 The World Bank’s October 2016 Country Snapshot notes that the Kumtor goldmine comprises about 10 percent of GDP. 33 An April 2015 World Bank Group program snapshot notes that education consumes 20 percent of government expenditure but continues to face the key challenge of turning spending into better learning outcomes (World Bank Group 2015). 34 A World Bank Public Expenditure Review for the Kyrgyz education sector notes a rapid expansion of public spending on education in recent years but continuing challenges relating to efficiency of spending, sustainability of sector financing (rapid wage growth 2012–12; World Bank 2014). 35 Annex 9 of the project appraisal document was also supported by sound Economic and Financial Analysis that further elaborated the association between education completion and future earnings (World Bank 2004). 36 For example, (i) the aide-mémoire for the January 18–26 supervision mission notes the project team organized a series of nine video conferences with the project implementation unit in November and December 2005 to address concerns about slow startup. Decisions were made to revise procurement arrangements for key goods and services contracts in order to accelerate implementation and disbursements; and (ii) the aide-mémoire for a December 11–20, 2006 mission reflects a supervision mission for the Rural Education Project and discussion with Government on funding allocated under the Fast Track Initiative, which would be used largely to support pre-school education. That aide-mémoire addresses the failure to establish the Textbook Approval Board and the cancellation of the textbook rental scheme and also addresses the question of merging the Asian Development Bank and World Bank implementation units, a question that had been raised by the Ministry of Education and Science at project preparation. It is notable that at the time of the PPAR mission in January 2017, this merger had not been implemented. 33 Appendix A. Basic Data Sheet RURAL EDUCATION PROJECT (GRANT H137-KG) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs 15.00 15.30 102% Loan amount 15.00 15.30 102% Cancellations 0.00 0.19 N/A Total project costs 15.00 15.30 102% Source: Project portal Cumulative Estimated and Actual Disbursements FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12 Appraisal estimate 0.36 2.97 8.78 12.90 14.42 15.00 15.00 15.00 (US$ million) Actual (US$ million) 0.50 0.92 3.23 8.465 10.39 12.865 14.35 15.30 Actual as % of 136% 31% 37% 66% 72% 86% 96% 102% appraisal Date of final disbursement: July 31, 2011 Source: SAP—Project disbursement data. Note: FY = fiscal year Project Dates Original Actual Concept Review 10/20/2003 10/20/2003 Negotiations 11/10/2004 11/10/2004 Board approval 12/14/2004 12/14/2004 Signing 02/03/2005 02/03/2005 Effectiveness 05/02/2005 05/02/2005 Closing date 03/31/2010 03/31/2011 34 Staff Time and Cost Staff Time and Cost (Bank budget only) Staff Weeks (number) US$ 000s (including travel Stage of Project Cycle and consultant costs) Lending FY03 32.08 157.65 FY04 36.02 163.11 FY05 20.00 36.53 Total: 88.10 543.27 Supervision and Implementation Completion and Results Report FY05 51.20 200.00 FY06 47.24 127.45 FY07 48.75 253.63 FY08 31.03 135.39 FY09 37.73 123.79 FY10 22.99 65.99 FY11 19.05 94.57 Total: 257.99 339.74 Source: Implementation Completion& Results Report 35 Task Team Members Name Title Unit Lending and Supervision Aisle Bedelbayeva Education Specialist ECSH2 Anarkan Akerova Counsel LEGEM Asel Abdyramanova Project Manager USAID\QRP Asel Sargaldakova Sr. Health Specialist ECSH1 Cristina Marosan Ling Evaluation Officer WBICR Dingyong Hou Sr. Education Specialist ECSH2 Djamansariev Ayat Coordinator on Normative Finance Elina Manjieva Junior Professional Associate ECSHD Ernesto P. Cuadra Lead Education Specialist MNSHE Hannah M. Koilpillai Senior Finance Officer CTRFC Gabriel Francis Program Assistant ECSHD Gulzhan Mamytova Teacher Training Expert Gyulaiym Kolakova Team Assistant ECCKG Ilyaz Sarybaev Specialist on Textbooks and Irina Goncharova Procurement Analyst ECSO2 Isak Froumin Lead Education Specialist ECSH2 John Otieno Ogallo Sr Financial Management Specialist ECSO3 Julie Wagshal Budget Assistant CCRDR Mira Mykyeva Standards Specialist Nadejda A. Mochinova Resource Management Officer AFTRM Natalia Tourchina Team Assistant ECCKG Naushad A. Khan Lead Procurement Specialist SARPS Nurbek Kurmanaliev Procurement Specialist ECSO2 Raisa Inkeri Venalainen Consultant ECSHD Paul Cahu Economist ECSHD Roland N. Clarke Lead Public Sector Management LCSPS Ross S. Pavis Senior Operations Officer AFTDE Zharkyn Ryskulova Assessment Specialist 36 Appendix B. Other Data Results of the Second Monitoring Learning Achievements in the Kyrgyz Republic The first Monitoring Learning Achievements (MLA) test of the aptitude of 4th grade students (basic knowledge and practical skills in mathematics, grammar, and life skills) was carried out in 2001 and the second, in 2005. The 2005 test was supported by a working group comprising leading specialists of the MoE, UNICEF, Kyrgyz Academy of Education, American University in Central Asia, the REP, and the El-Pikir Centre for Public Opinion Studies, which was the executor of the study (Monitoring Learning Achievements (4th grade) Nationwide Study of the Quality of Education in Primary Schools Center of Public Opinion Study, “El Pikir”). The study (p. 11) found: • The general quality of primary education in Kyrgyz Republic declined from 2001– 2005, due mainly to falling standards in mathematics and grammar that lay the foundation for all subsequent education. The decline in education was more marked in rural schools, and in schools where pupils were taught in Uzbek and Russian, and to a lesser extent in Kyrgyz; • The literacy level fell considerably 2001–05 due to the fact that primary schools were not paying adequate attention to the development of individual pupil’s work and were not practicing creative tasks enough, resulting in an overemphasis on work through textbooks, leading to passive absorption of information and the inability to make practical use of the skills obtained under real conditions. • In mathematics, 58.8% of pupils passed in 2005, compared to 81.4% in 2001. The number of pupils who got an excellent grade fell fivefold and those who got a good grade halved. The number of pupils assessed as satisfactory increased 2.5 times and the number of failed pupils doubled. • On the other hand, the achievements of pupils in life skills in 2005 exceeded the test results of 2001 by 2.9%. Almost one in six students in 2005 were assessed as excellent in the Life Skills test (none in 2001). The MLA also gathered other information regarding the state of the primary education system overall and found, inter alia: • Fourth grade teachers take their work seriously and try not to miss classes. • The teaching staff is growing older and schools do not have enough young teachers. • Schools lack modern teaching means (TVs, tape recorders, VCRs, and computers). Visual aids and graphs are handmade by teachers. Blackboard and chalk remain the principal teaching tools. • Primary school teachers use the available computers inadequately because they lack computer proficiency. • Primary school teachers have low internal motivation for self-improvement and improving proficiency. The appropriate training courses for them are initiated by school administrations. 37 • The prevalent teaching methods used in primary schools are homework and individual work. Practical methods and creative assignments are rarely used. Teachers shift the main burden of studying to homework and they do not adequately use methods aimed at developing schoolchildren. Supervision is focused more on completing homework and class work and testing is rarely used. • Most of the primary school teachers devote extremely little time to preparing for classes and looking at exercise books. This implies a lack of interest in this element of teaching. Many teachers never, or only nominally, conduct any extra-curricular training with pupils. • The level of information support compared to 2001 had considerably reduced. The majority of teachers do not get up-to-date informational support. • 40–66 percent of the teaching and educational manuals for teachers meet modern requirements and less than half of them are in good condition. • One in three primary school pupils do not have good quality textbooks in all disciplines. This all reflects the neglect of schools by state and local self-governance bodies and the low salaries of teachers. Key Points from Aide-Mémoire: Preparatory Missions for the REP The aide-mémoire for the October 27–31, 2003 pre-preparation mission, notes a discussion with the Ministry regarding the possibility of changing the name of the project from “Rural Education Project” to “Education Quality Improvement Project” to more accurately reflect proposed coverage and content. The Ministry argued to retain the original title because the generic actions proposed under the project are included primarily because they are necessary to improve the educational performance of rural schools. This is important in terms of the ultimate composition of the project PDO (which is pitched at national level) and associated indicators. In one high-level meeting reflected in the aide-mémoire for a July 14–23, 2004, preappraisal mission, the Deputy Minister for the Ministry of Finance raised a question about the sustainability of introducing teacher incentives in just two oblasts, and suggested that the Government and the Bank work together to mobilize other donors to finance performance incentives for teachers in additional oblasts. The mission initially presented a proposal for country-wide incentive payments to teachers in rural schools, the amounts of which would vary with the remoteness of schools from oblast and rayon (district) centers; however, the then Minister said he would prefer a more targeted program of rural teacher incentives to help attract new teacher graduates in scarce subject areas, and proposed a system of rural “teacher fellowships” for approximately 100 new teacher graduates each year who agree to serve in rural schools. This is also interesting in terms of the coverage and focus and the ultimate efficacy of the project again t its stated objective. The aide-mémoire for an initial supervision mission to coincide with the project launch (March 17 and 18, 2005) was carried out jointly with the appraisal mission for the ADB Second Education Project. It notes that the REP was designed to complement the activities supported under the first and second ADB projects and the ongoing USAID Aide-mémoire education project, and that the new ADB project will similarly reflect experience and the new 38 World Bank project. The aide-mémoire also notes that a proposal to establish The National Education Council, with reporting responsibilities and staffing as agreed at negotiations for the REP, had been submitted to the Prime Ministry for review and approval. To rationalize the functions of this new National Education Council and the existing Education Advisory Board, the mission proposed that: The National Education Council focus on broad questions of education strategy (for example, expansion of pilot interventions, education priorities in poverty reduction, and medium term budget priorities). The mission argues that the broad based composition of the National Education Council would equip it to address important strategic policy questions in the education sector, which transcend any single Ministry. It would be supported by the Education Budgeting and Strategic Planning Unit which was to be established under the REP. Meanwhile, the existing Education Advisory Board could focus on the more specific questions involved in coordinating donor-financed education activities, with the aim of maximizing complementarity and benefits of these initiatives. Principal Donors and Major Projects Supported in the Kyrgyz Education Sector, 1993– 2016 Table B.1 Timeline of Major Donor-Supported Projects in Kyrgyz Education Sector between 1993–2016 Start Donor & Project Name Objectives/aim Date Since The Soros Open Society Developing educational materials, procuring child- 1993 Institute: Step-by-Step size furniture, and training teachers and parents in interactive methods of instruction, focusing on multi- disciplinary topics, and involving cooperative learning. Since The Soros Open Society To provide participants with strategies for interactive 1997– Institute: Reading and methods of teaching that prepare pupils for citizenship 2000 Writing for Critical Thinking in open societies. 1997– ADB: First Education Assisted in setting up training for 7,027 teachers and 2004 Project / Education Sector administrators in 32 modular programs, 10-day Development Project methodology workshop was conducted for 3700 teachers and administrators. 1999– Save the Children, UK: Teacher training in active-learning methodologies, 2007 Inclusive Education Project and in particular those designed to create an inclusive environment for vulnerable children and children with disabilities. Targeted in Naryn, Osh, and Bishkek, the project provided inservice training workshops to teachers, parents, community members, and children, and provided funding for mini-projects for school- based children’s clubs. 2002– UNICEF: Global Education Developing active and responsive citizenship through 2005 Program the implementation of interactive and participatory methodologies of instruction and by including subject themes not traditionally part of the state-mandated curriculum 39 Start Donor & Project Name Objectives/aim Date 2002-date USAID: Supporting Equity In 2002, USAID helped design and launch the through National Admission National Admission Test in partnership with the Testing Ministry of Education and Science. At the same time, CEATM, an independent testing organization, was founded with USAID support to design and administer the test. Since 2002, USAID provided over $1,100,000 to help administer the National Admission Test, including upgrading the test database; revising and publishing study guides for students; and conducting public awareness campaigns throughout the Kyrgyz Republic. 2003– USAID: Participation, Teacher training in active-learning methodologies and 2007 Education and Knowledge in particular those designed to create an inclusive Strengthening (PEAKS) environment for vulnerable children and children with disabilities. 2004– World Bank: Rural To assist the Recipient in improving learning and 2011 Education Project learning conditions in primary and secondary schools, with priority attention to rural areas” – focus on the Issyk-Kul and Talas oblasts. Since Aga-Khan Development Quality and affordable pre-school education. 2005 Network: MSDSP Developing educational materials, procuring child- size furniture and, above all, training teachers and parents in interactive methods of instruction. Helped build or renovated 60 kindergartens in Osh and Naryn oblasts as well as 20 unique kindergartens in yurts. To address the shortage of children’s literature in Kyrgyz language, MSDSP KG issued 35 children’s books, including bilingual books on the theme of tolerance. To revive the culture of reading in families, with the support of MSDSP KG, 12–200 parents and teachers, as well as 260 librarians learnt the methodology of development of reading skills; a network of 66 resource centers for parents was created. 2005– ADB, Second Education To support of the Government of the Kyrgyz Republic 2011 Project / Education Sector in the implementation of policy reforms in the Development Program education sector: modernization of the curriculum, National educational standards, curricula, and so on), based on student centered approach, the development of a new generation of textbooks and training materials, rehabilitation of 90 schools located in rural areas. 2007– World Bank: Education for Improving primary education - includes providing 2010 All - Fast Track Inititive inservice training to primary level teachers designed to improve teaching methods, work more effectively with the community, and implement inclusive education. Teacher training that is focused on student- centered methodology. 40 Start Donor & Project Name Objectives/aim Date 2007– World Bank: Education for To support and accelerate the achievement of the six 2010 All (2) - Fast Track Inititive Eformative assessment goals through improving pre- school development; improving the national curriculum through creating and distributing textbooks, teaching material and equipment; improving school infrastructure, and supporting the development of human resources within the education system. подготовки. 2007– USAID: Quality Learning Emphasizes active-learning methodologies, the focus 2012 Project is on different subject areas (Kyrgyz language and math at the secondary level, math and arts in primary grades) and in preservice in addition to inservice teacher education. The project seeks to expand access to student-centered teaching methods, to enable students to demonstrate increased level of higher- order thinking skills. The project also seeks to secure state certification of newly developed teacher training courses and modules and to make the student-centered approach a part of the standard preservice and inservice curriculum at teacher training institutes. 2012– World Bank: Russia The improvement of national systems of evaluation in 2013 Education Aid for education. Development (READ) 2013– GIZ: Central Asia Program Development of a Regional Large-Scale Student 2015 for Student Assessment Assessment, working with Kyrgyz Republic and (CAPSA) neighboring countries. 2013– USAID: American Institutes Designed to improve the reading and critical thinking 2017 for Research- Save the skills of children in grades 1–4 through assistance to Children International in government ministries and school-based educators. In Kyrgyz Republic doing so, QRP “Reading together” is building a “Quality Reading Project” stronger foundation for learning in all areas of study 2015– ABD: Sector Development Will support the Government’s plans (Education 2021 Program: Strengthening Development Strategy for the Kyrgyz Republic, Education System 2012–2020) to improve education quality. Till 2019 EU Support to increase the level of transparency in the Education budget process. Support to the Education Sector in the Kyrgyz Republic. Fostering and monitoring of the education reforms in Kyrgyz Review of the Sector Policy Support Program in Education Sector IEG, 2017 - derived from data collated on the Kyrgyz Government’s Aid Management Platform (http://www.amp.gov.kg) 41 Appendix C. List of Persons Met Name Title Unit Abdrachmanova Buruljan Principal School-Gymnasium, Bosteri Bakirov Arthur Director National Testing Center, MoES Burkitova Alia Teacher Training Specialist Strengthening Sector Education Development Project, ADB Ivanov Aleksandr Director Foundation for Education Initiatives Support Janadilov Temirbek Monitoring and evaluation USAID “Quality Reading Coordinator Project” Jumabaeva Gulnur Methodologist Issyk-Kul Regional Institute of Education Junushalieva Kaliman Head of Education Kyrgyz Academy of Management Department Education Kazakbaeva Shairgul Trainer Sector Support for Education Reform, WB-PIU Khamzina Saule Regional Monitoring and USAID “Quality Reading Evaluation Manager Project” Mamytov Abakir President Kyrgyz Academy of Education Mamytova Guljan Training Specialist Sector Support for Education Reform, WB-PIU Matohina Tatyana Program Coordinator Foundation for Education Initiatives Support Minbaeva Elmira Methodologist Issyk-Kul Regional Institute of Education Mykyeva Mira Specialist on Standards Sector Support for Education Reform, WB-PIU Nasibulina Inna Principal Balykchi School №1 Omurzakova Ainura Senior Specialist Issyk-Kul Regional Sate Administration Osmonalieva Guliza Principal Karakol School-Gymnasium # 11 Rasaev Khalil Program Officer ADB 42 Name Title Unit Ryskulova Jarkyn Assessment Specialist Education Projects, WB-PIU Shakirov Rashid Education Technical Expert USAID “Time to Read” Shamshidinova Baktygul Head of the Department National Testing Center, MoES Sharshenbaeva Shaiyr Head of the Department Issyk-Kul Regional Institute of Education Sultanova Gulmira Operations Officer World Bank Country Office Temiraliev Malik Head Ak-Suu District Education Department Toktomametov Almaz Head of Preschool, and Kyrgyz Academy Primary School Department of Education Urumov Jenishbek Trainer Helvetas Projects in KR Valkova Inna Director Center for Educational Assessment and Teaching Methods Vorycheva Ludmila Principal Osmonov School- Gymnasium, Cholpon-Ata Yusupova Aikynai Monitoring and evaluation USAID “Quality Reading Coordinator Project”