Document of The World Bank Report No: ICR00003026 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-45980) ON A CREDIT IN THE AMOUNT OF SDR 18.9 MILLION (US$28 MILLION EQUIVALENT) TO THE REPUBLIC OF UZBEKISTAN FOR A BASIC EDUCATION PROJECT (SECOND PHASE) October 27, 2014 Education - Global Practice Central Asia Country Department Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective October 2014) Currency Unit = Uzbekistan Som (UZS) 1.00 = US$ 0.00042 US$ 1.00 = UZS 1.48810 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS APL Adaptable Program Loan BEP Bank-financed education project CAS Country Assistance Strategy ESP Education Sector Plan GDP Gross Domestic Product GNI Gross National Income GPE Global Partnership for Education IA Implementing Agency ICT Information and Communication Technology IO Intermediate Outcome MoPE Ministry of Public Education MoF Ministry of Finance MTBF Medium Term Budget Framework PAD Project Appraisal Document PCF Per Capita Financing PDO Project Development Objectives PIU Project Implementation Unit SBTT School-based Teacher Training SEDP School Education Development Program Vice President: Laura Tuck Country Director: Saroj Kumar Jha Global Practice Manager (Acting): Cristian Aedo ICR Team Leader: Dingyong Hou ICR Primary Author: Gerard Peart REPUBLIC OF UZBEKISAN BASIC EDUCATION PROJECT SECOND PHASE CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................ 1 2. Key Factors Affecting Implementation and outcomes ............................................ 6 3. Assessment of Outcomes ....................................................................................... 12 4. Assessment of Risk to Development Outcome ...................................................... 16 5. Assessment of Bank and Borrower Performance .................................................. 17 6. Lessons Learned..................................................................................................... 19 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners........ 20 Annex 1. Project Costs and Financing .............................................................................. 21 Annex 2. Outputs by Component...................................................................................... 22 Annex 3. Economic and Financial Analysis ..................................................................... 28 Annex 4. Bank Lending and Implementation Support/Supervision Processes................. 33 Annex 5. Beneficiary Survey Results ............................................................................... 35 Annex 6. Stakeholder Workshop Report and Results....................................................... 36 Annex 7. Summary of Borrower's ICR ............................................................................. 37 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ........................... 39 Annex 9. List of Supporting Documents .......................................................................... 40 MAP: IBRD 33508R A. Basic Information Second Basic Country: Uzbekistan Project Name: Education Project ID: P107845 L/C/TF Number(s): IDA-45980 ICR Date: 10/27/2014 ICR Type: Core ICR REPUBLIC OF Lending Instrument: APL Borrower: UZBEKISTAN Original Total XDR 18.90M Disbursed Amount: XDR 16.59M Commitment: Revised Amount: XDR 17.28M Environmental Category: C Implementing Agencies: Ministry of Public Education (MoPE) Cofinanciers and Other External Partners: Not applicable B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 11/26/2008 Effectiveness: 01/13/2010 03/10/2010 12/18/2012 Appraisal: 04/29/2009 Restructuring(s): 03/12/2014 Approval: 06/23/2009 Mid-term Review: 10/15/2012 10/10/2012 Closing: 03/31/2014 03/31/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Unsatisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Unsatisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Quality at Entry: Government: Satisfactory Unsatisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Moderately Overall Borrower Moderately Satisfactory Performance: Unsatisfactory Performance: i C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Pre-primary education 20 20 Primary education 50 50 Public administration- Education 10 10 Secondary education 20 20 Theme Code (as % of total Bank financing) Education for all 40 40 Participation and civic engagement 20 20 Public expenditure, financial management and 20 20 procurement Rural services and infrastructure 20 20 E. Bank Staff Positions At ICR At Approval Vice President: Laura Tuck Shigeo Katsu Country Director: Saroj Kumar Jha Motoo Konishi Practice Manager Cristian Aedo Mamta Murthi Project Team Leader: Naveed Hassan Naqvi Maureen Anne McLaughlin ICR Team Leader: Dingyong Hou ICR Primary Author: Gerard Peart F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The overall objective of the program and each of the phases is to continue supporting the Government of Uzbekistan’s (GoU) efforts to improve effectiveness of teaching and learning through (1) targeted interventions in selected general secondary education ii schools and preschools in rural poor areas; (2) development of institutional capacity to assess student learning; and (3) adoption of predictable and transparent school budgets in selected oblasts. Revised Project Development Objectives (as approved by original approving authority) The Project Development Objectives were not revised. (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years An increase in learning outcomes in Grade 4 that is greater in project schools Indicator 1 : than in non-project schools as measured by a new standardized test of learning achievement available for a sample of students in project and non-project schools Math: 478 (project Math: schools) 514 (project 487 (Non-project schools) Project schools schools) Value scores improve 496 (Non-project quantitative or Reading: 485 (project and move closer schools) Qualitative) schools) those of non- Reading: 482 511 (Non-project schools) project schools (project schools) 485 (Non-project schools) Date achieved 06/23/2009 12/01/2013 03/31/2014 Comments PARTLY ACHIEVED. The project schools experienced an increase in math (incl. % scores that was greater than in non-project schools, but did not experience an achievement) increase in reading scores. National standardized assessments of student learning carried out in Grades 4 Indicator 2 : and 8 with report analyzing and disseminating the results First national assessments Value Grade 4 (Years 2 carried out for Grades 4 Grade 4 (Year quantitative or and 4) and Grade 8 Grade 4 (Year 4) and 8 under phase one of 4) Qualitative) (Year 3) APL Date achieved 06/23/2009 12/01/2013 03/31/2014 ACHIEVED. Original target was not met, but the revised target was achieved. Comments The Grade 4 assessment was successfully carried out, and attributed to project (incl. % activities. The indicator was revised after the mid-term review to read, ‘National achievement) standardized sample-based assessments of student learning carried out in Grade 4 with report analyzing and disseminating the results.’ Per capita financing implemented in six oblasts with report analyzing and Indicator 3 : disseminating the results Per capita Begin implementation in Report on per PCF implemented Value financing three new oblasts and capita financing all over the country. quantitative or (PCF) refine implementation in prepared and Report prepared Qualitative) implemented three original oblasts disseminated and disseminated. with report iii analyzing and disseminating the results Date achieved 06/23/2009 12/01/2013 03/14/2014 03/31/2014 ACHIEVED. Report on national implementation of PCF prepared and Comments disseminated. In 2010 the Gov. scaled up PCF nationally and took full (incl. % responsibility for its implementation including nearly all activities. The indicator achievement) was revised after the mid-term review to read, ‘Per-capita financing implemented with report analyzing and disseminating the results.’ (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 : Percent of project schools that have received the requested learning materials Value 100% (or 1,501 (quantitative 0% 100% (or 2,198) Schools) or Qualitative) Date achieved 06/23/2009 12/01/2013 03/31/2014 Comments EXCEEDED. Savings of US$ 9.53 million were realized during procurement. (incl. % After level-2 restructuring, a second round of 694 schools were selected and achievement) received ICT equipment. Percentage of teachers at sampled project schools where learning materials Indicator 2 : acquired under the project are used effectively in the classrooms. Value (quantitative 0% 50% 93.8% or Qualitative) Date achieved 06/23/2009 12/01/2013 03/31/2014 Comments EXCEEDED. The independent evaluation found that 93.8% of general education (incl. % teachers in the project schools used the teaching-learning materials effectively. achievement) Indicator 3 : Strategy for School-Based Teacher Training (SBTT) MoPE has indicated Value SBTT Strategy that elements of (quantitative No SBTT Strategy adopted by MoPE SBTT strategy will or Qualitative) be used Date achieved 06/23/2009 12/01/2013 03/31/2014 PARTIALLY ACHIEVED. There is no formal commitment and budgeted plan Comments for training teachers using the SBTT, although the Minister of Public Education (incl. % has verbally indicated that the SBTT strategy will be adopted as a complement to achievement) existing regular teacher training courses. Indicator 4 : Number of teachers at pre-schools and schools trained Value (quantitative 0 50,000 59,147 or Qualitative) Date achieved 06/23/2009 12/10/2013 03/31/2014 iv EXCEEDED. 4,139 general secondary teachers & 534 kindergarten teachers Comments were trained directly; these in turn trained 52,264 general secondary teachers and (incl. % 2,210 kindergarten teachers, respectively. 1,506 general secondary School achievement) Directors and 1,023 Deputy Directors were trained, as were 376 kindergarten directors. Percent of teachers at sampled project schools and preschools who are changing Indicator 5 : their teaching practice towards a more student centered approach Value (quantitative 0% 60% 72% or Qualitative) Date achieved 06/23/2009 12/01/2013 03/31/2014 ACHIEVED. An independent evaluation found that 76.2 percent of general Comments education teachers who were directly trained by the project, and 84.8 percent of (incl. % pre-school teachers trained by the project, had adopted student-centered achievement) approaches. The evaluation also found that the quality of the direct teacher training was high. Percent of Schools Boards at project schools that have implemented a school Indicator 6 : improvement proposal Value 33% (or 500 School (quantitative 0% 30% Boards) or Qualitative) Date achieved 06/23/2009 12/10/2013 03/31/2014 PARTIALLY ACHIEVED. 500 School Boards implemented a school Comments improvement proposal. 197 schools secured only the first tranche, while 303 (incl. % schools also secured the second tranche (70 percent of funds were provided in achievement) the first tranche and the remaining 30% in the second tranche). Percent of schools at three additional oblasts participating in per capita financing Indicator 7 : that are getting more than 10 percent above or below average spending per student Percent of schools participating in per capita financing that Value are getting (quantitative 60% 30% 30% more than 10 or Qualitative) percent above or below average spending per student Date achieved 06/23/2009 12/01/2013 12/20/2013 03/31/2014 Comments ACHIEVED. The indicator is a measure of equity. The greater the percentage (incl. % of schools getting 10 percent more or less (on a per capita basis) than the national achievement) average, then the more inequitable the financing. Indicator 8 : Learning assessment results for 4th and 8th graders from national assessments Value Baseline surveys for Grade 4 Grade 4 Grade 4 assessment (quantitative grades 4 and 8 carried out assessments in assessment in carried out in year or Qualitative) under BEP-1 years 2 and 4; year 4. 4. Results analyzed v Grade 8 and disseminated. assessment in year 3 Date achieved 06/23/2009 12/01/2013 12/20/2013 03/31/2014 Comments (incl. % ACHIEVED. Original target not achieved, but revised target was achieved. achievement) Report prepared and disseminated comparing results in baseline survey of Indicator 9 : schools and households with results from follow-up survey Follow-up baseline Value Follow-up baseline survey carried out. (quantitative Carried out under BEP-1. survey carried out. Report prepared or Qualitative) Report prepared and disseminated. and disseminated. Date achieved 06/23/2009 12/20/2013 03/31/2014 Comments (incl. % ACHIEVED. achievement) Indicator 10 : Impact evaluation prepared and disseminated Value Impact evaluation Impact evaluation (quantitative N/A prepared and prepared and or Qualitative) disseminated. disseminated. Date achieved 06/23/2009 12/01/2013 03/31/2014 Comments ACHIEVED. Impact evaluation is a compilation of the results of three surveys: (incl. % student assessments, classroom teaching observation, and school grants achievement) implementation. G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (US$US$ millions) 1 07/31/2009 Satisfactory Satisfactory 0.00 2 11/17/2009 Satisfactory Satisfactory 0.00 3 04/27/2010 Satisfactory Satisfactory 0.20 4 12/17/2010 Satisfactory Satisfactory 0.51 5 05/29/2011 Moderately Satisfactory Moderately Satisfactory 0.81 Moderately 6 12/05/2011 Moderately Satisfactory 2.21 Unsatisfactory Moderately 7 06/30/2012 Moderately Satisfactory 12.74 Unsatisfactory 8 12/26/2012 Moderately Satisfactory Moderately Satisfactory 15.21 9 06/22/2013 Moderately Satisfactory Moderately Satisfactory 16.20 10 12/31/2013 Moderately Satisfactory Moderately Satisfactory 24.60 11 04/09/2014 Moderately Satisfactory Moderately Satisfactory 25.48 vi H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in US$ millions Reallocation of Credit proceeds between components and 12/18/2012 MU MS 15.21 revision of outcome indicators 2 and 3. Partial cancellation of IDA 03/12/2014 MS MS 25.48 financing I. Disbursement Profile vii 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Uzbekistan is an upper low-income resource rich, landlocked country, strategically located in the heart of Central Asia. At the time of appraisal in 2009, its population was 27 million. Having gained independence in 1991, by 2007 the Atlas Gross National Income (GNI) per capita had reached an estimated US$730. Uzbekistan had a young and predominantly rural population (65 percent). Its development strategy included a strong focus on the maintenance of basic public services, including education. Since the mid-1990s, Uzbekistan had devoted 5.6-9.5 percent of its Gross Domestic Product (GDP) to education. Until 2004, education expenditures had concentrated on new programs to upgrade secondary vocational education while funding shortages remained in general education (grades 1-9). 2. The general education sector faced constraints. Participation of the community and other stakeholders as a way to encourage public accountability for school policies, resource allocation, and school management was limited. While the MoPE monitored and openly discussed learning outcomes, there had been no experience with national standardized assessments. With support from the Bank-financed Basic Education Project (First Phase of an Adaptable Program Loan in Support of the Education Reform Program), the GoU took positive steps to address these issues in 2006 through a national standardized assessment of student learning for students in grades 4 and 8, and through school board activation and community participation in school decisions in selected schools, 3. The government was just completing implementation of its School Education Development Program (SEDP) for 2004-2009 to improve conditions of school buildings and learning resources and equipment in schools. Donor financing had also provided support to the SEDP, including textbooks, ICT, distance education, teacher training, equipment and learning materials. Teacher salaries also increased significantly during that period. Most schools continued, however, to need learning materials and resources, student-centered teacher training and other improvements to increase quality and reduce disparities. The government was preparing the next medium-term education program and planned to keep the SEDP Fund to continue development of basic education and increase its focus on preschools. 4. The most recent data available (2004) indicated that enrollment rates were above 99 percent. Learning outcomes and quality of education were a concern. Many factors hindered improvement of learning and quality: overloaded and outdated curricula, insufficient and outdated equipment and learning resources in schools, a lack of teachers’ professional competence to promote active engagement by students in their own learning, no incentives for schools or teachers to change the system or improve its outcomes, limited data for monitoring and evaluation, and limited community and stakeholder participation to encourage accountability. 5. Schools in rural communities, especially if remote, fared worse in most aspects than urban schools. For instance, only 37 percent of remote rural schools had sufficient numbers of teaching manuals as compared with 64 percent in urban areas. The urban/rural disparities in school and teacher characteristics had a disproportionate effect on low-income households, as these were more prevalent in rural areas. 1 6. The GoU was working to improve public finance management including introducing a Treasury system, performance-based budgeting, per capita financing in health and education, and a Medium Term Budget Framework (MTBF). In 2007 the GoU took a number of steps by Presidential decree to adopt a new way of financing schools, moving to begin implementation of per capita financing of schools in order to reduce inequities in the distribution of expenditures and increase the efficiency and equity of school financing. The GoU began to implement per capita financing in 2008 in Bukhara and Ferghana Oblasts and Tashkent City; and work was underway to extend per capita financing to three additional oblasts in 2009. As part of these reforms, responsibility for budget allocations for basic education schools was transferred from the Ministry of Public Education (MoPE) and its regional departments to the Ministry of Finance (MoF) and its regional departments. Basic education schools were declared legal entities and school directors became the primary administrators of budgetary funds and the Director’s fund for top-up of teacher salaries. 7. The project (entitled BEP-2) was designed to be the second phase of a two-phased Adaptable Program Loan (APL). The project was designed to support Uzbekistan’s education reform program, which aimed to make significant improvements in the school education system, including improvements in physical conditions of educational facilities, development of educational standards satisfactory to international standards, and creation of equal access and quality of education for children living in rural and urban areas. The first phase of the APL (BEP1) was designed to build capacity, while BEP-2 was designed to build upon and extend the capacity to encourage greater education reform. 8. The project was aligned with the Country Assistance Strategy (CAS), which included service delivery as one of its four pillars for Bank engagement with Uzbekistan. Education, especially support for improved quality and governance, was highlighted under this pillar. Another CAS pillar supported improved economic opportunities in rural areas. As project schools were in poor, rural areas, the project also supported this pillar. 1.2 Original Project Development Objectives (PDO) and Key Indicators 9. The overall objective of the APL and each of the phases was to continue supporting GoU's efforts to improve effectiveness of teaching and learning through (1) targeted interventions in selected general secondary education schools and preschools in rural poor areas; (2) development of institutional capacity to assess student learning; and (3) adoption of predictable and transparent school budgets in selected oblasts. 10. Progress towards achievement of the objective was to be measured through three key project outcome indicators: • An increase in learning outcomes in grade 4 that is greater in project schools than in non-project schools as measured by a new standardized assessment of learning achievement available for a sample of students in project and non-project schools. • National standardized assessments of student learning carried out periodically in grades 4 and 8 with report analyzing and disseminating the results. 1 1 This is the wording from the Results Framework in Annex 3. In ‘Section C. PDO and key indicators’, the word ‘periodically’ is omitted. 2 • Per capita financing implemented in six oblasts with report analyzing and disseminating the results. 1.3 Revised PDO and Key Indicators, and reasons/justification 11. The Project Development Objective was not revised. Revisions were made to outcome indicators 2 and 3 during a level-2 restructuring (approved on December 18, 2012) subsequent to the mid-term review. The revisions were was as follows: • Outcome indicator 2: National standardized sample-based assessment of student learning carried out in Grade 4 with report analyzing and disseminating the results. o This was a corrective restructuring. Given delays in the procurement teaching and learning materials for Grades 5 to 9 and in the training of teachers, it was no longer technically relevant to attribute any changes in the results of the student assessment for Grade 8 to the project. Further, it had become apparent that no institutional niche within the MoPE had emerged for carrying out regular standardized assessments, and the MoES decided to reduce the number of assessments to be undertaken. As such, this indicator was revised to remove the reference to Grade 8. • Outcome indicator 3: Per-capita financing implemented with report analyzing and disseminating the results. o This was an implementation adjustment. During the first year of project implementation, the Government decided to scale up per capita financing nationally. Therefore reference to six oblasts was removed, and the indicator phrased to encompass national coverage. 1.4 Main Beneficiaries • The main beneficiaries described in the project document were: • Students, teachers, and school management and Board in 1,501 general secondary schools located in low-income, rural areas who would benefit from teaching-learning materials; teacher and school management training to build capacities in child-centered active learning techniques; and grants to implement school improvements plans. • Students, teachers and directors in 598 Kindergartens in low-income, rural areas who would benefit from training to build capacities in child-centered active learning techniques. • The main stakeholders at schools operating in six oblasts benefiting from the strengthening and implementation of per capita financing reforms, designed to improve the transparent and equitable distribution of funds and provide incentives for the efficient use of resources. • The government, MoPE included, would benefit from periodic information being made available on system performance in terms of learning outcomes. 3 1.5 Original Components Component 1: Providing Learning Materials and Resources for Grades Five through Nine of General Secondary Schools (Total estimated cost of US$ 29.9 million of which IDA is US$ 22.8 million) 12. The objective was to improve the quality of teaching and learning in project schools through the effective use of learning materials and resources acquired under the project. The project would finance the procurement of learning materials, equipment and other resources for grades 5-9 in 1,501 general secondary schools. The project would also finance the holding of exhibition fairs at which the full range of materials, equipment and other resources available for procurement would be shown to school boards, teachers, parents and communities. School boards were to assess school needs through an open and participatory process, and then select a list of materials for procurement, working with a budget calculated based on the number of classes and students in grades 5-9, and the remoteness of the school. Component 2: Strengthening School-Based Training for Teachers and Directors in Preschools and General Secondary Schools (Total estimated cost of US$ 1.4 million of which IDA is US$ 1.0 million) 13. The objective was to change teaching in project schools and preschools towards a more interactive, student-centered approach to improve learning outcomes. The component was to develop and introduce School-based Teacher Training (SBTT) as a complement to existing training strategies. The training programs supported by the project would promote the use of student-centered active learning strategies, improvement of student assessment methods, cooperation-based work methods, community mobilization and effective use of modern teaching aids. 14. Four training programs were to be financed: (i) refresher training of 165 facilitators (oblast-level specialists and rayon methodologists) who had been trainers under BEP1; (ii) training directors of project schools and preschools on the teacher training program and on education management; (iii) training teams of school and preschool teachers on the new teaching methods and on adult training; and (iv) training teachers in project schools and preschools on the new teaching methods. Mentoring was to be provided in each training program. The project was to finance technical assistance to develop and adapt teacher training strategies, building on prior work done under BEP1; and the development, production and purchase of training modules and materials. The project was also to finance fees for trainers and mentors, travel expenses, and the accommodation and meals for trainers and trainees. To promote SBTT as a complementary approach to centralized teacher training schemes, the project would also support the development of a SBTT strategy. Component 3: Strengthening School Boards and Providing Competitive Grants to Selected Schools for School Improvement (Total estimated cost US$ 2.7 million of which IDA is US$ 2.7 million) 15. The objective was to increase further the involvement of school boards in project schools and provide competitive funding for school improvement. Building on the activation of school boards from BEP1, BEP2 was to continue to build capacity and move school boards to an even higher level of participation by encouraging them to develop proposals for Competitive Grants to finance Sub- 4 projects aimed at school improvement. The component therefore supported the establishment and implementation of a Competitive Grants program for school boards to design and implement their own plans to improve school quality. Positive discriminatory criteria were to be introduced in the annual allocation of funds to allow a significant number of high-risk project schools, the poorest and lowest performing project schools, to tap the competitive funds with good quality proposals. The proposals were expected to aim at improving the classroom pedagogical process and the school pedagogical climate. Component 4: Improving Education Financing, Budgeting and Management (Total estimated cost US$ 0.5 million of which IDA is US$ 0.3 million) 16. The objective was to improve education financing, budgeting and school management in selected oblasts. Building upon the GoU’s implementation of PCF in 2008 in Bukhara Oblast, Ferghana Oblast and Tashkent City, the BEP-2 was to support the GoU’s expansion of per capita financing to three additional oblasts, Namanghan, Syrdarya and Khorezm. The component would finance training on school budget planning, execution and accounting; establishment of a school financial management and public expenditure monitoring system, including software development and installation of a school financial management and accounting system on a pilot basis in the six oblasts; and monitoring and evaluation activities. Component 5: Supporting Project Management and Results Management (Total estimated cost US$ 1.5 million of which IDA is US$ 1.2 million) 17. The objective was to support project management, implement national standardized assessment of student learning, and evaluate project effectiveness. The component was to finance: (i) project management, including procurement, financial management, monitoring and evaluation and support to implement specific project activities; (ii) a final standardized national learning assessment for project and non-project schools; (iii) standardized national assessment for 4th and 8th grades; (iv) updating and analysis of the baseline survey collected during BEP1; (v) periodic monitoring and reporting of key educational indicators in a sample of project and comparable non-project schools; (vi) updating of the agreed BEP2 project’s indicators; and (vii) carrying out of an impact evaluation study of the overall BEP2, including on-site assessment of the classroom pedagogical changes towards a more student-centered approach and effective use of new learning materials. 1.6 Revised Components 18. In a level-2 restructuring (approved on 18 December, 2012), changes were made to three components, as detailed in the following table. Relevant features prior Changes at restructuring Reasons for change Comments to restructuring Component 1. Original To support an additional To use cost savings of US$ 9.53 target was 1,501 schools. 694 schools. To be located million after successful completion in oblasts targeted by of procurement of goods. project, and selected on the basis of previously agree criteria. Would benefit from multimedia 5 classrooms equipped with modern desktop computers and other ICT devices. Component 4. Originally Instead of an evaluation of GoU scaled up PCF to whole There is no mention designed to support scale the phased roll-out of PCF country in 2010, instead of six at restructuring of up to three additional in six oblasts as originally oblasts as had been anticipated changing or dropping oblasts on top of existing envisaged, this component during project design. other activities that three oblasts. would finance a rigorous were included in the Development of software evaluation of the PCF of component, even and systems for school schools across the entire though at mid-term management, accounting country. review it was clear and monitoring, as well as that the bulk of these reform monitoring and a activities would be final evaluation, to cover financed by the GoU all six oblasts. and not the project. Component 5. Originally Grade 8 and Grade 4 (year Given delays in the procurement of One of the reasons for included learning 2) assessments were teaching and learning materials for dropping the national assessment exercises of dropped. Grades 5 to 9 and in the training of Grade 8 exercise may Grade 4 (in year 2 and 4) teachers, it was no longer not have been fully and Grade 8 (in year 3). technically relevant to attribute to valid in that the the project changes in the results of exercise was not a student assessment for Grade 8. meant to be linked to There were implementation delays the provision of that led to the dropping of the teaching and learning Grade 4 assessment scheduled for materials for Grades 5 Year 2. As it had become apparent to 9, which of course that no institutional niche within the were provided only to MoPE had emerged for carrying out project schools. regular standardized assessments, the MoES decided to reduce the number of assessments to be undertaken. 1.7 Other significant changes 19. On the request from the MoF dated February 28, 2014, the amount of US$ 2.5 million (due to savings from the last procurement of ICT equipment for 694 additional schools) was cancelled in March 2014. 2. Key Factors Affecting Implementation and outcomes 2.1. Project Preparation, Design and Quality at Entry 20. BEP-2 was the second phase of a US$ 28 million adaptable program loan (APL). It was approved on June 23, 2009, and became effective on March 11, 2010. The BEP-2 was designed to build on the capacities developed under BEP-1, the first phase of the APL. The BEP-2 kept the same PDO and a similar set of activities, but had a longer duration of four years. (See Box 1). 21. The project’s background analysis provided a reasonable foundation for project design. Learning outcomes were cited as a fundamental concern, as was the lack of national standardized learning assessments. Immediate causal factors were provided, including an unsatisfactory availability of teaching-learning materials (particularly in rural areas); a lack of teacher competence in active learning techniques; and a limited community participation in the school to encourage 6 accountability. The inequitable distribution of funding was identified as an underlying problem. The analysis however lacked details on the in-service teacher training system (to explain the lack of teacher competence); as well as the existing system for monitoring learning achievements (to clarify why system results are generally unavailable and a standardized national assessment is required). Box 1.: Design, achievements and lessons of BEP-1 built in to BEP-2 The BEP was initially designed as a one four-year investment project but later changed into a two- phased Adaptable Program Loan (APL). The two projects shared the same PDO; there was strong continuity in the design; and the achievements and lessons of BEP-1 contributed to the design and results of BEP-2. In general, the BEP aimed to make significant improvements in the school education system, including improvements in physical conditions of education facilities, development of educational standards satisfactory to international standards, and creation of equal access and quality of education. The BEP-1 was designed to build capacity, while the BEP-2 was designed to build upon and extend the capacity to encourage greater education reform. In particular, the BEP-1 helped to make significant progress in capacity building in several key areas, including: • Community involvement in school decision-making increased at project schools through the training of school boards and their active involvement in needs assessment and selection of learning materials for the primary grades. BEP-2 built on this by extending the needs assessment and selection of learning materials to cover grades 5-9. • Teacher training materials were developed and facilitators and experts trained to build the capacities of teachers in the area of student-centered active learning. This enabled BEP-2 to implement this training to cover all project schools. • Per capita financing was designed and implemented in two oblasts and Tashkent City. This enabled BEP-2 to target scaling up to three more oblasts, thus expanding the transparency, predictability, efficiency and equity entailed by the reform. • The pilot of a national standardized student assessment was carried out for grades 4 and 8, with the results analyzed for grade 8. This provided a baseline for setting a Grade 4 learning target for the PDO for BEP-2, and also laid the foundation for planning a series of learning assessments under BEP-2. BEP-1 also provided lessons, which were built into BEP-2 design. These included the need to have longer time frames for decision-making, procurement and ensuring community participation inherent in a highly hierarchical and controlled environment. Indeed, BEP-2 was a four-year project, whereas BEP- 1 was designed to last 18 months. Another lesson was the importance of brining in the experience of other countries, and this was built into BEP-2 design by including study tours in key technical areas. Finally, as BEP-1 was the first Bank-financed education project without a PIU in Uzbekistan, MoPE staff needed to learn Bank procedures and how to implement without a PIU; under BEP-2, implementation arrangements were designed to provide implementing departments of the MoPE with adequate capacity. 22. The components were designed to respond to the analysis and to complement one another so as to contribute synergistically to the achievement of the PDO, which was to be measured along three dimensions. (See Graphic 1 below). An improvement in learning achievements was to be brought about through making teaching-learning materials more available, building the capacities of teachers and school directors to introduce pedagogically more effective interactive teaching-learning techniques, and supporting the implementation of School Implementation Plans that aimed to increase the quality of learning and improve the availability of pedagogical materials. However, the 7 target for learning achievements pertained only to Grade 4, and thus neglected to take into account the contribution of C1 (teaching-learning materials for Grades 5-9); rather the target seems to have been the planned outcome of providing teaching-learning materials under BEP-1 and the continued support under BEP-2 to teacher training and school improvements. This resulted in a fundamental disconnect between the first outcome measured and the actual interventions financed under BEP-2. The expansion and analysis of per capita financing was to be brought about through the technical and other support provided under C4. The standardized and sample-based assessment of learning outcomes was to be achieved by C5, which financed the design and implementation of such assessments. 23. The components on in-service teacher training (C2) and learning assessments (part of C5) might have usefully included activities to understand, reform and/or build institutional capacities, in order to promote the conditions required for the long-term sustainability of the innovations supported by the project in these two areas. In the case of SBTT, there was no analysis in the PAD of the existing training delivery system (and why an alternative model was needed as a complement), and the parameters of the model proposed in the PAD had constraints (particularly in terms of locating training at the district level where there are no in-service training institutions or staff) that ran a risk of being resisted by existing institutions. With respect to learning assessments, the PAD did not analyze the existing system of assessing learning or identify its shortcomings, and project activities did not foresee a comparison of the assessment approach supported by the project with the existing system. Such exercises would have highlighted various technical and economic advantages of the project-supported approach, and might have helped lay the foundations for ensuring the sustainability of standardized assessments. The funds allocated for learning assessments under C5 were very limited, particularly in light of the prominence of learning assessments in the PDO. 8 Graphic 1: Simplified display of links between components and PDO indicator targets Main Component PDO Indicator Activities Targets Component 1: Teaching- learning materials grades 5-9 (plus TLMs 1-4 provided under BEP-1) Component 2: Building teaching capacities at An increase in learning primary and secondary level outcomes in Grade 4 in interactive and child- centered methods Component 3: School improvement plans Component 4: Increased transparency, autonomy, Implementation of PCF efficiency and equity through per capita financing reforms Component 5: Conducting National standardized learning assessments sample-based assessments 24. The project was well aligned with the School Education Development Plan (SEDP) in force at that time, which focused on improving the physical conditions of the learning environment. The government satisfied two conditions (‘triggers’) to move from phase one to phase two, indicating government commitment to the project. Substantial government commitment was also built into project design, with counterpart contributions estimated at US$ 8 million and covering inter alia the delivery of goods, payment of taxes and duties, certain training costs, and study tours. 25. Adequate implementation arrangements and capacity were foreseen during project preparation. Rather than establishing a PIU, implementation responsibility laid with the MoPE. The Deputy Minister acted as project Director, with responsibility for overall leadership, management and oversight. The relevant MoPE and Ministry of Finance departments were made responsible for the implementation and coordination of the components and their activities, as well as for procurement and financial management. The implementation arrangements were implicitly designed to build institutional capacity by making the relevant departments responsible for the implementation of their respective parts of the project, supporting them with consultants to augment their capacity and manage the additional tasks entailed by the project. 9 26. The risks were generally well defined and provided with appropriate mitigation measures. With respect to implementation capacity and sustainability, more might have been said about sustainability risks pertaining to project-supported innovations (particularly SBTT, PCF and learning assessments) and corresponding mitigation measures. The quality of the design of the results framework and monitoring and evaluation arrangements is discussed in Section 2.3. 2.2. Implementation The following factors affected project implementation. 27. Government’s commitment. The political stability and GoU consistency in supporting the sector’s goals generally, and the project’s objectives specifically, contributed to successful implementation. Further, the bulk of the government’s significant counterpart contributions were realized and in some cases exceeded (see Section 5.2), and the government was supportive in resolving issues hindering implementation. However, fewer learning assessments were carried out than planned, the result in part of a failure to find the right institutional niche within the government’s administrative apparatus. 28. Cost savings. There were significant cost savings (US$ 9.53 million) during the procurement process for goods under Component 1. This was the result of the effectiveness of the bidding process, as well as the prudently high unit prices built into the procurement plan to start with. These savings were used to cover an additional 694 schools, which were located in oblasts targeted by the project and selected on the basis of previously agreed criteria. These schools benefited from computers and other ICT devices to allow teachers and students to use contextualized digital learning content. 29. Implementing Agency (IA) Factors. The MoPE’s performance in terms of management effectiveness, staffing adequacy and quality, use of technical assistance, monitoring and evaluation and beneficiary participation generally enabled the project’s successful implementation. However, a hesitant follow-up by the MoPE of issues hindering School Boards to absorb school grants provided under Component 3 led to only 303 of 500 beneficiary schools receiving the full grant, with the remainder receiving 70 percent of the grant. Not all activities were completed; a learning assessment exercise for Grade Four scheduled for Year Two was never conducted. There was some improper sequencing, largely the result of implementation delays. In particular, the bulk of the training under Component 2 took place after the Grade 4 learning assessment exercise in Year Four. 2.3. Monitoring and Evaluation (M&E) Design, Implementation and Utilization 30. Design. The monitoring framework was designed to capture project outcomes and outputs. Project development outcome indicators focused on improved learning achievements (to reflect effects of inputs to project schools), national learning assessments (to reflect the project contribution to the conduct of periodic national assessments), and PCF (to reflect project support for scaling up). However, there were serious shortcomings in the framework indicators. The first PDO indicator was overly ambitious and raises a question of attribution. It is difficult to ensure that all inputs are effectively delivered on a timely basis in order to produce learning results within four years; the schools operate in a multi-factorial environment, some of which might affect learning in ways that overwhelm project inputs; and the education system might change learning benchmarks (e.g. through 10 curriculum reforms). Further, by restricting the measurement to Grade 4, the indicator neglected to measure any effects of Component 1 (accounting for the largest share of the loan), which purchased teaching-learning materials for grades 5-9. The indicator might also have been more tightly phrased in terms of the project schools’ performance relative to the non-project schools, so as to capture ‘difference-in-difference’ rather than simply ‘an increase.’ Including a PDO pertaining to learning assessments (entailing three separate national assessments within a four-year period), was disproportionate to the activities identified and resources allocated. The third PDO indicator captured processes (PCF implementation) rather than outcomes (e.g effects on equity or efficiency). At the level of intermediate outcomes, the indicator for Component 2 pertaining to numbers of teachers trained did not distinguish between teachers directly and indirectly trained; these two are essentially different, and at the least the indicator should have been disaggregated. (The project financed the two-week training of ‘key teachers’. In turn, these key teachers trained their colleagues over a shorter period, during which time project-financed trainers visited the schools to support this second level of training). (See also Annex 2). 31. Implementation and utilization. Arrangements were largely implemented as designed in the PAD, regular reporting was conducted, and the information was discussed and used to track progress, identify bottlenecks and course corrections, and contribute to policy review and development. For instance: • Component 2 monitoring and evaluation information was regularly reviewed with MoPE, leading to improvements in model design, revision of materials, reinforcement/replacement of trainers, and decisions on how the training would be incorporated into national systems. • Component 3 monitoring highlighted difficulties in releasing the final tranche representing 30 percent of grants’ allocation to school boards benefiting from the first wave of school grants. These were discussed and eventually resolved. 32. Component-wise project monitoring and evaluation of implementation and intermediate outcomes were generally robust. However there were some aspects that might have been improved. • The learning achievement exercise was conducted before the 2nd and 3rd (of 3) rounds of teacher training were completed, and therefore did not pick up the effects of the training. Further, the PAD RF called for two Grade 4 exercises (in years 2 and 4), whereas only one was done (year 4). • The intermediate outcome indicator for Component 2 was implicitly based upon having a baseline measurement, but this was not done. Instead, the evaluation used two samples: teachers directly trained (in round 2) under the project; and teachers from schools that would be indirectly covered under round 3, but who would not be directly trained. That is, the ‘trained’ group consisted of ‘key’ teachers who had just been trained; while the ‘untrained’ group consisted of teachers who would not be directly trained by the project, but who worked at schools where key teachers were about to be trained by the project. The ‘key’ teachers who received direct training from the project were generally Heads of Methodological Units. These are teachers who tend to be relatively experienced and qualified. By comparing such teachers with teachers in a control sample that generally excludes Heads of Methodological Units, there may have been a bias in the sampling such that the ‘trained’ sample may have contained better teachers than the ‘untrained’ sample. Further, the evaluation was meant to cover 10 percent of general secondary project schools, but in fact the sample contained only 100 (or 6.6 percent) general secondary project schools. The evaluation of PCF was not done. 11 2.4. Safeguard and Fiduciary Compliance 33. The project was rated C for environmental safeguards as major civil works were not envisaged. 34. Fiduciary performance was rated Satisfactory over the life of the project, except during two implementation support visits, when financial management rating was downgraded to 'moderately satisfactory'. The reason for downgrade was due to identification of ineligible project expenses in relation to income tax expenses. This issue was subsequently clarified as they pertained to interpretation of country financing percentages but a reimbursement from the GoU to the Bank is still pending. Procurement performance was considered moderately satisfactory. The MoPE gradually gained experience during the life of the project and efficiently managed a large number of contracts. Significant savings were generated through lower prices obtained from competitive bidding. Procurement under the school grants scheme could have been designed so as to allow greater oversight of local School Boards. Procurement capacity could have been strengthened with targeted capacity building within MoPE while managing risk and without overly relying on consultants. 2.5. Post-completion Operation/Next Phase 35. There are post-completion considerations to take into account for the SBTT (under Component 2), per capita financing (Component 4) and learning assessments (Component 5). As amplified in Annex 2, it is unclear to what extent SBTT can be incorporated into national in-service training systems. In its role as Grant Supervisor for the forthcoming Global Partnership for Education (GPE) grant, the Bank will continue to explore with the MoPE and other donor partners how this mainstreaming can take place. With respect to per capita financing, the government has already undertaken extensive measures to ensure the reform’s sustainability. The Bank should continue to encourage the government to conduct an evaluation; in light of the government’s intention to expand PCF to other sectors, the Bank may also consider the planned IDA operation in tertiary education as an opportunity for such expansion. As for learning assessments, the GPE presents an opportunity for the government to continue carrying out standardized assessments at grades 4 and 9 with a view to longer term institutionalization. This should be complemented by assistance to help government identify the objectives, uses and tools of standardized assessments, along with their advantages and drawbacks. 3. Assessment of Outcomes 3.1.Relevance of Objectives, Design and Implementation Rating: Moderately Unsatisfactory Objectives 36. The PDO retains high overall relevance at the time of the ICR (as it did at the time of design). Efforts to improve the effectiveness of teaching and learning remain relevant, including through targeted interventions such as those supported by the project. The effort to develop institutional capacity to assess student learning remains highly relevant, particularly in the form of standardized assessments as supported by the project. Finally, the need to ensure predictable and transparent school budgets in selected oblasts remains relevant. The ongoing relevance is underlined by the 12 congruence between the PDO and the recently approved Education Sector Plan (ESP) for 2013-17; as well as the Country Partnership Strategy for FY 2012-2015. Design 37. The design was based on a generally sound situation analysis that retains much of its relevance. The PDO was aligned with national development priorities and sectoral constraints that continue to be relevant; and the components were designed to achieve their own objectives as well as contribute jointly to the project’s development objective. The design’s relevance was undermined in that the target chosen to measure the teaching effectiveness dimension of the PDO did not correspond to the grades that received the greatest project support (in dollar terms). The implementation arrangements were generally adequate and built upon those first tested under BEP-1, whereby implementation responsibility was given directly to the MoPE and MoF, which were provided with technical consultants to augment their capacity including for fiduciary aspects of implementation. These arrangements continue to be relevant, and can mostly be used again in future. Implementation 38. The project’s implementation retained its relevance through a timely restructuring, which enabled a significantly increased coverage for Component 1. At restructuring, the results framework and evaluation activities were adjusted to be in line with the accelerated schedule to scale up PCF; and to drop any reference to a Grade 8 learning assessment, which was no longer taking place under the project. However, the restructuring was a missed opportunity to revise the results framework; in particular, the first outcome indicator might have been modified to take into account the largest component (C1). Other areas were also overlooked during the restructuring. First, it was already clear at that point that most of the activities in Component 4 (pertaining to PCF) were being fully financed by the Ministry of Finance, and the resources originally allocated to those activities should have been reallocated to other project activities. Similarly, under Component 2 (pertaining to SBTT), the government had already taken financial responsibility for certain training activities that were originally part of the project’s financed activities, thus those funds should also have been reallocated. Finally, the mid-term review and restructuring were also an opportunity to tackle the sustainability issues mentioned above. 3.2.Achievement of Project Development Objectives Rating: Moderately unsatisfactory 39. Project achievement is rated moderately unsatisfactory as the project development objective was partially achieved, with significant shortcomings. The project contributed to an increased effectiveness in teaching at the primary and secondary levels, but the evidence for improved student learning at the primary level is mixed while no data were adduced for the grades that the project’s largest component was targeting. The national standardized assessment at grade 4 was carried out but was not institutionalized. Per capita financing was scaled up nationally; and reporting was conducted, analyzed and disseminated with project support. 40. Therefore the target of the first outcome indicator was partly achieved; the second outcome indicator’s original target was not met, but the revised target was achieved; and the third outcome indicator’s original target was exceeded, while the revised target was achieved. Given that the first 13 outcome indicator was designed to reflect interventions in project schools, and the bulk of project funding was spent on these interventions, the performance of this PDO receives a preponderant weight in the overall rating. The analysis below is broken down by the three PDO indicators, which pertain to improvements in learning, national standardized assessments, and education financing reforms. 41. Improvements in learning. The first outcome indicator target was partly achieved, in that the targeted schools experienced a significant increase in Grade 4 math scores (from 478 at baseline to 514 at completion) that was greater than in non-project schools (from 487 to 496); reading scores stayed substantially the same (485 at baseline and 482 at completion), though in non-project schools there was a significant decline (from 511 to 485). 2 That is, while the project schools did not achieve absolute gains in test scores in both subjects (as targeted by the project), there was an improvement in both subjects relative to non-project schools. There was no measurement of learning at the secondary level, even though the bulk of the BEP-2 resources was targeted at that level. As teaching effectiveness is a key contributor to the learning process, it is essential also to consider measured changes in teacher behavior and practice. There is evidence for an increased teaching effectiveness of teachers (at both primary and secondary levels) who received project-financed training. An independent evaluation found that 76.2% of project-trained teachers had adopted more effective student-centered teaching practices, a substantially higher prevalence than among those not trained (62.1%). 42. By design, an improvement in learning outcomes should also have been the synergistic result of an increased availability of key inputs of teaching and learning materials and an improvement in school planning and management. Teaching-learning materials for Grade 4 were supplied under BEP-1 at least 3 years before the learning assessment. (BEP-2 was designed to be continuous with, and build upon, BEP-1). The bulk of the Grade 5-9 learning materials supplied under BEP-2 arrived in the second half of the third year and at the beginning of the fourth year, leaving limited time in which to have an impact on learning outcomes. Some of the learning materials acquired under the school improvement plans were relevant to the primary curriculum; delivery to schools started in the 2nd year of the project, in time to have an effect on learning before project closure. The community and School Board participation that was stimulated under Component 3 contributed to participatory school improvement planning and management, and sustaining a learning culture in school. 43. Given that the project substantially achieved all its targets in increasing teaching effectiveness through training and mentoring of proven practice, in making teaching-learning materials available, and in enabling all targeted schools to implement an Improvement Plan; but only partially met its learning targets for Grade 4 while failing to provide evidence for improved learning in grades 5-9: the ICR team concluded that the project was moderately unsuccessful in supporting the GoU’s efforts to improve the effectiveness of teaching and learning through targeted interventions in selected general secondary education schools and preschools in rural poor areas. 2 The increase in maths scores in project schools was statistically significant (95% confidence interval); the increase in maths scores in non-project schools was not significant. The decline in reading scores in project schools was not statistically significant; the decline in reading scores in non-project schools was significant. 14 44. National standardized assessments. The second revised outcome indicator target was fully achieved, although partially achieved when compared to the original target. In the PAD, the indicator referred to Grade 4 and 8 assessments, with exercises scheduled for years 2, 3 and 4; in the Restructuring Paper, reference to Grade 8 was removed as requested by the government, and the RF targets included only one Grade 4 assessment in year 4. The Grade 4 assessment was successfully carried out under the project. 45. Education financing reforms. The third outcome indicator target was achieved, both in its original and revised forms. Initially, the project had targeted implementing PCF in three additional oblasts. However, in 2010, the Government scaled up PCF nationally and fully financed its implementation – including activities originally foreseen to be financed by the loan (e.g. training on school budget planning, execution and accounting; establishment of a school financial management and public expenditure monitoring system, including software for a school financial management and accounting system). After the mid-term review, the indicator was revised to cover the whole country rather than the three additional oblasts. The indicator also includes ‘report analyzing and disseminating the results.’ The PCF was implemented nationwide and a report was done and disseminated, though a rigorous evaluation was not undertaken as foreseen under the project. 3.3.Efficiency 46. Traditional values for measuring efficiency gains such as the project’s Economic Rate of Return or Net Present Value were not calculated at appraisal. Instead, the economic analysis put its emphasis on equity in the allocation of educational resources. Such choice of methodology precluded an ex- post evaluation to quantify the project costs and benefits. An alternative to assess the project’s cost effectiveness was also constrained by the lack of data to construct substitute scenarios to compare their costs and benefits for demonstration of value for money and or the least cost. Given the above, the ICR team re-visited the original economic analysis of the project, focusing on the gains generated by the expansion of per capita financing. Annex 3 details the analysis. The ICR team concluded that the national expansion of PCF has in fact led to a more equitable allocation of resources, and more efficient spending at the system level. Specifically, PCF has led to a significant reduction in the variance in allocation of resources across schools from 60 to 10 percent, as well as to the rationalization of the school network as evidenced by an increase in the average classroom size and in the teacher-student ratio. The analysis also concluded that improved pedagogy of teachers resulting from project interventions contributed to better student learning outcomes, a lower cost scenario to achieve better student learning than higher cost of remedial education. 3.4.Justification of Overall Outcome Rating Rating: Moderately unsatisfactory 47. The project’s design and objectives were substantially relevant at its onset and remain so today. There were significant shortcomings in achieving the project’s main objective – to improve the effectiveness of teaching and learning through targeted interventions in rural poor areas - in that only one of the two primary-level learning targets was achieved while no measurement of learning was undertaken in those grades that received the bulk of the project’s resources. There were moderate shortcomings in developing the institutional capacity to assess student learning, as there remain concerns regarding the medium to long-term sustainability of this intervention. There were only minor shortcomings in achieving the adoption of predictable and transparent school budgets in 15 selected oblasts; while the PCF reform was scaled up beyond initial targets and successfully yielded efficiency and equity gains, a thorough evaluation remains to be done. Given the project’s substantial relevance and efficiency, but taking into account significant shortcomings in achieving the main objective and minor or moderate shortcomings in achieving the other objectives, the overall outcome rating is moderately unsatisfactory. 3.5.Overarching Themes, Other Outcomes and Impacts (a) Poverty 48. The project contributed to more equitable access to quality basic social services through targeted support to schools in poor and disadvantaged regions and communities. As already stated, the nationwide introduction of PCF led to a significant reduction in the variance in allocation of resources across schools from 60 to 10 percent. The targeted interventions helped equalize the opportunity for poor households and their children of improved access to schooling, increased availability of didactic materials in schools in relatively poor areas, and consequently to improve learning. (b) Governance 49. The project also supported interventions contributing to enhanced governance of the decentralizing system. The roll-out of School Boards system-wide enabled stakeholder participation in decision making on school development and planning which empowered the voice and participation of stakeholders and citizen engagement. The introduction of per capita financing has made the allocation and use of resources transparent. 4. Assessment of Risk to Development Outcome Rating: Moderate 50. The PDO was measured using three dimensions: increased learning outcomes in project schools; the periodic conduct of national standardized assessments with results being analyzed and disseminated; and the implementation of per capita financing. There is a low risk that the learning outcome results in project schools will be reversed. Any contribution made by the training should strengthen over time as teachers gain more experience in applying the teaching-learning techniques and the school’s pedagogical culture strengthens in these areas. More broadly, the teaching-learning materials on interactive learning developed by the project have been incorporated into the curricula of teacher training institutions, and a forthcoming project under the Global Partnership for Education (GPE) includes training targeting this competency. Any contribution made by the teaching-learning materials should last at least as long as the useful life of the materials. The strong stakeholder commitment to the objectives of the project also augurs well for maintaining the gains. There is a deep commitment to funding public education, and communities appear to be supportive of their local schools. 51. The risk to sustaining the periodic conduct of national standardized assessments has been moderated recently, while this report was being prepared. According to updated information on the GPE’s forthcoming operation, the government has agreed to standardized assessments at grade 4 and grade 9 during GPE project implementation over the next three years. This recent decision has enhanced 16 the probability that the government will sustain such exercises over time and continue building on the technical capacities for test design, sampling methods, and administration procedures supported by the BEP-2. 52. There is a negligible to low risk that the per capita financing reforms will be reversed. The government greatly accelerated the reform’s implementation schedule; remains strongly committed to the reform, and continues to invest extensively in capacity building and the development of information and IT systems to strengthen the reform’s effective implementation. The government has plans to extend the principles of PCF to other educational sub-sectors. 53. Given these differential risk ratings for the different dimensions of the project’s development outcome, the overall risk to development outcome is moderate. 5. Assessment of Bank and Borrower Performance 5.1 Bank performance (a) Bank performance in ensuring quality at entry Rating: Moderately Unsatisfactory 54. While in most respects the Bank’s performance at entry was satisfactory, there were serious shortcomings in supporting the client to develop the right strategic approach to build institutional aspects into the project design, and to elaborate an adequate results framework and indicators that captured the causal linkages between project inputs, outputs and outcomes. . (See sections 2.1 and 2.3). (b) Quality of supervision Rating: Moderately Satisfactory 55. The Bank’s supervision focused on development impact, supervision of fiduciary aspects, adequacy of supervision inputs and processes, candor and quality of performance reporting, including the timely carrying-out of the MTR and restructuring and the role in ensuring adequate transition arrangements. However, the restructuring might have better mitigated the shortcomings of the M & E design, particularly the lack of a project development indicator that took into account the project’s largest component, addressed sustainability issues, and adjusted the financing plan in light of changes in implementation. The performance is therefore rated as moderately satisfactory. (c) Justification of rating for overall Bank performance Rating: Moderately Unsatisfactory 56. The rating is based on the serious shortcomings of quality at entry and moderate shortcomings in supervision in three important areas:, supporting the client to develop the right strategic approach, to build institutional aspects into the project design, and to elaborate an adequate results framework. 17 5.2 Borrower Performance (a) Government performance Rating: Satisfactory 57. Government ownership and commitment to achieving the development objective was strong. The enabling environment in the education sector was conducive to the successful implementation of the project. Implementation issues were generally well resolved, though issues that prevented either the timely or full implementation of certain activities could have been resolved more quickly with stronger oversight of the IA. (These issues are discussed in Annex 2 and summarized below). The government made significant financial contributions to the project, as per project design. However, none of the study tours foreseen in the PAD (to be paid by government) were undertaken. (b) Implementing Agency (IA) performance Rating: Moderately satisfactory 58. The project was implemented by the MoPE with the assistance of consultants. The MoPE’ss commitment to achieving development objectives was strong. The implementation of monitoring and evaluation arrangements provided by project design, including the utilization of M&E data in decision-making and resource allocation, was done well. For Components 1-3, the IA elaborated monitoring arrangements beyond the minimum specifications of the PAD, and the IA seems to have extensively monitored implementation progress, including keeping extensive documentation. Information was used in decision-making and resource allocation. The fiduciary aspects of project implementation were acceptable. 59. At the same time, the IA’s performance could have been better if the following issues had been addressed on a timely basis. Under Component 3, only 303 out of 500 School Boards that were awarded a school grant fully used both the first and second tranches (of 70 and 30 percent). The Bank supervision team noted in the final ISR that ‘intensive support to schools at the beginning of project implementation by the client team would have ensured greater capacity building and ownership of the process in the schools.’ This in turn would have enabled a higher use of both tranches. Further, procurement delays resulted in teaching-learning materials arriving later than originally scheduled; and roughly two-thirds of teachers were trained behind schedule, in the last nine months of the project, such that the effects of these trainings could not be picked up in the learning assessment exercise for Grade 4 conducted in Year 4 as a project evaluation exercise. Implementation delays led to the Grade 4 learning assessment scheduled for Year 2 not being conducted. (c) Justification of rating for overall Borrower performance Rating: Moderately satisfactory 60. While the Government’s performance was satisfactory, there were moderate shortcomings in the performance of the IA in terms of resolving disbursement issues for School Boards on a timely basis, and the timely and proper sequencing of activities. 18 6. Lessons Learned 61. Implementation arrangements must continuously strive to accommodate yet build MoPE capacities. The need for an implementing agency is dictated by the lack of MoPE capacities to implement a project given its current structure and staffing levels, as well as the different fiduciary procedures followed by the Government and the Bank. Under BEP-2, instead of creating a PIU, the project hired a team of consultants who were to support MoPE and MoF departments, with the departments having direct implementation responsibility for the components. It is undoubtedly difficult to reconcile MoPE staffing levels and structure with the implementation needs of a project, particularly when different fiduciary procedures are followed by the Government and the Bank. Nonetheless, more might be done to increase MoPE implementation ownership and capacity within existing institutional arrangements. For example, MoPE departments that already have strong implementation capacity (e.g. in procurement) should take a stronger role. While all implementing team consultants worked in coordination with the relevant department, having among their terms of reference the duty to transfer capacity could have strengthened the transition and long term capacity. 62. Attention to developing sustainable institutional and budgetary arrangements for innovations should be built into project design and implementation. As explained in Sections 2, 5 and Annex 2, project design and implementation gave relatively little attention to the institutional and budgetary reforms that would be required to enable project innovations in Component 2 and 5 to become sustainable. This in turn contributed to the SBTT model and project approach to learning assessments not being formally adopted for eventual scale-up by the education sector. Therefore the design and implementation of innovative models should from the beginning take into account or foresee the issues of sustainable scale-up. For instance, project activities in such cases should include: an analysis of existing systems and their shortcomings, along with some identification of the superior aspects of the model; and the identification of institutional and financial requirements of scaling up the model, and contrasting these with existing institutions and budgets, thereby leading to the identification of institutional and budgetary reforms that would be required. 63. The purpose of learning assessments should be defined, and the approach designed accordingly. One of the key factors impeding the institutionalization of standardized assessments is the absence of a clearly defined assessment strategy. While the project implemented sample-based testing, the MoPE was focusing its effort to monitor system performance using an existing monitoring tool without thinking through the purpose of different assessment instruments. Countries that have a functioning assessment system should have a balance between summative and formative assessments. Institutionalizing standardized testing could help MoPE to track and monitor system wide performance and student learning trends, while classroom-based assessment would provide more effective support for teachers to measure progress of student learning and improve teaching. Assessment remains an area where further assistance may be needed to put in place a coherent strategy to clearly define the purpose of the assessment at system, school and classroom levels and align the instruments and support for capacity building. 64. A properly designed results framework is essential to project design and evaluation. The indicators and targets must capture not only the project outputs, but also the causal relations between these and the project outcomes. Further, the outcomes must be carefully articulated to capture the PDO, and must realistically take into account issues of implementation timing and attribution. Deficiency in these aspects makes it subsequently difficult to assess project performance. In 19 retrospect, the BEP-2 outcome capturing the teaching effectiveness dimension of the PDO should have been more realistically restricted to achieving improvements in teaching-learning practices (including the use of newly available teaching-learning materials) at both primary and secondary levels. This would have enabled one to discern continuity with the BEP-1 while, more importantly, setting a realistic yet rigorous standard against which the success of the BEP-2 could be measured. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 65. The Borrower made extensive comments on the ICR shared. The comments were incorporated in the finalized ICR where appropriate. The details of the Borrower comments are provided in the Annex 7. (b) Cofinanciers (c) Other partners and stakeholders 20 Annex 1. Project Costs and Financing (a) Project Cost (Loan proceeds) by Component (in US$ Million equivalent) Appraisal estimate Actual/Latest Percentage of Components (US$ mln.) Estimates (US$ mln.) Appraisal 1. Providing Learning Materials and 21.53 22.49 104.46 Resources for Grades Five through Nine of General Secondary Schools 2. Strengthening School-Based Training 0.92 0.19 20.65 for Teachers and Directors in General Secondary Schools and Preschools 3. Strengthening School Boards and 2.53 1.95 77.08 Providing Competitive Grants to Selected Schools for School Improvement 4. Improving Education Financing, 0.28 0.002 0.71 Budgeting, and Management 5. Supporting Project Management and 0.97 0.82 84.54 Results Management Total Baseline Cost 26.23 25.45 97.03 Physical and Price Contingencies 1.77 - - Total Project Costs: 28.00 25.45 90.89 Total Financing Required: 28.00 25.45 90.89 (b) Financing Actual/Latest Appraisal Estimates Source of Funds (mln. US$) Estimates Percentage of Appraisal (mln US$) 1 Borrower 7.95 0.93 11.70 International Development Association (IDA) 28.00 25.45 90.89 Total 35.95 26.38 62.33 1 Without considering the GoU’s contribution in the form of tax and customs privileges and the PCF roll-out. 21 Annex 2. Outputs by Component Component 1: Project intermediate Baseline Indicator after Target Achievements at project closure outcome indicators revision, if applicable Percent of project 0% N.a. 100% (or 1,501 100% (or 2,198). schools that have schools) 1,501 schools received materials, as received the requested originally planned. Savings of learning materials US$US$ 9.53 million were realized during procurement. After level-2 restructuring, a second round of 694 schools were selected and received ICT equipment. Percentage of teachers 0% N.a. 50% 93.8%. at sampled project schools where learning materials acquired under the project are used effectively in the classrooms. 1. Component One exceeded its intermediate outcome targets. The project planned to, and did in fact supply 1,501 general secondary schools with teaching-learning materials for grades 5-9. As per project design, the materials were selected from a list by the schools themselves. The list was designed by the MoPE, and the school’s selection was elaborated and finalized by School Boards in consultation with teachers, school management, students and parents. There were US$ 9.53 million savings as an outcome of the procurement of these materials. Following a level-2 restructuring (approved on December 18, 2012), the project financed materials for a further 694 schools in the project oblasts. The government made significant contributions to the achievement of the component results, covering all taxes and delivery costs. Project monitoring and an independent evaluation confirmed that materials were received, highly appreciated and effectively used by teachers, students and other stakeholders. The independent evaluation also found that 93.8 percent of general education teachers in the project schools used the teaching-learning materials effectively, higher than the project’s 50 percent intermediate outcome target. 22 Component 2: Project intermediate Baseline Indicator after Target Achievements at project closure outcome indicators revision, if applicable Strategy for School- No SBTT N.a. SBTT strategy MoPE has indicated that elements Based Teacher Strategy adopted by of SBTT strategy will be used Training (SBTT) MoPE Number of teachers at 0 N.a. 50,000 59,147 pre-schools and schools trained Percent of teachers at 0% N.a. 60% 72% sampled project schools and preschools who are changing their teaching practice towards a more student centered approach 2. Component Two partly achieved or exceeded (with some qualifications) its intermediate outcome targets. • The project exceeded its intermediate outcome target of training 50,000 pre-school and school teachers, and reported training 59,147 teachers. The project financed the training of 4,139 general secondary teachers and 534 kindergarten teachers directly; these in turn trained 52,264 and 2,210 teachers, respectively, while receiving mentoring support from master trainers. Further, the project trained 1,506 general secondary school directors and 1,023 deputy directors, as well as 376 kindergarten directors. • The project exceeded its intermediate outcome target of teachers ‘changing their teaching practice towards a more student-centered approach’ as measured by an independent evaluation. An independent evaluation found that 76.2 percent of general education teachers who were directly trained under the project, and 84.8 percent of pre-school teachers directly trained under the project, had adopted student-centered approaches; the target in the PAD was 60 percent. The evaluation also found that the quality of the teacher training was high. • The project partially met its intermediate outcome target of having a SBTT strategy adopted by the MoPE. While the Minister has verbally indicated that the MoPE is favorable to adopting elements of the SBTT as developed by the project, there is no detailed and budgeted plan for training teachers using the SBTT. During the ICR mission, the Deputy Minister indicated that adopting elements of the SBTT might entail using it during the traditional August district-level methodological meetings at which teachers participate. 3. The component was proactively managed by the project implementation team and partners in the MoPE and in-service teacher training institutes. The training materials produced by the project were formally approved for use by the relevant academic and training institutions, widely lauded for their quality and usefulness, and broadly disseminated beyond the confines of the project. They are reportedly having an impact on training programs in the central and oblast-level in-service teacher training institutes. The course provided under the project was redesigned during implementation to be formally recognized as satisfying official in-service training requirements, such that in the second and third rounds of training, some participating teachers were able to receive their in-service training 23 certificate and satisfy the formal three-yearly requirements for professional development. In all, 1,149 of the 4,139 directly trained teachers (general secondary) received an official certificate from the oblast in-service teacher training institute. The SBTT also has four distinctive features that were appreciated by participants. First, the training delivery was broken up into different weeks, between which teachers had an opportunity to apply what they had learned; and during which they had an opportunity to discuss their classroom experience. Second, trainers came to the school to observe and mentor the teachers. Third, a substantial part of the course was devoted to building the teacher’s capacity to train their colleagues in child-centered teaching-learning techniques. Fourth, the training took place at the district level, instead of at the institute in the Oblast capital. 4. The counterpart financing contribution to this component was substantial. Teacher training institutions covered most training fees, as well as the accommodation and meals for teachers trained at oblast-level institutions; while schools covered expenses associated with sending their teachers for training. Indeed, the range of expenses covered by government was larger than initially planned, and this contributed to the relatively low level of disbursement (roughly 20 percent) of IDA funds against the component’s IDA budget. 5. This component contributed to the achievement of the PDO as measured by the first indicator (pertaining to learning achievements in Grade 4); indeed its contribution is likely understated by the values measured at the end of the project. By the time the final round of Grade 4 learning achievements was conducted, only the first round of teacher training had been fully completed (covering approximately 21,000 teachers directly and indirectly trained). Had the training been completed as originally scheduled, its impact on learning achievements might have been more marked. Component 3: Project intermediate Baseline Indicator after Target Achievements at project closure outcome indicators revision, if applicable Percent of Schools 0% 30% 33% (or 500 School Boards) Boards at project schools that have implemented a school improvement proposal 6. Component 3 exceeded its intermediate outcome target, with 500 project schools implementing a school development proposal, that is, 33 percent of project schools versus the 30 percent target. As per project design, all 1,501 School Boards elaborated a proposal, and an independent commission awarded funds to the 500 winning proposals. Not all funds were disbursed, however, the result of initial difficulties in reconciling requirements of the project and of the treasury system. Further, government counterparts and the project team identified implementation constraints pertaining to the limited disbursement period, maximum expenditure levels for hardware, and School Board capacities. The project team worked proactively with the government to resolve these issues, though this took some time. Measures taken included adoption of more flexible treasury rules, raising of hardware expenditures ceiling, and capacity building. As a result, while in the first and second rounds, only 42 percent of schools secured the second tranche (70 percent of funds were provided in 24 the first tranche and the remaining 30% in the second tranche), in the third round 99.4 percent of schools did so. 7. This component contributed to the PDO as measured by the first PDO indicator, in that some of the learning materials acquired were relevant to the primary curriculum. Delivery to schools started in the 2nd year of the project, in time to have an effect on learning. More broadly, the community and School Board participation that was stimulated by the component (and focused as it was on learning materials and learning) may have contributed to creating or sustaining a learning culture in the school. Component 4: Project intermediate Baseline Indicator after Target Achievements at project closure outcome indicators revision, if applicable Percent of schools at 60% Percent of schools 30% 0% three additional oblasts participating in participating in per per capita capita financing that financing that are are getting more than getting more than 10 per cent above or 10 per cent above below average or below average spending per student spending per student 8. Component 4 exceeded its intermediate outcome indicator target. The project had originally intended to support the scaling up of PCF to six oblasts (from three covered under BEP-1). This support entailed financing training on school budget planning, execution and accounting; establishment of a school financial management and public expenditure monitoring system, including software development and installation of a school financial management and accounting system on a pilot basis in the six oblasts; and monitoring and evaluation activities. 9. However, in 2010 (the year the project became effective), the government scaled up PCF nationally and directly financed most the activities that had originally been slated for project financing. Therefore after the mid-term review, the PDO and intermediate outcome indicator were revised to cover the whole country rather than the six oblasts. The Restructuring Paper noted that the component would finance a rigorous evaluation of the PCF across the entire country instead of in six oblasts, as originally envisaged. The Ministry of Finance did conduct training and established a school financial management and expenditure monitoring system, based on software it developed and made available to all schools. The project did support the development and disseminating of a report analyzing on PCF implementation, though a rigorous evaluation was not undertaken. 10. By the end of the project, according to Ministry of Finance data, the percentage of schools participating in PCF getting more than 10 percent above or below average spending per student reached 0 percent, better than the RF target of 30 percent. What this basically means is that all schools were receiving allocations on the basis of the norms established under the PCF, thus promoting greater equity and transparency in resource allocation. The Ministry of Finance data also showed a more efficient use of resources. For instance, the national average teacher to student ratio 25 increased from 11.3 in 2009 to 12.1 in 2013, while during the same period the average class size increased from 25.1 to 25.5. 2 Interviews with school management and board members during implementation support visits and ICR field trips revealed widespread satisfaction with the transparency, autonomy and increase in funding that have all been associated with the introduction of PCF. Component 5: Project intermediate Baseline Indicator after Target Achievements at project outcome indicators revision, if closure applicable Learning assessment Baseline Learning Original: Grade 4 Grade 4 assessment results for 4th and 8th surveys assessment results assessments in years 2 and carried out in year 4. graders from national for grades for 4th graders 4; Grade 8 assessment in Results analyzed and assessments 4 and 8 from national year 3 disseminated. carried out sample-based Revised: Grade 4 under assessments assessment in year 4 BEP-1 Report prepared and Baseline As per indicator Follow-up baseline disseminated survey survey carried out. comparing results in carried out Report prepared and baseline survey of under disseminated. schools and BEP-1. households with results from follow-up survey Impact evaluation N.a. As per indicator Impact evaluation prepared and prepared and disseminated disseminated. It is a compilation of the results of three surveys: student assessments, classroom teaching observation, and school grants implementation. 11. Component 5 met its revised intermediate outcome indicator targets. The project had originally aimed to finance national assessments for Grades 4 (years 2 and 3) and 8 (year 4). After the mid- term review, the Grade 8 exercise was dropped and the target was revised to cover one Grade 4 assessment only. Having missed the Grade 4 exercise in year 2 as a result of implementation delays, the project was restructured to remove it from the RF . Therefore, while the project reached its revised outcome target, it missed its original target. This may have been due in part to design shortcomings, whereby learning assessments were allocated relatively few funds and relegated to being an activity in a component that included project and results management. Indeed, as noted in ISR 5 (November 2011, or roughly 1.5 years into implementation), ‘The Bank team has noted that carrying out standardized assessments and the design of National Assessment System is an enormous undertaking, and the time and effort required to set up such a system may have been 2 These are unweighted national averages using oblast-level data. 26 underestimated. It is essential to plan this carefully in order to ensure a sustainable and appropriate design for Uzbekistan.’3 Further, the project does not seem to have made significant progress in developing ‘institutional capacity to assess student learning’ (as per the PDO), particularly from the point of view of sustainability. As noted in ISR 7 (June 2012, or roughly two years into implementation), the ‘work (on learning assessments) is done mostly by consultants, with limited involvement of regular MoPE staff, which raises a question about the sustainability of this exercise after project closure.’ (p. 5). In ISR 8 (December 2012), it is noted that ‘it is clear that there is no institutional niche for this exercise in MoPE and this lessens the sustainability after project closure.’ (p. 6). One must nonetheless acknowledge that the assessment carried out was done under the authority of the MoPE and with the direct involvement of the specialists of MoPE and the Republican Education Centre. 12. The intermediate outcome indicator targets pertaining to the survey of schools and households and to the impact evaluation were also met. 13. All the indicators and their targets for this component include an element of dissemination. In the event, workshops about the learning assessments were held for teachers and specialists of district and regional departments, as well as teachers of the regional teacher training institutes and members of trustee boards in 6 regions for the whole country. Regional conferences for dissemination of the outcomes of component 2 implementation were held for teacher training institute specialists, project trainers, and teachers of the pilot schools and preschools. All evaluation and assessment results were also discussed at the project’s closing workshop. 14. This component contributed significantly to the PDO as measured by the second PDO indicator (revised), in that it carried out and analyzed a learning achievements exercise for Grade 4. 3 P. 2 before the ellipsis; p. 8 after. 27 Annex 3. Economic and Financial Analysis 15. The economic analysis at the time of project design did not include an Economic Rate of Return (ERR) or Net Present Value (NPV) estimates; as such analyses would not be an appropriate tool for a project focused on pre-school and primary education. An ERR or NPV would have required making many arbitrary assumptions, undermining the accuracy of its conclusion. The economic analysis here therefore includes reference to the effectiveness of key interventions as well as the efficiency gains in terms of the student teacher ratio and average class size, which implies savings in public funding. Rational for public investment 16. In the absence of NPV or ERR estimates, the ICR analyses the project’s efficiency using any other appropriate cost-effectiveness criteria to determine whether the project represented the expected least-cost solution to attain identified and measurable benefits by either an analysis of cost per unit of input or cost per unit of output. 17. Since the early 1990s, Uzbekistan has consistently allocated 8-9 percent of its Gross Domestic Product (GDP) to education, 4 which is one of the highest in the world, and much higher than neighboring countries (5.2 percent for all income levels in Europe and Central Asia (2010) or 4.4 percent for developing countries only in ECA (2009) 5). The composition of education spending has changed over time, however. Until 2004, considerable budget growth was concentrated in specialized secondary education, underfinancing other sub-sectors. Preschool education suffered from severe budget constraints after independence because there was no alternative funding once enterprises and communities that had financed the subsector during the pre-independent period could no longer afford it. 6 Not only have preschool and basic education received relatively low allocation of the education budget, it has also been predominantly spent on wages, leaving less than five percent of the total recurrent budget for non-wage items to improve teaching and learning conditions. 18. The role of the government also includes equitable distribution of public resources. The specific attributes of basic education in Uzbekistan are such that the inequities do not lie predominantly in enrollment or attendance rates (which are near universal). Instead, they manifest themselves in the school infrastructure as well as availability of learning and teaching inputs. For example, 72 percent of urban schools had sufficient number of classrooms and desks, whereas this rate was only 66 percent for rural schools. 7 A study on school financing also observed that per student spending had 4 Uzbekistan State Budget 2000-2011; and World Development Indicators. 5 EdStats. 6 In 2007, 70 percent of 5-6 year-olds in the country, or 87 percent in villages, were not enrolled in preschool. World Bank, 2007, Republic of Uzbekistan: Living Standards Assessment Update, p. 73. 7 Uzbekistan Regional Panel Survey 2004, presented in PAD. 28 been allocated inequitably between oblasts, rayons and schools. 8 As a result of inequitable distribution of resources and learning conditions, the high reliance on informal private spending on uniforms, textbooks, learning materials and informal contributions to schools has limited access to good quality education, especially among the poor in rural areas. 9 Therefore, at the outset of the project, the PAD highlighted system-wide poor quality school infrastructure along with disparity in education inputs such as the number of classrooms and availability of student desks by urban and rural location as the economic rationale underlying this investment. Then, the project successfully provided learning materials to 1,501 schools and ICT equipment to 694 schools in rural areas to narrow the gap. 19. Addressing these types of inequities is a worthy use of public funds since school resources have an impact on learning outcomes for basic education students. Equity in education has strong merits from various perspectives, including human rights and social cohesion. From an economics point of view, one of the strong arguments takes into consideration that inequitable access to, and delivery of, education leads to an underutilization of talent. As a result, portions of society are not able to develop their skills and abilities to the best of their ability, which can limit their contributions to economic growth and society (OECD, August 2003). Bank value added 20. To redirect the government budget more towards preschool and basic education, the government launched the off-budget School Education Development Program (SEDP) 2004-2009, adding about 30 percent of the consolidated government spending on education, primarily to invest in infrastructure and teacher salaries annually between 2005 and 2009. 10 Both the BEP1 and BEP2 were designed to complement the government’s effort in addressing inequity in learning environment of preschools and basic schools across countries. This complementarity was strategic and cost-effective. The largest amount of the project cost—US$ 29.9 million (of which IDA US$ 22.8)—was spent on the provision of learning materials and resources, and the project was designed to ensure that these inputs be used effectively by complementing with school-based teacher and director training and the activation of school boards. It is essential that the project is designed to involve teachers and school boards in the selection of learning materials because they need to align with the needs, interests and abilities of the learners, about whom teachers and parents know the best. 21. Learning outcomes are influenced by many factors at the student and school level, primarily interactions with people (teachers and peers) and instructional materials (textbooks, workbooks, instructional software, web-based content, homework, projects, quizzes, and tests). Instruction materials have a direct influence on student learning via their effects on teachers. Teachers vary 8 Avanesyan, Vahram, 2006, Per pupil funding formula for Uzbekistan general education, draft. 9 The average education-related expenditure per student in basic education in 2000 was more than the poverty level monthly expenditure on food (World Bank, 2007, Republic of Uzbekistan: Living Standards Assessment Update, p. 89). 10 Salaries for general education teachers annually went up by 40 to 50 percent between 2005 and 2009 before slowing down, yet substantial, to 32 % (2010), 27 % (2011) and 15 % (2012) annual increases. 29 considerably in the way they use textbooks, teacher’s guides, and assessment materials, but in general, teachers are much more likely to cover topics presented in the selected materials. Instructional materials exercise their influence on learning directly as well as by influencing teachers’ instructional choices and behavior. 11 For instance, the relationship between learning outcomes and related factors can be examined on a comparable basis across countries using Program for International Student Assessment (PISA) 2009 data. PISA tests 15 year olds in mathematics, reading, and science. Although Uzbekistan did not participate in PISA 2009, other countries in the region were included in the study, such as Azerbaijan, Kazakhstan, Kyrgyz Republic, and the Russian Federation. The PISA results demonstrate that it is crucial for schools to have a minimum level of resources in order to achieve results, though beyond that threshold, additional resources may no longer affect test scores. 12 22. Without the project investment in learning materials and associated activities, the government’s investment in infrastructure and teacher salaries alone would have resulted in less effective to impact on student learning. Uzbekistan has about 394,000 general education teachers and 33,000 preschool teachers, of which 59,147 benefitted from training under the project. Teacher performance is a strong catalyst for student learning, which is one of the most important metrics in looking at the efficiency of an education system. According to an observation of teacher training and teaching practices in classroom, by improving skills and competence in using newly acquired educational materials and training aids, teachers in project pilot schools were observed to have adopted active teaching methods, student-centered and participatory learning, and classroom assessments, compared to those in control schools(see Table 1). 13 Results from classroom observation indicate that teacher training had a large impact on improving teaching and learning techniques applied in the classroom which is anticipated to positively impact student learning in the future. The classroom observation also suggests, however, that the investment could have been more cost effective if there were more training and support to teachers in using complicated materials (which were less popular among teachers due to difficulties in using them) and also to school directors in monitoring and evaluation of teaching practices by teachers. 11 Chingos, M.M. and Whitehurst, G.J., 2014, Choosing Blindly: Instructional Materials, Teacher Effectiveness, and the Common Core, Brown Center on Education Policy at Brookings, April 2012. 12 World Bank, 2014, Project Appraisal Document for the Uzbekistan: Improving General Secondary and Pre-school Education Project. 13 The Republic of Uzbekistan Ministry of Education, 2014, Final Report on Research: Observation of the teacher training and teaching practices in the classrooms (This report for internal use only and it is not provided for wide distribution and discussion.) 30 Table 1: Impact of training on teaching and learning techniques in the classroom Lesson plan Content and Assistance Utilizes ICT aligns with stated implementation of provided to target lessons match stated underperforming targets students Received training 71.5% 73.5% 83.4% 37.4% No training 44.4% 42.3% 65.5% 17.%5 Source: The Republic of Uzbekistan Ministry of Education, 2014, Final Report on Research: Observation of the teacher training and teaching practices in the classrooms (This report for internal use only and it is not provided for wide distribution and discussion). Public spending on education and project investment 23. The project provided substantial benefits to Uzbekistan’s education budget for general secondary schools. In 2009, the government allocated recurrent budget of US$ 1,310 million to basic education and US$ 248 million to preschool education, and capital budget (i.e., SEDP) of US$ 382 million to basic education. Of the recurrent budget, around 25 percent was allocated to personnel training, while most of the remaining was spent on wages. It is estimated only about 5 percent of non-training recurrent budget (approximately US$ 49 million for basic education and US$ 9 million to preschool) was allocated to non-wage items. In 2011, whereas the recurrent spending increased to US$ 1,834 million for basic education and US$ 334 million for preschool education, capital budget dropped to US$ 198 million, as expected partly due to the international economic crisis then. Assuming that 75 percent of the recurrent budget was allocated to non-training spending and 5 percent of which was allocated to non-wage items, about US$ 69 million for basic education and US$ 13 million for preschool, respectively were spent on non-wage recurrent items. Compared to the government’s own education budget on non-wage, non-training recurrent spending, the project’s contribution of US$ 29.9 million to enhance learning and teaching materials, US$ 1.4 million for strengthening school- based training, and strengthening school boards, and providing competitive grants for school improvement (US$ 2.7 million) were substantial. Due to the importance and effectiveness of these investment, the recently approved Global Partnership for Education Project: Improving Pre-primary and General Secondary Education (US$ 23.2 million out of US$ 49.9 million in total) will continue investing in training activities for teachers and school managers and increasing the availability of adequate equipment and teaching and learning materials for general secondary education schools offering grades 5-9 in rural areas. Per capita financing reform 24. In 2007, the government decided to adopt a new way of financing schools, and began implementing per capita financing (PCF) of schools to reduce inequities in the distribution of expenditures and increase the efficiency and equity of school financing. With support from the First Basic Education Project (BEP1), the government began implementing PCF in 2008 in Bukhara and Ferghana Oblasts and Tashkent City; and work was underway to extend per capita financing to three additional oblasts in 2009. As part of these reforms, the responsibility for budget allocations for basic education schools was transferred from Ministry of Public Education (MoPE) and its regional departments to the Ministry of Finance (MoF) and its regional departments. Basic education schools were declared legal entities and school directors became the primary administrators of budgetary funds and the Director’s fund for top-up of teacher salaries. 31 25. An evaluation of the PCF reform suggests that its implementation has resulted in “rationalization and optimization of budget expenditures (reduction of unnecessary staff positions, resupply of classes, economical use of funds for utilities, etc.),” 14 and its impact is sustainable and significant. The Ministry of Finance data for select Oblasts demonstrates that introduction of PCF improved financial allocation per student so that there was less than 10 percent variance between oblasts, which marks a huge improvement beyond the baseline of 60 percent at beginning of project. Although improvements were small in many cases, the student-teacher ratio and the average class size also improved in all of the 14 regions 15 in the country. From 2009 to 2012, the average teacher to student ratio increased from 11.3 to 12.1 (7.1 percent increase), and the average class size increase from 25.1 students to 25.5 (see Table 2). Of the total basic education budget of USD 1,834 million in 2011 (data for 2012 is not available), it can be estimated that USD 1,284 (70 percent) was spent on teacher salaries. Assuming that the STR was 12.1 in 2011, USD 91 million (7.1 percent of the teacher salaries) was saved due to the improvement in the STR, compared to the teacher salaries at the same STR as that in 2009. The saving in 2012 would have been more because the total budget was higher than in 2011, though the precise data is not available. USD 91 million saving is more than three times as much as the total project credit amount (USD 28 million). Table2: Average teacher to student ratio and average class sizes 2009 2012 Average Student Teacher Ratio 11.3 12.1 Average Class Size 25.1 25.5 Source: The Republic of Uzbekistan Ministry of Education, 2014, Final Report on Research: Observation of the teacher training and teaching practices in the classrooms (This report for internal use only and it is not provided for wide distribution and discussion). 26. The BEP2 was designed to support the government’s expansion of PCF to three additional oblasts, Namanghan, Syrdarya and Khorezm Oblasts. However, after the successful implementation of the PCF under the BEP1, the government decided to accelerate the reform and expand it nationwide using its own fund rather than slowly expanding under the BEP2. Because of this government’s decision to institutionalize the successful reform initiated under the previous project, the BEP2’s fund was no longer needed to carry out most of its planned activities related to PCF. As a result, a negligible amount of funds for this component was disbursed, but this result should be rather welcome and positively evaluated as the project’s objective to implement PCF was achieved in a bigger scale than the project originally intended. On the other hand, the government has not produced or disseminated an evaluation report on the outcomes of the PCF reform, which is one of the PDO indicators. 14 The Republic of Uzbekistan Ministry of Education, 2014, Final Report to assess the effectiveness of implementation of the order of budgeting and financing of schools expenditures based on the basic norms of costs per student in secondary schools of separate regions in 2010-2013 within the framework of the project. 15 Including the Republic of Karakalpakstan, 12 regions, and Tashkent city. 32 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Stefania Rodica Cnobloch Consultant MNSSP Team Member Gabriel C. Francis Program Assistant GEHNR Team Member Dilnara Isamiddinova Senior Operations Officer GPSOS Team Member Gulnora Kamilova Operations Analyst ECCUZ Team Member Sachiko Kataoka Senior Education Economist GEDDR Team Member Naushad A. Khan Consultant GGODR Team Member Maureen Anne McLaughlin Lead Education Specialist ECSH2 Team Leader Imelda Mueller Operations Analyst ECSH2 Team Member John Otieno Ogallo Sr Financial Management Specia OPSOR Team Member Juan Prawda Consultant GEDDR Team Member Ana Maria Sandi Consultant ECSH2 Team Member Supervision/ICR Iqboljon Ahadjonov Consultant GEDDR Team Member Galina Alagardova Financial Management Specialist GGODR Team Member Gabriel C. Francis Program Assistant GEHNR Team Member Dilnara Isamiddinova Senior Operations Officer GPSOS Team Member Gulnora Kamilova Operations Analyst ECCUZ Team Member Sachiko Kataoka Senior Education Economist GEDDR Team Member Imelda Mueller Operations Analyst ECSH2 Team Member John Otieno Ogallo Sr Financial Management Specia OPSOR Team Member Fasliddin Rakhimov Procurement Specialist GGODR Team Member Yuling Zhou Lead Procurement Specialist GGODR Team Member Elena Klementyeva Team Assistant ECCUZ Team Member Dingyong Hou Senior Education Specialist GEDDR ICR Team Leader Tatyana Shin ET Consultant GHNDR Team Member Gerard Peart Consultant/Main Author GEDDR ICR Main Author Sujani Eli Program Assistant GEDDR Team Member 33 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle US$ Thousands (including No. of staff weeks travel and consultant costs) Lending FY08 3.50 38.81 FY09 31.26 275.97 Total: 34.76 314.78 Supervision/ICR FY10 19.75 124.17 FY11 30.01 92.04 FY12 24.98 95.85 FY13 30.30 157.12 FY14 32.40 102.38 Total: 137.44 571.56 Grand Total: 172.22 886.34 34 Annex 5. Beneficiary Survey Results Not applicable. 35 Annex 6. Stakeholder Workshop Report and Results Not applicable. 36 Annex 7. Summary of Borrower's ICR 27. The Government’s report describes the project background and design, including its objectives and components; as well as changes to the project during implementation including revisions to Project Outcome Indicators and the reprogramming of funds under Component 1 following savings through procurement. 28. With respect to the achievement of Project Development Objectives, the Government noted: • all 1,501 project schools (100%) received all (100%) planned teaching-learning materials for Grades 5-9 based on requests from schools. • more than 62,000 teachers and school principals were trained, exceeding the target of 50,000 teachers. • nearly all teachers are using effectively the learning materials and resources required under the project, as observed by an independent evaluation; this greatly exceeded the target of 50%. Effective use is highest among teachers trained by the project. • 76% of trained teachers (vs. 62% of non-trained) use interactive teaching methods and critical and creative thinking development methods. For preschool teachers, these indicators are 85% among trained teachers and 66% among those not trained. These results exceed the 60% target. • School boards of 98% of schools who won competitive funding fulfilled all events specified in the school improvement plan. • Per capita financing was introduced nationwide, and deviations from budget estimates according to the formula were detected only in 8% of schools. • The Grade 4 mathematics score in project score increased from 486 in 2008 to 500 in 2013 (494 to 502 in non-project schools). In Reading, the scores were 482 in 2008 and 480 in 2013 in project schools (502 and 485, respectively, for non-project schools). 29. The results of the project were disseminated and discussed at a Republic Conference in March, 2014 attended by stakeholders, indicating the high level of public education sector accountability. 30. With respect to project implementation, the report notes that it gained from BEP-1 experience to conduct appropriate monitoring and evaluation. Under BEP-2, the project used international and national technical assistance to design, build capacities in, and execute a standardized learning assessment; and hired an independent firm to evaluate training effectiveness and the use of teaching- learning materials. The MoPE regularly reported on implementation and results, and plans to use survey results to improve teaching/learning process and quality of education. There was regular monitoring of schools in terms of their provision with learning materials for grades 5-9 and their use by teachers. 31. With respect to fiduciary requirements, the Bank’s initial assessment of financial management capacities concluded that financial management arrangements for BEP 2 were adequate in terms of staffing, budgeting and internal controls. At the time of initial assessment, however, the accounting system was not capable of supporting the preparation of regular and reliable financial statements. Following implementation of the action plan agreed during the assessment, the project accounting system was modified and was able to process all financial transactions under BEP 2 and develop interim financial reports. Further adequate mitigating measures were added, including making the project accounting system fully operational with built-in controls and updating the Operational 37 Manual (OM) to include detailed requirements to the management of competitive school grants (FM procedures, accounting, reporting, internal controls and audit requirements). 32. With respect to procurement, the Bank’s initial assessment concluded that the local procurement consultant and the international specialist hired under BEP 1 had gained sufficient working knowledge of World Bank procurement procedures and guidelines to carry out activities successfully, but MoPE had insufficient DSE staff to provide adequate support to the implementation of activities under BEP in a timely manner. During project implementation, MoPE had overall responsibility for procurement functions, with support of a local and international consultant and DSE participation. All procurement under the project were carried out in accordance with the World Bank's "Guidelines: Procurement under IBRD Loans and IDA Credits" published in May 2004 and revised in October 2006 (Procurement Guidelines); and the provisions stipulated in the Financing Agreement. 33. With respect to post-completion operations, the Government noted: • the SBTT strategy (Component 2) has been adopted to support teachers between regular professional development trainings. • a large number of school boards, using acquired knowledge and skills under Component 3, are able to find sponsors, develop new project proposals for teaching implement, and win grants for these purposes. • per capita financing has been introduced throughout the Republic. To facilitate its ongoing implementation, the Ministry of Finance developed an automated accounting system for budgetary units (UzASBO) designed for computerized accounting for public education, healthcare, higher education, and financial institutions. 34. The full Borrower ICR is on file as supporting document. 38 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable. 39 Annex 9. List of Supporting Documents  Project Appraisal Document, Basic Education Project, Phase II – June, 2009  Aide Memoires – 2010-2014  Final report from Basic Education Project team – March 2014 including o Final report with indicator matrix o Report on School Boards activities o Classroom observation report o Report on national students assessment o Report on implementation of per-capital financing (PCF)  Borrower’s Implementation Completion Review Report – May 6, 2014  Implementation Status and Results Reports – 2010-2014  Level 2 Restructuring of Financial Agreement, December 2012  Country Partnership Strategy – FY 2012-2015 40 41