ICRR 13904 Report Number : ICRR13904 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 10/19/2012 Country : Georgia Project ID : P098217 Appraisal Actual Project Name : Education System US$M ): Project Costs (US$M): 23.73 23.85 Realignment & Strengthening Program (apl #2) L/C Number : C4246 Loan /Credit (US$M): Loan/ US$M ): 15.00 15.84 Sector Board : Education US$M): Cofinancing (US$M ): 4.95 4.95 Cofinanciers : Japan Ministry of Board Approval Date : 11/15/2005 Finance- Population Closing Date : 12/31/2009 08/31/2011 and Human Resource Development Grant (PHRD) Sector (s): Central government administration (50%); Secondary education (20%); Primary education (20%); Tertiary education (10%) Theme (s): Education for all (50% - P); Law reform (25% - S); Education for the knowledge economy (25% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Susan Ann Caceres Judyth L. Twigg Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The loan was the second phase of a planned three -phase Adaptable Program Loan. The Program Objective was to "effectively realign the objectives of the education system and enhance the policy and the management capacity to improve efficiency, quality, and relevance of learning outcomes . The overarching objective was to better prepare students to meet the demands of a market economy and a democratic society " (Project Appraisal Document, p. 4). According to the Project Appraisal Document, the Development Objective of the project was " to increase the quality and efficiency of primary and secondary education in Georgia through the implementation of the institutional and policy framework developed under APL 1, as well as the reconstruction of schools in dire physical condition " (p. vi). The same objective was stated in the Financing Agreement (p. 4). b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project had three components : 1. Improving the Pedagogical Environment (appraisal, US$ 7.2 million; actual, US $5.23 million) was to support changes in curriculum content, teaching methodology, student assessment, initial teacher education, and continued teacher professional development, building on the institutional and policy frameworks supported by the previous loan. It contained two subcomponents : National Curriculum and Student Assessment System Development (appraisal, US$ 5.58 million; actual, US$ 4.13 million) was to improve teaching methodologies in line with the national curriculum framework in almost all grades and subjects, establish a system of student assessment for 4th grade, and assist Initial Teacher Education institutions to adapt their programs to changing standards, curriculum, and student assessment methodologies. Professional Development of Teachers (appraisal, US$ 1.62 million; actual, US$1.10 million) was to improve the qualifications of an unspecified number of teachers through the development and implementation of a teacher professional development system . 2. Improving the Physical Learning Environment (appraisal, US$ 13.95 million; actual, US$ 17.09 million) was to improve the physical learning environment in schools in emergency (delapidated) conditions to complement investments in improving the quality of the pedagogical learning environment and adoption of more efficient building standards and school consolidations . 3. Project Management, Monitoring and Evaluation (appraisal, US$ 2.58 million; actual, US$ 1.53 million) was to provide institutional support to the Ministry of Education and Science, the Municipal Development Fund (MDF), the National Curriculum and Assessment Center (NCAC), and the Teacher Professional Development Center (TPDC) related to project operations, audits, and project monitoring and evaluation . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost : There was significant exchange rate fluctuation between the local currency, the US dollar, and the Special Drawing Rights (SDR). As a result, the project gained approximately US$ 1.1 million, which was used to cover increases in construction costs . An undisbursed balance of US$ 9,458 was canceled after the end of the grace period for reconciliation of accounting (ICR, p. 12). Project costs were higher than anticipated for the second component, and lower than planned for the other components . After the August 2008 conflict between Georgia and Russia, there was more demand for construction and so costs increased for the civil works financed by the second component (ICR, p. 10). Financing : The project was financed by an International Development Association Credit in the amount of US$ 15.84 million equivalent and complemented by a Japanese Population and Human Resource Development (PHRD) grant in the amount of US$ 4.95 million (ICR, p. 29), which was used to support in -service teacher training, curriculum reforms, and textbook translation (ICR, p. viii). Borrower Contribution : Government funding was anticipated to be US$ 3.78 million, but was actually US$ 3.06 million (ICR, p. 29). Dates: Dates The project closed 20 months later than initially planned . The first extension was granted on November 17, 2009 to change the closing date from December 31, 2009 to December 31, 2010 to complete civil works underway and to support the Borrower with key education reform activities related to in -service teacher training, curriculum reforms, and textbook translations into minority languages . The second extension was granted on December 16, 2010 to revise the closing date from December 31, 2010 to August 31, 2011 to complete activities under the PHRD Grant. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Objectives : High Given the low funding that Georgia devoted to education in comparison to other similar countries, the education sector was faced with constraints at the time of project preparation : inadequate physical infrastructure with major deterioration in the physical conditions of many schools and the need to modernize the curricula and teaching methods, while at the same time trying to use resources more efficiently by increasing the student -teacher ratio, given the declining school age population in the country . Education was an essential pillar of the Georgian development strategy, Economic Development and Poverty Reduction Program (2006). The focus of education reform was to bring the Georgian education system in line with European standards by introducing a national system for assessment and examinations . The Bank's Country Partnership Strategy (2010-2013, current at project closure) focuses on improving efficiency of use of public resources and the quality of education . The project's objectives were highly relevant to the Country Partnership Strategy and to the goals of the Government's strategy, as the project aimed to establish an outcomes -based national curriculum framework, invest in the in -service training of teachers to upgrade their pedagogical skills, design and implement a per-student financing system, and begin plans for better school optimization, all of which would address the key challenges in the sector . The government's current Strategy for Development of Education (2010-2015) contains initiatives reflective of project objectives (e.g. improvement of quality), which also demonstrates their continued relevance. b. Relevance of Design: Design : Modest The activities outlined in the project's components could be expected to lead to the intended outcome of improved quality, since there was a logical causal chain between that objective and the activities intended to achieve it (PAD, pp. 31-33). However, there was a weak logical chain relating the efficiency objective to associated components/activities. As noted by the ICR (p. 8-9), the design did not address the underlying causes of the increasing student-teacher ratio or explore other solutions to increase school efficiency . As well, there were insufficient project activities (and inadequacies in the case of the per student funding formula ) to lead to increased system efficiency, as well as increased student -teacher ratio. 4. Achievement of Objectives (Efficacy): Increase the quality of primary and secondary education in Georgia : Substantial Outputs related to teacher professional development : Guidelines and standards were developed for the accreditation of teachers' continued professional development programs. 172 Continuing Professional Development Programs were accredited by the Ministry, and training was outsourced to private providers . The ICR (p. 16) notes that this program was discontinued in 2010, when the quality of some of the accredited programs was questioned and responsibility for training was recentralized at the TPDC. 25,000 primary and secondary teachers and 2,000 school principals were trained in the new national curriculum . According to the project team, when further revisions were made to the national curriculum, additional training was then provided to 8,000 teachers. Professional development was provided for National Curriculum and Assessment Center core staff in the areas of curriculum reform and national assessment . 10 videos on student-centered teaching were developed by the National Curriculum and Assessment Center, and monthly bulletins on this subject were provided to all schools . The ICR does not specify how many teachers were exposed to these videos . 67 books/literature to support professional development were translated from English to Georgian . Teacher Professional Standards were developed and finalized (Code of Ethics, Second Language Standard, Special Needs Teacher, Vocational Teacher, School Principal ). The ICR does not state whether and how these standards were implemented. Professional teacher resource materials were developed, such as a three -volume manual for teachers and videos on teaching methods . 2,163 teachers used government -provided vouchers to participate in a literacy training program that they selected from private providers, and 239 teachers used the voucher program to participate in a Georgian as a Second Language (GSL) program by the end of 2010. This voucher program was discontinued during the project, given the lack of monitoring of the private providers . By the end of the project, training was provided by the government's Teacher Professional Development Center . A Teacher Induction Pilot Program was established, providing support for 48 inductees and 28 mentors between 2008 and 2009. According to the project team, in 2010 this program was continued and expanded by the government through budget support . Outputs related to curriculum and teacher certification standards : The new national curriculum developed under the first phase of the project was piloted and then implemented in all schools across the country, except for grade six in Georgian schools and grades five and six in minority schools. New textbooks were published, and these new textbooks were translated into minority languages (e.g. Russian, Armenian, and Azeri). The ICR did not provide the number of textbooks developed and distributed . However, the project team provided a list of 52 titles covering grades one through twelve . 167 new textbooks were accredited by the National Curriculum and Assessment Center . 90% of children were taught the improved national curriculum, which exceeded the target of 70% of students. New teacher certification standards were developed, based on the new national curriculum framework . The ICR does not state whether or how these standards were implemented . A Teacher Certification Charter was adopted by Ministry decree No . 1101 in December 2009. A database of registration for teacher certification examination was established . Outputs related to civil works: 38 schools identified in emergency conditions were rehabilitated and refurnished and found to be of satisfactory quality. These schools supported 3,066 students (ICR, p. 36). Outputs related to improving education quality : National assessments in Georgian and Mathematics for grade 4 and 9 students were conducted . The assessment was based on a sample of students from schools across Georgia . Both assessment reports are publicly available on the National Curriculum and Assessment Center website, but the ICR did not provide this data. 43 teachers participated in the Teach for Georgia Pilot Program, which was designed to attract and retain teachers in remote areas. According to the project team, of the total number of participants, 41 remained for the first year of teaching, 32 continued for the second year, and 16 remained for the third year. Outcomes 71% of Georgian teachers were able to apply at least one active instructional methodology, which attained the target of 70% and was an increase from the baseline of 10%. (Source: Classroom Observation Research, ICR, p.14 and PAD, p. 34). The ICR (p. 16) also reported survey data showing more positive attitudes by more teachers towards more active teaching practices and student engagement in comparison to a previous survey conducted in 2007. 14.8% of teachers who took the teacher certification examination passed and were certified (or 6.4% of the total teaching force), which did not meet the target of 20% of teachers certified according to the new standards . The ICR (p. 18 and p. 55) provides two reasons why this target was not met . A relatively low number of teachers participated in the exam, since the new certification requirement will not become mandatory until 2014, and the implementation of the exam was late and only began in 2010. Of those that participated in the examinations, 85% did not pass both required tests . There was a seven-point increase in 4th grade Mathematics and Georgian scores between 2004 and 2010 (ICR, p. 22). The ICR does not present the exact data from these assessments, but it indicates (p 22) that there were comparison and attribution issues . The ICR (p. 31) reported that assessments of ninth grade students were conducted in 2009 and 2010 and that these results were publicly available on the NCAC's website . However, the results of these assessments are not provided in the ICR. The ICR does not discuss other factors outside the project that could also have contributed to improving education quality. Increase the efficiency of primary and secondary education in Georgia : Modest Outputs Revisions to the per student funding formula, which was designed during the first phase of the APL, were made to better account for the cost of operating small rural schools . 100% of schools are in charge of their maintenance budgets . Several manuals, Improving Schools' Physical Learning Environment , Human Resource Management Manual, and School Sanitation Maintenance , were published and distributed by the Ministry of Education and Science, and 500 school principals were trained on their content . Since the funding formula financed routine school maintenance, these documents supported new approaches to school maintenance planning and management to better sustain the physical environment . A teacher vacancy website was designed to post regular information on existing teacher vacancies across the country to support easy access to this information . Outcomes The number of schools carrying a deficit decreased from 1,400 schools in 2010 to 167 schools in 2011. This means that 93% of schools are managing their resources without any deficits . These remaining schools either have more than one school building under their jurisdictions or more than one language of instruction, which increases their operating costs (ICR, p. 19). While all schools had autonomy in the management of resources through the per student funding formula, this new mechanism did not become the sole source of public funding for schools, which did not attain the target of 100% of schools having autonomy in managing resources related to the teaching and learning process through the use of per -student financing as the sole source of public funding and accountable to the appropriate authorities in the use of those resources . The project aimed to increase efficiency by raising the student -teacher ratio. In fact, it decreased from 12:1 in 2006 to 10:1 in 2011, which did not meet the target of increasing to 16:1 and was a deterioration from baseline . The ICR (p. 19) indicates that the school consolidation undertaken by the government was not sufficient to increase the national average student -teacher ratio. The Government did not elaborate a strategy to increase the student -teacher ratio based on the specific situations of urban, rural, and mountain areas, which impacted the ability to increase the efficiency of primary and secondary education. As well, the revisions made to the funding formula during this project did not adequately account for schools having more than one school building under their jurisdictions or more than one language of instruction, which increases their operating costs (ICR, p. 19). Program Objective : Effectively realign the objectives of the education system and enhance the policy and the management capacity to improve efficiency, quality, and relevance of learning outcomes to prepare them to meet the demands of a market economy and a democratic society . By the end of the operation (the second phase of the overall program ), there were improvements in education quality and relevance of the learning with the development of a new national curriculum and training of 25,000 teacher and 2,000 school principals, and conducting of national assessments . Limited progress is evident in improving efficiency in the system. There is no indication that students are better prepared to meet the demands of a market economy or democratic society. 5. Efficiency: Efficiency : Substantial The PAD did not conduct an economic analysis, but justified the project interventions on the basis that poor school infrastructure negatively impacts learning and performance, since lack of heating constitutes the main reason for long-term absenteeism in Georgia (PAD, p. 65). The PAD (p. 66) reported that 9.03% of all school buildings in Georgia were in classified as in emergency conditions . The PAD (p. 68) noted that the educational investments of this project were justified, since they would mainly benefited children from poor households . The ICR did not conduct an economic analysis, nor did it provide unit cost comparisons . However, information contained in the ICR and in the Government ’s reply indicated substantial efficiency . First, the cost of project management was 41% lower than the appraised estimate (ICR, p. 29). The ICR states that implementation went smoothly and that fiduciary obligations were completed satisfactorily, suggesting that the Implementing Agency was able to use fewer resources than planned to manage and monitor the project . Second, 38 schools in emergency condition were rehabilitated and refurnished and seven schools fully reconstructed (ICR p. 18). This was accomplished despite the escalating cost of materials as a result of the increased demand for construction materials with the need for post-conflict reconstruction (ICR, p. 10). Third, the new schools could promote efficiency gains, since each school was consolidated with two or more other existing schools, since the Ministry provided a school bus to facilitate this process (ICR p. 18). The new schools offer space that could be used for extracurricular activities or vocational programs, if the Government encourages effective utilization of the buildings . Fourth, maintenance costs and implications for the recurrent budget of the education sector have been factored in the per student financing formula, so that schools have an adequate level of resources with the most recent revision of the per student financing formula (ICR p. 13). 93% of all schools were able to manage their resources without any deficit (ICR p. 22). ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The quality objective was substantially achieved, while the efficiency objective was modestly achieved . The project’s efficiency was rated substantial . There was high relevance of objectives, and project design was modestly relevant due to shortcomings in the results chain to increase system efficiency . The project contributed to the nationwide implementation of a new national curriculum, development of standards for teacher competencies and teacher certification mechanisms, and the construction /rehabilitation of schools in emergency conditions in remote and rural areas, replacing unsafe structures . However, it fell short of some targets related to teacher certification and impact on student-teacher ratio. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: While the government remains committed to the education sector and its reform agenda, the institutions supporting this modernization agenda are young and continue to need strengthening . There are concerns about the ability of these institutions to sustain the progress made so far . There is also concern that the institutional capacity developed during the project period could be lost with staffing changes . Another risk to the development outcome is the possibility that inadequate resources are devoted to the education budget at a time when more investment is needed, given the high percentage of school buildings that are in need of urgent repair (PAD, p. 66). There was progress with the revised per capita funding and manuals established to delineate maintenance responsibilities under the project, but 7% of schools still operate under a deficit because the per capita funding formula provides inadequate resources to these schools . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The Bank was able to foster collaboration with several development partners, which was leveraged to develop a comprehensive education strategy in Georgia, enabling it to qualify for the Education for All -Fast Track Initiative. The Bank team coordinated efforts to avoid overlap with other donors . Adequate risk mitigation measures were established at entry . The design was based on the lessons learned in Romania and Latvia that showed that it was possible to strengthen the institutional capacity of public agencies through intensive training and technical assistance . Preparation included analysis of educational and financial data from the Ministry of Education and Science and Georgia Statistics Bureau to substantiate the need for investment in school infrastructure. However, an important shortcoming was the lack of analysis of the rural situation, the political economy of undertaking reforms to increase the student -teacher ratio, and the reasons for increases in the teaching force (ICR, p. 8-9). As a result, policy options were not considered to reflect the special circumstances of small remote schools. . In addition, some of the project's indicators were not well defined and therefore difficult to measure. at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: Transitions between the three Bank Task Team Leaders were smooth . A cooperative working relationship was established with the Implementing Agency . Supervision was adequate and was enhanced by the staff located in country. Implementation of monitoring and evaluation was satisfactory . Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government exhibited commitment to the education sector and maintained steady interest in reform, as demonstrated by continuing budget allocations to continue modernization efforts . The Government's attention understandably was temporarily focused on the August 2008 conflict that left 160,000 people internally displaced. The Government did not elaborate a strategy to increase the student -teacher ratio based on the specific situations of urban, rural, and mountain areas . The government has taken over and provided budget support to continue the Teach for Georgia Pilot Program (described in Section 4). Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The National Curriculum and Assessment Center (NCAC) and Teacher Professional Development Center (TPDC) were charged with implementing the project, including performing fiduciary tasks, implementing the project’s components, monitoring key indicators, and creating bi -annual progress reports. At the close of the first phase of this program loan, the financial management staff of the Project Coordination Unit (PCU) were placed within the NCAC and TPDC. With qualified staff part of these implementing agencies, it was not necessary to have a separate project implementation unit . There was effective coordination with the Municipal Development Fund (MDF), which was particularly helpful for the civil works component of the project . Fiduciary reports were of satisfactory quality and were produced on time, and there were only minor delays in procurement due to the August 2008 conflict. Furniture and equipment for the refurbished /constructed schools were provided in a timely manner. With the need for rebuilding following the 2008 conflict, costs for civil works increased, resulting in delayed implementation due to the need to rebid contracts . Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: M&E design included a comprehensive set of indicators that reflected project objectives . The Project Appraisal Document provided a detailed plan for M&E, including data collection, reporting timeline, and responsibility for various tasks. However, some of the indicators lacked specificity and therefore were difficult to measure . b. M&E Implementation: NCAC had M&E responsibility. The subordinating implementing agencies (TPDC and MDF) provided progress reports on the status of project implementation and also reported on output and outcome data . The Education Management Information System (EMIS) developed under the first loan was used to provide data, particularly related to school consolidation and use of the per -student financing system. The ICR (p. 10-11) notes that the M&E responsibility was implemented smoothly, benefiting from the capacities gained during the first phase (APL 1), and that reports were satisfactory . Student assessment instruments were developed . The following studies were also conducted during the project : classroom observation research, and evaluation of teacher professional development programs such as the Teacher Induction Pilot Program, Teach for Georgia Pilot Program, and Teacher Voucher Program. c. M&E Utilization: While effective M&E structures were established and data were collected, there is no evidence that M&E data and analysis were used in any way to inform policy or the course of project implementation . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project was categorized as "B" under OP 4.01 Environmental Assessment because of the school construction . An Environmental Management Framework was reviewed by the Environmental Specialist of the Appraisal Team and was found to be adequate to address planned school construction (PAD, p. 80). The Framework was published on the Municipal Development Fund's website in English and Georgian (ICR, p. 11). A Municipal Development Fund enforced environmental compliance, which was noted to be satisfactory (ICR, p. 11). b. Fiduciary Compliance: Three agencies had responsibility for fiduciary compliance (Municipal Development Fund, National Curriculum and Assessment Center, and Teacher Professional Development Center ). These agencies were staffed with qualified personnel to carry out their core financial management responsibilities . Audit reports were submitted on time and unqualified. However, one report in 2009 found irregularities (e.g. payment to an NCAC deceased staff ), which were addressed promptly, and no other issues were encountered . Interim Financial Reports were produced by each agency and were generally on time and of satisfactory quality . Disbursements went smoothly, with only minor delays due to the August 2008 conflict. There were no significant challenges or delays in procurement procedures . c. Unintended Impacts (positive or negative): The public funding for the outsourcing of teacher professional development resulted in the discouragement of private providers and a decline in their supply, which contributed to the re -centralization of teacher training by the Ministry of Education and Science. This was not the intent of the project, but since privatization did not result in the consistent delivery of quality in-service training, the government decided that it would be better to centralize this responsibility within the Teacher Professional Development Center (TPDC). d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Moderately The Government did not elaborate a Satisfactory strategy to increase the student-teacher ratio based on the specific situations of urban, rural, and mountain areas. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: While the ICR (p. 27-28) reported several lessons, the overarching lesson that it derived was : Education reform in countries with declining student demographics need to understand and account for these changes while still attending to quality and efficiency in the system . When school consolidation is proposed, Bank staff need to have adequate understanding of local political issues in order to gain support from stakeholders, design reasonable solutions, and avoid damaging the interests of specific groups . Another important lesson drawn by IEG is that government hiring of project implementation staff can result in significant capacity strengthening for government ministries . In this case, the absorption of PCU staff into the Ministry of Education enabled the ministry to execute the project smoothly and oversee all of its responsibilities . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is concise and filled with quality analysis . It provides a candid assessment of the shortcomings in the Bank's analysis that impacted project design and implementation . Insufficient information is presented related to the relevance of the project. The ICR does not relate the project ’s objectives to the current Country Assistance Strategy (2010-2013) , and it does not discuss the relevance of the project ’s design. a.Quality of ICR Rating : Satisfactory