Document of The World Bank Report No: ICR1665 IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF-05804) ON A GRANT IN THE AMOUNT OF US$29.4 MILLION TO MONGOLIA FOR AN EDUCATION FOR ALL FAST TRACK INITIATIVE (EFA-FTI) CATALYTIC TRUST FUND PROJECT June 30, 2014 Human Development Department China and Mongolia Country Management Unit East Asia and Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective May 30, 2014) Currency Unit = Tugriks 1,000 MNT = US$0.67 US$1.00 = 1,502 MNT FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS ADB Asian Development Bank CF Catalytic Fund EDCM Educational Donors’ Consultative Mechanism EFA Education for All EGSPRS Economic Growth Support and Poverty Reduction Strategy ESMP Education Sector Master Plan FM Financial Management FTI Fast Track Initiative GA Grant Agreement GAP Government Action Plan GER Gross Enrollment Ratio GPE Global Partnership for Education ICR Implementation Completion and Results Report ICT Information Communication Technology IDA International Development Association ISR Implementation Status and Results Report M&E Monitoring and Evaluation MES Ministry of Education and Science MTR Mid-Term Review PAD Project Appraisal Document PD Project Document PDO Project Development Objective READ Rural Education and Development TVET Technical and Vocational Education and Training UNESCO United Nations Educational, Scientific and Cultural Organization UNICEF United Nations Children’s Fund Vice President: Axel van Trotsenburg Country Director: Klaus Rohland Sector Manager: Luis Benveniste Project Team Leader: Prateek Tandon ICR Team Leader: Prateek Tandon ii MONGOLIA Education for All Catalytic Trust Fund Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 4 3. Assessment of Outcomes ............................................................................................ 9 4. Assessment of Risk to Development Outcome......................................................... 14 5. Assessment of Bank and Borrower Performance ..................................................... 15 6. Lessons Learned ....................................................................................................... 17 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 18 Annex 1. Project Costs and Financing .......................................................................... 19 Annex 2. Outputs by Component ................................................................................. 20 Annex 3. Economic and Financial Analysis ................................................................. 22 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 26 Annex 5. Beneficiary Survey Results ........................................................................... 28 Annex 6. Stakeholder Workshop Report and Results................................................... 29 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 30 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 37 Annex 9. List of Supporting Documents ...................................................................... 38 MAP iii A. Basic Information MONGOLIA: EDUCATION FOR ALL FAST TRACK Country: Mongolia Project Name: INITIATIVE (EFA-FTI) CATALYTIC TRUST FUND Project ID: P108776 L/C/TF Number(s): TF-58043 ICR Date: 06/30/2014 ICR Type: Core ICR GOVERNMENT OF Lending Instrument: SIL Grantee: MONGOLIA Original Total USD 29.40M 1 Disbursed Amount: USD 29.40M Commitment: Revised Amount: USD 29.40M Environmental Category: C Implementing Agencies: Ministry of Education and Science Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 07/01/2005 Effectiveness: 03/16/2007 03/16/2007 12/23/2009 Appraisal: Restructuring(s): 12/20/2010 12/07/2012 Approval: 03/16/2007 Mid-term Review: 10/15/2008 09/11/2008 Closing: 12/31/2008 12/31/2013 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Grantee Performance: Moderately Satisfactory 1 The total grant amount originally approved was US$29.4 million consisting of three grants. However, the system only captures the initial grant of US$8.2 million and not the first additional grant of US$8.9 million, as well as the second additional grant of US$12.3 million. iv C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Satisfactory Implementing Quality of Supervision: Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments (if Indicators Rating Performance any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any time Quality of Supervision Yes None (Yes/No): (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Primary education 100 100 Theme Code (as % of total Bank financing) Education for all 100 100 E. Bank Staff Positions At ICR At Approval Vice President: Axel van Trotsenburg James W. Adams Country Director: Klaus Rohland Arshad Sayed Sector Manager: Luis Benveniste Christopher J. Thomas Project Team Leader: Prateek Tandon Cristobal Ridao-Cano ICR Team Leader: Prateek Tandon ICR Primary Author: Dean Nielsen F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The overall EFA FTI goal is to ensure universal completion of basic education and to close the gender gap and rural/urban gap in enrollment and school completion by 2015. To help Mongolia move towards v the goal of quality basic education for all, the FTI grant aims to: (1) improve access to and completion of basic education (primary and secondary education), with particular attention to rural areas; (2) improve the quality of basic education; (3) expand access to pre-school education in rural areas; and (4) improve project management. Revised Project Development Objectives (as approved by original approving authority) The PDO was not revised. (a) PDO Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 : Completion rate in secondary education Value quantitative or 97% 99% 99% Qualitative) Date achieved 09/01/2005 12/31/2013 04/23/2014 Comments The target was fully met. The indicator value increased steadily during the project's life. (incl. % (Source: EMIS) achievement) Indicator 2 : Completion rate in primary education Value quantitative or 98% 99% 100% Qualitative) Date achieved 09/01/2005 12/31/2013 04/23/2014 Comments The target was missed, but the final value reached 99% of the target value. (Source: (incl. % EMIS) achievement) Indicator 3 : System for learning assessment at the primary level (Core) Value quantitative or No No No Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments Target of implementing a system of learning assessment at the primary level was not (incl. % achieved by project closing as the standardized national assessment has not been achievement) institutionalized. (Source: PIU report) Indicator 4 : Utility of the learning assessment system (Core) Value quantitative or 0 1 0 Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments The standardized national assessment was not institutionalized nor utilized during the (incl. % project period. (Source: PIU report) achievement) Indicator 5 : Direct project beneficiaries (Core) Value 0 115,000 124,860 vi quantitative or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments Target was met and exceeded. The final value was about 108.6% of the target value. (incl. % (Source: PIU report) achievement) Indicator 6 : Female beneficiaries (Core) Value quantitative or 0% 50% 51% Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments Target was met and exceeded. The final value was approximately 102% of the target (incl. % value. (Source: PIU report) achievement) Indicator 7 : Completed baseline measures of Grade 5 achievement in rural schools Value quantitative or No Yes Yes Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments Target was met. Grade 5 assessment was conducted in 2008 as part of READ project. (incl. % The new Bank-supported education project will continue to work with MES on achievement) assessment with a planned Grade 5 assessment in 2016/17. (Source: PIU report) Indicator 8 : Gross enrollment ratio of pre-school education Value quantitative or 53.8 72.7 77.6 Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments Target was met and exceeded. The final value was approximately 106.7% of the target (incl. % value. (Source: EMIS) achievement) Indicator 9 : Gross enrollment ratio in secondary education Value quantitative or 94.2% 95.5% 98.0% Qualitative) Date achieved 09/01/2005 12/31/2013 04/23/2014 Comments The target was met and exceeded. The final value of the gross enrollment ratio of (incl. % secondary education was approximately 105.2% of the target value. (Source: EMIS) achievement) Indicator 10 : Gross enrollment ratio in primary education Value quantitative or 96.4% 96.2% 102.7% Qualitative) Date achieved 09/01/2005 12/31/2013 04/23/2014 Comments The target was met and exceeded. The final value of the gross enrollment ratio of (incl. % primary education was approximately 101.8% of the target value. (Source: EMIS) achievement) vii (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values documents) Target Years Indicator 1 : Number of dormitories rehabilitated Value (quantitative 0 22 22 or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments (incl. % The target of rehabilitating 22 dormitories was fully met. (Source: PIU report) achievement) Number of joint supervision missions conducted with donors and findings reported Indicator 2 : to EDCM Value (quantitative 0 5 5 or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments (incl. % The target number of joint supervision missions was fully met. (Source: PIU report) achievement) Establishment of EDCM to supervise and approve FTI technical proposal and Indicator 3 : procurement plan Value (quantitative No Yes Yes or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments Target met. EDCM was established and operated during the project period. (Source: (incl. % PIU report) achievement) Indicator 4 : Rehabilitation of kindergarten teachers’ college Value (quantitative 0 1 1 or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments The target for the rehabilitation of kindergarten teachers’ college was fully met. (incl. % (Source: PIU report) achievement) Percentage of ger-kindergartens supplied with furniture, equipment and learning Indicator 5 : materials to ger pre-schools Value (quantitative 0% 100% 100% or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments Target was 100% met. All of the newly deployed ger-kindergartens were supplied with (incl. % necessary furniture, equipment and learning materials. (Source: PIU report) viii achievement) Indicator 6 : Using Grade 5 assessment results to inform student assessment Value (quantitative No Yes No or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments Grade 5 assessment was conducted in 2008 as part of another Bank-administered (incl. % project in Mongolia, READ, and the results were used to inform student performance. achievement) (Source: PIU report) Indicator 7 : Provision of ICT mobile units to enrich teaching and learning experiences Value (quantitative 0 550 550 or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments (incl. % The target number for ICT mobile units was 100% met. (Source: PIU report) achievement) Indicator 8 : Provision of laboratories and learning centers to teachers’ college in Ulaanbaatar Value (quantitative 0 7 7 or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments The target number for laboratories and learning centers was fully met. (Source: PIU (incl. % report) achievement) Provision of in-service training in teaching methods in five subjects in secondary Indicator 9 : schools and to Grade 1 and dormitory teachers Value (quantitative 0 6,826 6,826 or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments (incl. % The target number for in-service training provision was fully met. (Source: PIU report) achievement) Number of additional qualified primary teachers resulting from project Indicator 10 : interventions. (Core) Value (quantitative 0 4,162 4,162 or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments The target number for additional qualified primary teachers was fully met. (Source: PIU (incl. % report) achievement) Indicator 11 : Number of new kits of furniture and equipment supplied Value (quantitative 0 375 375 or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 ix Comments The target number for new kits of furniture and equipment was fully met. (Source: PIU (incl. % report) achievement) Number of additional classrooms built or rehabilitated at the primary level Indicator 12 : resulting from project interventions (Core) Value Classrooms built: 30 Classrooms built: 30 (quantitative 0 Rehabilitated: 270 Rehabilitated: 270 or Qualitative) Total: 300 Total: 300 Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments The construction targets for classrooms to be built and rehabilitated were fully met. (incl. % (Source: PIU report) achievement) Indicator 13 : Deployment of mobile ger pre-schools Value (quantitative 0 171 171 or Qualitative) Date achieved 09/01/2005 12/31/2013 12/01/2013 Comments (incl. % The target number for the ger pre-schools was 100% met. (Source: PIU report) achievement) Indicator 14 : Number of new schools built Value (quantitative 0 3 3 or Qualitative) Date achieved 09/01/2005 12/31/2013 12/31/2013 Comments The construction target of school construction was fully met. The third school in (incl. % Baganuur district of Ulaanbaatar was completed by the closing date. (Source: PIU achievement) report) G. Ratings of Project Performance in ISRs Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 05/21/2009 Satisfactory Satisfactory 17.10 2 06/12/2009 Satisfactory Satisfactory 17.10 3 12/19/2009 Satisfactory Satisfactory 25.10 4 01/15/2010 Satisfactory Satisfactory 25.10 5 06/16/2010 Satisfactory Unsatisfactory 25.10 6 01/01/2011 Satisfactory Moderately Satisfactory 29.40 7 07/05/2011 Satisfactory Satisfactory 29.40 8 02/10/2012 Satisfactory Satisfactory 29.40 9 07/04/2012 Moderately Satisfactory Moderately Unsatisfactory 29.40 10 12/24/2012 Moderately Unsatisfactory Moderately Satisfactory 29.40 11 09/17/2013 Moderately Satisfactory Moderately Satisfactory 29.40 12 12/06/2013 Moderately Satisfactory Moderately Satisfactory 29.40 x H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Restructuring Disbursed at Reason for Restructuring & Key Approved PDO Date(s) Restructuring Changes Made Change DO IP in USD millions The project closing date was extended due to the following two reasons: (i) delay in disbursement of the First Tranche in2009 because of a change in requirement by the FTI Secretariat to align with 12/23/2009 S S 25.10 Bank lending processing procedure; and (ii) delay in procurement of civil works. The original closing date of December 31, 2008 was moved to December 31, 2010. The project was restructured to extend the closing date from December 31, 2010 until December 31, 2012 in order to complete the construction of the 12/03/2010 S U 29.40 Baganuur school complex. The reason for the construction delay was challenges with implementation such as delayed bidding process and short construction seasons in Mongolia. The project was restructured again to: (i) extend the project closing date by 12 months from December 31, 2012 to December 31; and (ii) adjust the results framework 12/07/2012 MS MU 29.40 indicators and targets to reflect the extension periods, facilitate measuring progress towards achieving the Project Development Objective (PDO) and track the required core indicators. xi I. Disbursement Profile xii 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Country Context. Mongolia has made remarkable economic progress since its transition in 1990 from a planned economy to a market-based one, while democratizing along the way. Between 2003 and 2008, Gross Domestic Product (GDP) grew at an annual rate of 8 percent in real terms, and this helped expand government revenues and reduce poverty. Nonetheless, nearly a third of the population remained poor, living on less than US$1 a day. 2. Sectoral Context. Contributors to such pockets of poverty have included significant urban-rural disparities in both educational access and quality. A 2006 household survey revealed a 7.3 percentage point completion rate difference between urban and rural students at the primary level and even higher ones at the lower and upper secondary levels (17.4 and 14.3, respectively). At the pre-primary level in 2007, a high proportion of the 57 percent of the relevant age group enrolled in early childhood education and development programs lived in urban areas, putting a damper on the readiness of 6-year-olds in rural areas for primary schooling. 2 Urban-rural differences in education quality were revealed in the “2005 National Assessment of Students’ Achievement” in which eighth grade students averaged a disappointing 50 percent correct on the mathematics test, with students in Ulaanbaatar scoring well above the mean and those from provincial centers and district schools well below. 3. Determinants of inequitable access and uneven quality included both demand-side and supply-side factors. Among the demand-side factors were poor parents’ inability to pay the direct and indirect cost of schooling – books, transportation, uniforms, and foregone labor in herding or other income-generating activities. The supply-side factors included inequitable supply of school infrastructure (including eco-friendly and low-cost energy and heating systems), boarding facilities for herders’ children, teaching and learning materials, equipment, school supplies, and teachers’ professional development. 4. Within this context, both Mongolia’s Government Action Plan (GAP) and its Economic Growth Support and Poverty Reduction Strategy (EGSPRS) prioritized strengthening basic education in the country. In addition, the Fast Track Initiative, established to assist countries meet their goals of attaining a perfect 100 percent basic education completion rate was seen as an extremely relevant partner. 5. These priorities were reflected in the nation’s Education Sector Master Plan 2006-2015 (ESMP) and its goals to: (i) to upgrade education quality at all levels of schooling and produce citizens who can function effectively in a modern market economy; (ii) to provide education services that can be accessed by children in all parts of the country, particularly rural areas, and by poor and vulnerable groups; and (iii) to improve the management capacity of central and local educational institutions at all levels. A major education reform in the early years of the ESMP (2008-2009) was the increase of the length of basic education from 11 to 12 years. This reform required more resources to finance the additional grade level and achievement targets, including 2 A change in policy in 2006 required 6 year-olds to enter grade 1 starting in 2008-09. 1 enhanced teacher training for the new grade, additional furniture and learning materials, expansion and rehabilitation of the infrastructure (including dormitories for herders’ children), and the development of appropriate curricula. In addition, modern facilities were needed, such as Information and Communication Technology (ICT), to keep teachers up-to-date. 1.2 Original Project Development Objectives (PDO) and Key Indicators 6. The original Project Grant Agreement of March 2007 was prepared according to the requirements of the Education For All-Fast Track Initiative (EFA-FTI), a multi-agency partnership, and as a Recipient-Executed Trust Fund. Given that this was a trust fund for which the Bank had fiduciary responsibility, the grants were supervised by the Bank, but the grant guidelines were formulated by the EFA-FTI governing board. These guidelines did not call for alignment with Bank operational procedures and guidelines, including the preparation of a Project Appraisal Document (PAD) with its project development objectives and key indicators. Instead, grant documents were simply to present the purposes and activities of the first year of a possible 3-year of EFA-FTI grant support with continued funding contingent upon successful project implementation and funds disbursement over the grant year. The purpose and activities of the Mongolian EFA-FTI grant for the first year (2007) were: (i) strengthening the capacity of primary school teachers; (ii) building mobile pre-schools; (iii) construction and rehabilitation of schools; (iv) improvement of rural school facilities; and e) grant management. In accordance with EFA-FTI guidelines at the time, the key outcome indicators were aligned with the goals and targets of the country’s ESMP, mainly covering basic education enrollment and transition measures. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 7. On July 31, 2009, shortly after the beginning of the third grant year, a Program Document (PD) – similar to a World Bank PAD – was prepared for the project. The PD retrofitted on the three annual Grant Agreements (for 2007, 2008 and 2009) an integrated/overall Project Development Objective (PDO) statement and key performance indicators, plus a results framework, and general agreements about project management, procurement, financial management, safeguards, and risks. 8. Project Development Objectives. The overall EFA-FTI goal was to ensure universal completion of basic education and to close the gender gap and rural/urban gap in enrollment and school completion by 2015. To help Mongolia move towards the goal of quality basic education for all, the EFA-FTI grant aimed to: (i) improve access to and completion of basic education (primary and secondary education), with particular attention to rural areas; (ii) improve the quality of basic education; (iii) expand access to pre-school education in rural areas; and (iv) improve project management. 9. Key Indicators. The key project outcome indicators specified in the 2009 PD were: (i) gross enrollment ratio in primary education, disaggregated by gender; (ii) gross enrollment ratio in secondary education, disaggregated by gender; (iii) net intake in grade 1; (iv) transition rate from primary to secondary; (v) completed baseline measures of grade 5 achievement in rural schools; and (vi) gross enrollment ratio in pre-school education. 2 10. In late 2012, a third project extension was processed and in it the key performance indicators (and their targets) were modified in order to “reflect the Project’s previous extension periods, facilitate measuring progress towards achieving the PDO, and track required core indicators.”3 The main changes from the list in the PD were: (i) the deletion of “disaggregated by gender” from the gross enrollment ratios; (ii) the dropping of “net intake in grade 1,” and “transition rate from primary to secondary,” both judged to be beyond the scope of project interventions; and (iii) the addition of core indicators required by the International Development Association (IDA), including completion rate from primary and secondary education and others designated by an asterisk. The revised list of PDO-level results indicators became: (i) gross enrollment ratio in primary education; (ii) gross enrollment ratio in secondary education; (iii) gross enrollment ratio in pre-school education; (iv) completed baseline measures of Grade 5 achievement in rural schools; (v) completion rates: (a) primary, (b) secondary; (vi) direct project beneficiaries (number), of which are female (%); and (vii) system for learning assessment at the primary level (rating scale) and utility of the learning assessment system. This list includes the required IDA core indicators. 1.4 Main Beneficiaries 11. The primary beneficiaries of this project were children and youth who were brought into the school system by the upgrading of classrooms, dormitories, and facilities, and the pre-school children in rural areas who have gained access to pre-primary education through the mobile ger 4 kindergartens, plus those already in the system whose learning was enhanced by improved learning environments. Also benefiting were primary, pre-primary, and secondary school teachers whose skills were upgraded through pre- and in-service training and access to ICT mobile training units, and their students. Furthermore, Teachers Colleges in UlaanBaatar (regular and kindergarten) and their staffs benefited from enhancements to their facilities and programs, as did the Ministry of Education and Science (MES) staff members involved in the project. 5 1.5 Original Components 12. A full description of Project components integrating those specified in the three grant agreements (2007-2009) appeared in the 2009 Program Document 6, as follows: 13. Component 1: Extending access (approximately US$6.2 million). This component includes three sub-components: (i) rehabilitate and improve service facilities in 30 rural schools 3 See World Bank, Mongolia: EFA-FTI Catalytic Fund Project, Restructuring Paper, December 6, 2012. 4 Ger refers to Mongolia’s traditional mobile housing. 5 The Mongolian government’s Ministry of Education has used many acronyms over the years, but the latest is MES, the Ministry of Education and Science, which is used throughout this document. 6 This ICR evaluates the project based on the four components established in the PD. The Grant Agreement (GA) was drafted at a time when there was no PD and was therefore structured along activities conducted under the first grant (US$8.2 million) rather than project components. In developing the PD, the activities described in the GA as well as activities conducted under the first and the second additional grants, were combined into the four components. This ICR thus refers to the number of components as four. 3 and dormitories; (ii) provide furniture and equipment to schools and dormitories; and (iii) build a new general secondary school complex. 7 14. Component 2: Improving teaching & learning environments (approximately US$9.4 million) by providing adequate and continuous in-service training. It has three sub-components: (i) teacher training – of primary school teachers in teaching methodologies in mathematics, language and ICT; of first grade teachers to cope with the new in-take of 6-year-olds, and of dormitory teachers to provide better care to boarding students; (ii) supplying mobile ICT stations and electronic books for teachers; and (iii) rehabilitating and equipping laboratories in teachers college in UlaanBaatar and in selected provinces (aimags). 15. Component 3: Extending access to pre-school education in rural areas (approximately US$13.2 million) by providing mobile ger pre-schools and rehabilitating the kindergarten teachers’ college in UlaanBaatar. This component includes three sub-components: (i) deployment of mobile ger pre-schools and supply of furniture, equipment and learning materials; (ii) rehabilitation of kindergarten teachers’ college; and (iii) provision of laboratories and learning centers to kindergarten teachers’ college. 16. Component 4: Project management (approximately US$0.6 million). This component includes operational costs of the Project Management Unit and technical audit of the grant. 1.6 Revised Components Not Applicable. 1.7 Other significant changes 17. The EFA-FTI Project was extended three times – in 2009, 2010, and 2012 – as described in Section 2.2 (Implementation). The extension to 2012 provided an opportunity for the Project’s results framework indicators and targets to be formally revised, as explained in paragraph 10 above. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 18. Mongolia launched 2006-2015 ESMP in 2006. It was subsequently appraised and endorsed by members of the Educational Donors’ Consultative Mechanism (EDCM), which declared it “a sound and credible pathway towards EFA goals” and recommended that Mongolia be granted full membership in the EFA-FTI Partnership, which it was on September 11, 2006. The EFA-FTI grant proposal was prepared in conjunction with the ESMP with the expectation that EFA-FTI catalytic funds could be used by the government to help it meet its sector plan goals. When EFA-FTI pronounced Mongolia eligible for a Catalytic Fund (CF) grant, the CF 7 In the 2012 restructuring, the target number of new secondary school complexes was changed to 3. 4 Committee set the funding level at US$28.5 million for 2007-2009, based on a funding gap analysis of the ESMP. 19. Soon after that the Mongolian Government established an EFA-FTI working group, consisting of the MES, the Ministry of Finance, and the EDCM, to steer the CF planning process. The initial step of that process was the MES’s production of a technical proposal, which included an action plan, a monitoring framework, and a description of implementation arrangements. The action plan was to be focused on basic education, but contextualized as a sub-set of the sector’s Master Plan (MP). Consistent with EFA-FTI requirements, this meant using the MP’s global indictors (e.g., enrollment and completion rates) to assess project outcomes. Also, by EFA-FTI rules, the initial grant was for one year, extendable to three, but the drafting team wrote it so that it could be easily expanded to cover all three. 20. Initial action plan ideas were submitted for review during an EDCM technical meeting in October 2006, allowing donors not only to give substantive feedback but to identify whether and how the proposed EFA-FTI funding meshed with their own plans for financial support. Feedback from the EDCM led to revisions and improvements, setting a pattern of government-led cooperation among partner agencies, which in Mongolia at the time was unique to the education sector, and which set an example of good practice in aid effectiveness. Notable at the time was how the proposed project covered parts of the MP that were not well covered by Government or other donor funding, such as expansion and improvement of classroom and dormitory facilities in rural areas, teacher training for new first grade teachers, and provision of mobile pre-school facilities for children of herders. 8 21. Once the technical proposal was finalized, it was submitted to the EFA-FTI Secretariat for review and to the Bank, acting as the supervising entity of the CF trust fund, for further processing. For the Bank, this meant formalizing the proposal into a Grant Agreement which, besides covering goals and activities, specified procurement and financial management arrangements, and a project management structure, built upon an existing Project Management Unit that had been effective in managing ADB projects. This organizational structure and the clear action plan for year 1 were sufficiently well developed that the Project was able to move into high gear very quickly. 22. The project was also notable for its rapid preparation time (less than three months between Mongolia joining EFA-FTI and its submission of a revised technical grant proposal), its alignment with EFA-FTI and World Bank requirements, and its addressing uncovered parts of the ESMP, both in terms of substance and what donors were already covering. At entry, design 8 This gap filling feature is detailed in the 2009 Program Document as follows: “The assistance provided to Mongolia by the Catalytic Fund is especially important since it addresses a number of issues that donors are not currently engaging with. The ADB’s Third and Fourth Education Development Projects have focused efforts on curricular reform, educational planning and budget management, and the Technical and Vocational Education subsector (TVET). UNICEF is currently addressing early childhood education and development in the country. UNESCO is providing project support for non-formal education. The Government of Japan is providing support to protect cultural institutions and community development projects. The USA’s Millennium Challenge Corporation (MCC) is supporting the TVET subsector. While these donor efforts have been extremely beneficial for the sector, a large gap remains between aspiration and achievement for the basic education subsector.” 5 challenges were mainly those related to the terms of EFA-FTI grants at the time, for example, the need to plan and get financing for only one year at a time even though many of the gaps identified took longer than that to fill, and the need to use global outcome income indicators for assessing fulfillment of goals, even when Project interventions were of a more limited scope. 9 2.2 Implementation 23. Implementation moved ahead briskly in the first year (2007) led by the Deputy Project Implementation Coordinator and a small staff housed within the ADB Project Implementation Unit (PIU). Activities went according to an annual plan drawn up by the sub-unit based on the Grant Letter and cleared by the EDCM, which, by stakeholders’ agreement, also became the Project’s Steering Committee. In fact, progress was so rapid that by the middle of the year there was some concern about overly rapid disbursement of funds. This was rectified by reordering some of the planned activities across the different years of the 3-year project. Other challenges were difficulties in defining technical specifications for goods and services (something that the PIU was not so experienced at), the scarcity of good construction contractors, and slow processing of financial requests. 24. The mid-term review (MTR), which was held 18 months after the start-up of the Project, determined that training and construction goals were moving along as planned (often ahead of schedule), and confirmed that project goals and activities were appropriate. The weakness identified at this point was the lack of solid outcome indicators in the Monitoring and Evaluation (M&E) results framework for demonstrating whether Project goals were being achieved and for helping decision-makers to take appropriate corrective actions. However, this was a problem rooted on the EFA-FTI requirement that global indicators be used for tracking outcomes, as mentioned above, and which was a challenge for all EFA-FTI projects prepared at that time. 25. Near the end of the Project’s second year in 2009, EFA-FTI, the Bank and the government agreed that a PD should be developed to be retrofitted on the Project, formalizing the objectives, components, results framework and other technical features that had shaped up over the Project’s 2 years of implementation.10 This allowed project managers and supervisors to view the Project as an integrated whole with improved project documentation consistent with Bank requirements. 26. Time devoted to PD preparation put the Project behind schedule; that, plus slow progress in constructing a new school complex in Baganuur (a mining town not far from UlaanBaatar) and the need for more time for a few other Project deliveries called for extending the completion date of the Project by one year (to December 31, 2010). 11 12 In late 2010, continued delays with 9 As the EFA-FTI CF program matured over the next 3-4 years, these requirements were loosened. 10 This had become a requirement for new EFA-FTI projects by then. 11 Reasons for delays in the construction of the Baganuur School complex as stated in the Recipient’s ICR: “… (i) a booming construction market and rising construction costs; (ii) limited financial ability of the hired firm; (iii) limited ability of the Ministry to supervise; (iv) contractual issues between the Ministry and the contractor; and (v) extremely heavy summer rains in 2012.” 6 the Baganuur School required a time extension for another two years, during which time the “access to basic education” component of the Project was put into “at risk status.” However, the construction of the Baganuur School complex involved some innovative features such as the use of new technology and an eco-friendly design, which added a level of complexity for MES and the PIU to properly supervise this activity. This complexity coupled with the challenges already mentioned required one more extension in 2013 to complete the school and fully furnish it. This was successfully achieved by the end of the 2013 calendar year. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 27. M&E designs for CF grants were, as explained above, relatively simple: EFA-FTI expected grant proposals to be built upon education sector plans and for the M&E outcome indicators to be those pertaining to the plan in general. Thus, the Project put forward, as outcome indicators, national-level primary and secondary enrollment rates (gross enrollment ratios) and completion rates, plus net intake into grade 1 and transition rates from primary to secondary education. By the middle of the Project’s second year (during the MTR) it was noted that the performance indicators used under the project are highly aggregated and that it would be difficult to assess project attribution to the education system using these indicators. While this design feature limited the assessment of project attribution, it was consistent with the EFA-FTI focus on countries meeting the Millennium Development Goals (MDG) and the established MDG indicators for that purpose. The MTR also expressed concern about the quality of MES statistics and recommended that at least enrollment and completion rates be broken down by gender, urban-rural and aimag in order to show the narrowing of access gaps across gender and region (urban-rural). 28. The Bank and other partners (e.g., ADB) worked with the Ministry to improve its management information system, but some systemic features of the reporting system (such as the highly aggregated nature of data received from aimags) made it difficult to obtain disaggregated statistics. When the retrofitted Project design was formalized in the 2009 PD, the key performance indicators in the results framework did include enrollment breakdowns by gender but not by urban-rural, limiting the possibility of assessing objectively whether the urban-rural gaps has been narrowed. In the 2012 restructuring, the gender breakdown was also dropped due to the fact that MES does not currently collect this data. This was in part due to the fact that attribution would have remained difficult: even if there had been good breakdowns and results showing positive changes, it is not clear how much the changes could have been linked to the Project, since the scale of the interventions was not likely to have made much of an impact on global indicators. Again, this was a common problem with early EFA-FTI grants. 13 29. With respect to implementation and utilization, there is abundant evidence (e.g., from progress reports and aide memoires) that output data were both diligently collected to track 12 In this Project, as in the case with most (if not all) EFA-FTI projects, the three year CF grant period was found to provide too little time to complete the Project’s activities and to disburse almost US$30 million. Challenges in tendering the construction of a medium-size school alone took three years. 13 See also World Bank (2013), Implementation Completion Report, Kingdom of Cambodia: EFA-FTI Education Sector Support Scale-up Action Program. 7 implementation progress (by both the MES and the Bank) and effectively utilized to solve a variety of problems (output qualities and quantity, budgetary issues, logistics, etc.). For outcome indicators the global data were collected and reported. As stated, project attribution is difficult, which is consistent with EFA-FTI project indicators of this time period. However, the global outcomes showed progress confirming that EFA goals of universal participation in basic education were being reached. 2.4 Safeguard and Fiduciary Compliance 30. Safeguards. The project triggered the Environmental Assessment (OP/BP 4.01). Due to the fact that the EFA-FTI program at that time were not approved by the World Bank’s Board of Executive Directors and were not required to follow World Bank guidelines, an Environmental Assessment was not necessary and, hence, was not conducted at the time of preparation. The task team was aware of Mongolian laws governing school safety, standards, and environmental management and made sure that all civil works undertaken under the project complied with domestic laws. 31. The project financed remodeling, rehabilitation, and construction of buildings and dormitories, as well as the construction of one school complex. There was no land acquisition, involuntary resettlement or loss of livelihood as a result of the project’s civil works activities. 32. The 2009 retrofitted PD for the project included a social assessment, which noted the existence in Mongolia of a few ethnic minority groups in Mongolia, mainly living in rural areas, the largest of which is the Kazakh, whose student population is concentrated in the far Western aimag, Bayan-Ulgii. Overall, Kazakh schools have suffered from the same systematic deficiencies seen in other rural schools in Mongolia, i.e., lower overall quality, lack of learning materials and highly qualified teachers. Kazakhs have their own language and the government has taken actions to address this in an educational context by providing instruction and textbooks in Kazakh. The EFA-FTI Project was designed to include aimag Bayan-Ulgii as one of its intervention sites, providing teacher training and rehabilitation to schools serving Kazakh children. 33. In the course of retrofitting and producing the PD, the team classified the project as Category C for its modest environmental impact per the World Bank Operational Policy 4.01. An Environmental Assessment was undertaken prior to the construction of the new Baganuur school complex and did not raise any concerns. 34. Financial Management. Financial management in the Project was problematic at the beginning, with problems noted in comparing actual and budgeted expenses, establishing a proper chart of accounts (including reconciliations), managing contracts (including tracking output quality and timeliness), financial management reporting (substance and timeliness), documenting the receipt of goods, and tagging assets. By late 2009 many of these problems had been overcome, to the extent that FM received an ISR rating of moderately satisfactory. During the Project extensions (2010-2013) most of the weaknesses were overcome, and the FM rating was elevated to satisfactory. All financial audits of the project were unqualified (clean). 8 35. Procurement. In response to the Project’s ambitious agenda, the PIU worked hard to award many contracts within a short period of time. However, at the beginning, due to inexperience and unfamiliarity with Bank procurement procedures, it often encountered difficulties putting out adequate technical specification and following procurement guidelines, e.g., for tendering and shopping. In response, during the first year, the Bank provided procurement training and extra support (reviewing and clearing contracts before they were finalized), such that by early in the Project’s third year, things had improved considerably. By mid-2009, however, Project procurement ratings fell again, due to delays and difficulties in selecting a contractor for Baganuur School complex (requiring a second tendering). When by mid-2010 the contractor still had not been selected, procurement ratings fell to unsatisfactory, only to rise again to satisfactory once the contract got underway in early 2011. As mentioned before, the school’s innovative and eco-friendly design added a level of complexity to this activity, which was new for MES and the PIU and therefore contributed to the delays. However, MES and the PIU were ultimately successful in completing and fully furnishing the school by project closing. 2.5 Post-completion Operation/Next Phase 36. At the close of the Project, Mongolia had already graduated from EFA-FTI (renamed GPE) due to its being reclassified as a middle-income country. As a transition gesture GPE awarded Mongolia a “graduation grant” of US$10 million in 2012 to continue its work in Early Childhood Education, considered to be crucial at the time given an abrupt up-turn in the pre- school population in the country. The new GPE grant currently being implemented is a direct extension of EFA-FTI, covering 100 additional mobile ger-kindergartens and 17 fixed pre-school buildings (using the model pioneered in the EFA-FTI Project) in urban and peri-urban areas. In the pipeline is a new IDA-funded Project for US$27 million (Education Quality Reform Project) which will extend the EFA-FTI sub-objective of “improving the quality of basic education,” and will cover some aspects of student learning (baseline assessments) that the EFA-FTI project aspired to support but was unable to bring to fruition. Also, the new project emphasizes in- service teacher education for improving math and language instruction, an effort that was also emphasized, in a preliminary way, in the EFA-FTI project. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 37. Relevance of Objectives. The project development objectives were aligned with Mongolia’s sectoral needs when the project was developed. The objectives fit squarely into the government’s Economic Growth and Support and Poverty Reeducation Strategy (EGSPR), ESMP, and the Bank’s 2005-2008 Country Assistance Strategy (CAS). They shared the common objectives of establishing an accessible and high quality education system that would nourish the intellectual development of Mongolian citizens while fostering economic growth. The project objectives continue to fit into the 2013-2017 Country Partnership Strategy (CPS), which seeks to address vulnerabilities through improved access to services and better service delivery, which includes education, health, justice and infrastructure. 9 38. A strong feature of the Project objectives was their relevance to Mongolia’s Economic Growth and Support for Poverty Reduction Strategy (2003), which recognized “the challenges of increasing access to schooling, rural-urban and gender disparities, improving quality, and strengthening the overall sector framework.” Its relevance to the country’s Master Plan (MP) to Develop Education (2006-2015) was also solid, given it was developed to help deliver MP results, especially the parts that prioritized disparities in access to quality education and the creation of “environment and conditions” for improved primary and secondary education services. Within the EFA-FTI arena, the Project was well aligned with the partnership’s main goal (as of 2006) of helping partner countries reach the millennium development goals of universal primary school completion and reducing gender disparity, and also with the Partnership’s more recently articulated (2012-2015) goals of Access for All, Reaching Every Child, and Learning for All. 39. For the EFA-FTI catalytic fund grants, relevance to county education plans generally did not imply financing broad systemic change or big strides towards closing MDG gaps; instead the Fund encouraged flexible, high-impact, short-term goals that would advance important and often neglected parts of the country’s educational sector plan, positioned in a way to attract other (larger) sources of funds (foreign and domestic) for further development. The original goals of Mongolia’s EFA-FTI were of this gap-filling variety: renovating and provisioning schools in underserved areas, providing a limited boost to teacher skill levels; and creating high tech teaching stations and mobile kindergartens. Therefore, relevance of objectives is judged to be high. 40. Relevance of Design. Relevance of project design is judged to be substantial. This judgment is based on the fact that the project was in compliance with EFA-FTI CF guidelines at the time, the outcome indicators adopted by the project were those for the sector plan in general, including those that would show movement towards EFA goals of improved access to and completion of basic education and improved educational quality, all at the national level. This was appropriate for assessing progress on overall EFA-FTI goals, but early in the Project’s implementation it became apparent that these indicators were less appropriate for showing Project attribution. During Project implementation these became M&E issues, as they did for most EFA-FTI CF projects of this generation. The challenge became how to evaluate project accomplishments with these global indicators (inevitably showing weaknesses) while at the same time giving the projects credit for strong performance on their original short-term goals. 41. Relevance of Implementation. The relevance of implementation is rated substantial due to the fact that every year the Project was successful in implementing its action plan and in executing -- on time -- all but one of its planned contracts (more about that below). The goods and services it delivered to target areas were of the expected quality and reached expected beneficiaries. Although there were some validity issues with Project indictors, observers in the field (including supervision missions and study tours from neighboring countries) noted how the Project was able to deliver improved educational “environments and conditions” to rural and remote areas, despite Mongolia’s notoriously difficult geography. 10 3.2 Achievement of Project Development Objectives 42. Improve access to and completion of basic education. The results on revised outcome indicators for this PDO show a pattern of meeting or exceeding targets, as follows: • Gross enrollment ratio for primary: actual value achieved, 102.7 percent, exceeded the target level by 6.5 percentage points. • Completion rate for primary: actual value achieved, 100 percent, exceeded the target level by 1 percentage point. • Gross enrollment ratio for secondary: actual value achieved, 98 percent, exceeded the target by 2.5 percentage points. • Completion rate for secondary: actual value achieved, 99 percent, is the same as the revised target. 43. Moving enrollment and completion rates from the mid-90s to around 100 percent is a significant accomplishment, especially in a country like Mongolia which faces so many geographic and demographic challenges, and this can be used by the Government to indicate that it has reached or is nearing EFA goals of universal enrollment and completion. However, as mentioned in the M&E section above, these measures have limitations in their capacity to show Project effectiveness, based on their global scope, the quality of their data and attribution issues. 44. Another way to show Project effectiveness is to examine the intermediate outcome indicators, which were included in the original grant agreement (derived directly from the country’s Education Sector Master Plan) and rigorously tracked throughout the Project (see specific outputs in Annex 2). These indicators show the Project’s setting the stage for improved enrollments and completion by building 3 new exemplary schools, rehabilitating nearly 100 rural school buildings and dormitories, furnishing and supplying learning materials for all classrooms used for the new six-year-old first-grade cohort, and furnishing/equipping six schools for children with special needs. For all of these indicators, outputs met or exceeded Project goals. 45. Improve the quality of basic education. The indicator used to measure quality was completed baseline measures of Grade 5 achievement in rural schools. This indicator’s target was reached although the actual Grade 5 assessment was conducted under the IDA-supported READ project which was being implemented at the same time as the EFA-FTI Project. It should be noted that the Grade 5 assessment results were used to inform policy decisions and led to a change in the teacher training curriculum. Moreover, the Grade 5 assessment was made available on the READ website and the assessment design was used to develop 8 booklets with writing rubrics that have helped to promote better writing skills and are being used by teachers to assess student writing. The new Bank-supported education project will continue to work with MES on assessment with a planned Grade 5 assessment in 2016/17. The intermediate outcome indicators (see Annex 2) do reveal considerable Project success in “improving the teaching- learning environment,” specifically by extending training to nearly 9,000 teachers during 2008- 2010, covering improved math and ICT teaching skills and the teaching of six-year-olds (for the new cohort of first grade teachers). It also created mobile ICT stations and electronic books for teachers, and improved facilities and equipment at four teacher training colleges. All output targets related to quality were met. These outputs were considered by the education community to have positively influenced the quality of basic education, and observations during supervision 11 missions documented examples of improved learning environment, but the evidence of Project impact is less clear due to the lack of data on student learning outcomes and teacher performance. 46. Expanded access to pre-school education in rural areas. Gross enrollment rate for pre- primary is shown to increase from a baseline of 53.8 to 77.6 percent, a very significant improvement over the 8 year period (almost 3 percentage points per year), made even more impressive by the difficult access and logical conditions that exist in most of the country. The Government and many external agencies contributed to this change, but the Project included important interventions especially in rural areas that provided more spaces for herders’ children. One shortcoming is the lack of urban-rural breakdowns of the enrollment data to substantiate improved preschool enrollment in the target (rural) areas. 47. Overall rating for meeting the PDO is Moderately Satisfactory. 3.3 Efficiency 48. Project efficiency can be assessed in various ways. If one considers value for money, it appears that the Project did well, given the fact that it exceeded many of its production targets but kept to its original budget; also some cost savings at the end were used to finance additional outputs. Also, as shown in Annex 3, the Project was able, in many cases, to provide goods and services at significant lower unit costs than comparable programs resulting in many improvements in the cost-effectiveness of the system. This could be one explanation for how the Project was able to deliver all of expected physical outputs, even in the face of escalating materials and construction costs. Finally, one could consider administrative costs in the Project compared to expenditures for actual deliverables (goods, services, and civil works). On this the Project can be seen as very efficient, since administrative costs (funding for the Project Implementation Unit) were only 2 percent of the total. 49. If the consideration is the pace and smoothness of operations, the record is mixed. From the beginning, the Project moved at a brisk pace, sometimes disbursing funds almost too fast. 14 On the other hand, since many project activities were quite new to the Ministry (e.g., creating specifications for civil works, goods and equipment, and training services, and then shaping the procurement processes for them acceptable to the Bank), things often moved slowly and/or had to be done over. This was most evident in the construction of the Baganuur secondary school complex, which took four years longer than expected (requiring retendering and 3 project extensions) due to procurement issues, contract problems, spiraling construction costs, flow of funds issues, heavy rains and quality (inspection) problems. Combining these factors leads to an efficiency rating of substantial 14 For example, the second tranche of 2008 Project funding was on track to be disbursed in June of that year well ahead of this phase’s planned completion in December. 12 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 50. This rating is based on a combination of ratings in which relevance is rated “substantial,” efficacy “substantial,” and efficiency “substantial.” 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 51. The Project was fully aligned with the country’s poverty reduction strategy, which viewed the role of education institutions as closing existing access gaps between urbanites and the rural poor (most of whom are herders). Although no statistics on the increase enrollments among the children of herders were specifically reported on, it can be deduced that project activities did reach them since most school improvement efforts were carried out in rural areas. Concerning gender, the 2009 objective statement mentioned EFA-FTI’s goal to close the gender gap in basic education, but this never became a specific objective in the Mongolian EFA-FTI project mainly because the country is unique in having a “reverse gender gap” (enrollment rates for girls are higher than for boys, especially at the secondary school level). Therefore, no gender gap-specific activities were included in the project. The 2009 outcome indicators did ask for gender breakdowns for enrollment rates, but these were never reported, and in the 2012 restructuring gender breakdowns were no longer called for as explained above (see paragraph 24). Since enrollment and graduation rates were both nearing 100 percent by then, it can be assumed that gender parity was close to being reached. Finally, social development was addressed in many ways, among which were: (i) students enjoying more child-friendly teaching- learning and living (dormitory) environments, which did occur given the success of the classroom/dormitory rehabilitation efforts, and (ii) reaching rural students whose mother tongue is not Mongolian (e.g., Kazakh), who were reached by the Project through rehabilitation of schools and teacher training in the Kazakh cultural areas. (b) Institutional Change/Strengthening 52. The only planned institutional strengthening feature of the Project was that for strengthening “Grant Management,” which was relatively well implemented through training and mentoring. In addition, the Project strengthened the country’s primary schools by training all teachers assigned to the new 6-year-old first grade cohort and dormitory supervisors. This is an element alleged by a PIU deputy director to have had a lasting effect on that cohort, given that teachers move with their students. Moreover, some new building designs were pioneered (wood frame kindergartens and the energy efficient Baganuur School complex) that are expected to be models for schools to come. The Kindergarten Teachers College of UlaanBaatar was also strengthened by receiving computers and computer labs, musical instruments, physical upgrading, and materials, allowing it to strengthen its professional atmosphere and to attract better students to this growing education subsector. With the respect to offices in the MES, many mission reports describe the high levels of engagement of government offices, which were likely to have been strengthened by this experience and its high output levels. 13 53. For some time the Project did enhance the role of the country’s donor coordination mechanism (EDCM), which was established prior to the Project’s approval, but which was given a significant advisory role during the Project’s first three years. After 2009, in the extension years, during which supervision was mostly focused on the Baganuur School, the EDCM’s role and “raison d’être” diminished. However, there has been recent interest in reviving the mechanism since donor coordination is currently more ad hoc and often limited to just two or three agencies at once. (c) Other Unintended Outcomes and Impacts (positive or negative) 54. The Project, following the model set by the EFA partnership, relied on and strengthened the EDCM. Designated by the Project as its Steering Committee, and active in reviewing and endorsing the Project’s annual Technical Proposal and work plan, the mechanism played an important role in the Project success. Additionally, there is evidence of this enhanced donor cooperation during the life of the Project, for example: in the way the Project focused on building and renovating rural classrooms while JICA did the same in the urban areas, and in the way the ADB supported the training of first grade teachers for another two years, after EFA-FTI ran its three year course. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not Applicable. 4. Assessment of Risk to Development Outcome Rating: Moderate 55. In the year the Project was launched, 2007, Mongolia was hit hard by the global recession, so it welcomed EFA-FTI grant money as a way cover capital expenditures and to further its program development ideas, including model building and testing. This included scaling up ger kindergarten deployment, launching a new model for kindergarten buildings (Canadian, wood- frame), using new energy-efficient and cost-effective designs for basic school construction and renovation, and undertaking a massive effort to train all teachers of the new 6-year-old first grade cohort. According to Ministry officials the ger kindergarten scale up, which covered about 170 of a potential 1,400 herder areas during EFA-FTI and another 100 during the GPE follow-up (see section 2.5) has given the government confidence in this model. Given its stronger budgetary situation, the government is now able to allocate its own funds for further expansion (in 2014 1.4 billion MNT were allocated for the procurement of 180-200 new units). Similarly, the Canadian technology is being used in the construction of new pre-school buildings using state funds, inter alia, under the “New Soums” project, conceived to stem the flow of migrants to UlaanBaatar, including a school, kindergarten, and dormitories in each of three soums. The recently launched Baganuur School Complex, which pioneers eco-friendly power consumption and heating systems, will be observed over the next 1-2 years, and if considered successful, will be used as the model for new schools from 2015. 56. The goal of providing “adequate and continuous” training to first grade teachers would have required five annual training events, starting in 2008, since this was the first year for the age- 14 six first grade intake, and teachers move through the five grades with their students. The Project did provide training for all grade 1 teachers in 2008, 2009 and 2010. Training for the remaining two years was beyond the scope of the EFA-FTI project, but was covered by an ADB project, Education for the Poor – Financial Crisis Response Project, which allowed training to continue over 2011 and 2012. One Project manager indicated that this effort had enduring impact, since it helped first grade teachers create a welcoming and supportive atmosphere for 6 year olds, which gave parents confidence to enroll their youngsters. Continuous upgrading for other teachers has now become routinized through the work of the Mongolia’s new Teacher Training Institute, which was established in 2012. 57. Going forward, the MES will need to ensure that funds are made available for maintenance and replacement of project inputs since site visits have confirmed that some Project inputs are now showing signs of deterioration (e.g., dormitory equipment and upgrades). Even first grade classroom furniture and toys, provided 5-6 years ago but generally very robust, are starting to wear out or get lost (toys). 15 58. Concerning the Project aspiration to create an assessment system for tracking basic student learning outcomes, which never got off the ground during EFA-FTI, the recently approved IDA- financed Education Quality Reform Project will help build a national assessment system (baseline and periodic repeats) as one of its main components. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 59. CF grant, project preparation, and thus Bank professional engagement at entry, were not as extensive as with IDA projects. As per EFA-FTI requirements, the Bank did draw up the basic grant documents for the Project, and they were sufficient to launch the project and get it moving quickly. During the second grant year, when the need for a more strategic “program document” was decided upon covering all three grant years, the Bank effectively expanded documentation to include retrofitted overall objectives, outcome indicators, fully elaborated project components (covering all three years), a results framework, integrated procurement and financial management arrangements, and back-up analyses of social and economic issues. This allowed for more integrated planning of project activities and improvements in both procurement and financial management. One project shortcoming that was noted by Bank task team leaders in the first Project year was that the outcome indicators that had been drawn up according to EFA-FTI specifications were too global making project attribution difficult. Given that these types of indicators were preferred by EFA-FTI, the 2009 PD did not replace them with more project- specific indicators. 15 Based on ICR team observations during its March 3-12, 2014 field trip. 15 (b) Quality of Supervision Rating: Satisfactory 60. Bank supervision was attentive from the outset, constructively engaging both the government in general (and MES in particular) and the other members of the EDCM, and attending to important substantive and management issues with a good mix of experts in basic education, economics, procurement, architecture, and financial management. In the early years of the Project, when procurement and financial management were below Bank standards, the Bank team provided capacity building and mentoring, which resulted in substantial improvements. Supervision support reached its peak in 2009 when the Bank, in response to new EFA-FTI guidelines, took the lead in the creation of a PD (see Section 2.2). After its approval, Bank supervision (and its Aide Memoires) were followed by technically strong Implementation Status and Results (ISR) reports. In 2012, in conjunction with a Project extension, the Bank team successfully restructured the Project in order to reflect the Project’s previous extension periods, facilitate measuring progress towards achieving the PDO, and track required core indicators. When completion dates came due and had to be extended, the Bank was flexible and supportive and was able to help steer the Project through sometimes tense times (during which the Bank downgraded Implementation Progress to moderately unsatisfactory) to ultimate and satisfactory completion. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 61. The Bank performance at entry was consistent with expectations and country needs for a short-term year-to-year EFA-FTI Grant. Solid Bank performance also typified the supervision phase of the Project. The one shortfall noted was the missed opportunity during the new beginning in 2009 to update the Project outcome indicators and results framework so that Project attribution could be measured more systematically. 5.2 Borrower Performance (a) Government Performance Rating: Satisfactory 62. The government of Mongolia, represented by the Ministry of Education, was a strong partner in the EFA-FTI program, having prepared the ground for EFA-FTI CF support by presenting a credible poverty reduction strategy and educational sector strategic plan (2006-2015). The country’s proposal to EFA-FTI was clearly aligned with both its own poverty reduction and sector strategies and EFA-FTI’s goals and objectives. The Government was constructive in the way it dealt with the education donor community (turning its consultative mechanism – EDCM) into the Steering Committee for the Project and in putting out a mandate to the various branches of the MES that they cooperate with the EFA-FTI PIU in implementing the Project. Ministry ownership of the project is reflected in the fact that annually it incorporated EFA-FTI grant funds into its annual budget. Ownership was further evidenced by the fact that when it was honored by EFA-FTI as a best-performing project in 2009, the Minister himself went to Stockholm to accept the honor and describe the Project. 16 (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 63. Through a MOU between the MES, the ADB, and IBRD-IDA a project coordination unit was established within a pre-existing ADB PIU, headed by a Deputy Project Implementation Coordinator. This was a good experienced platform for moving ahead quickly, which the PIU did. The collaboration with relevant departments at MES was also relatively smooth. During the several years of project management the PIU distinguished itself by completing project activities on schedule and within budget. To be sure, in the early years the PIU was often stretched and deviations in procurement and financial management were noted, but these were mostly overcome. As mentioned previously, the procurement issues related to the Baganuur school construction, combined with the challenges of contract management and civil works supervision, as well as heavy rains did impact the timely completion of the school. However, the project was successful in completing and furnishing the school prior to project closing. The otherwise strong agency performance is thus attenuated to the moderately satisfactory level. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 64. The government performance rating of satisfactory is combined with the agency moderately satisfactory rating to form an overall rating of moderately satisfactory, since the agency short-comings occurred in a single area (managing the final school contract) within an otherwise satisfactory record. 6. Lessons Learned 65. Low income countries, that are experiencing severe financial constraints and are able to afford little more in education than the payment of salaries and other recurrent costs, can often benefit greatly from the infusion of short-term grant funds for financing capital expenditures and other development expenses, especially if that is used as a bridge to a time when they can begin to cover capital expenditures on their own. This may be particularly true for countries in transition like Mongolia. 66. Connecting the delivery of services to traditional ways of doing things (for example, putting kindergartens in movable buildings within nomad communities) can often make these services more attractive, affordable, and culturally acceptable. 67. The use of a consultative framework like the Education Donors’ Consultative Mechanism, which was effectively used as a steering group during the first three years of the Project, can be a good way to bring donors together and to promote cooperation and interagency planning. Even more effective could be a mechanism that includes and is led by the government (a “local education group” like those promoted by GPE) which puts the government in charge of the external assistance agenda. 68. In designing outcome indicators for an intervention or set of interventions like those financed by the EFA-FTI grant, it is critical that the indicators be linked causally to project inputs, processes and outputs so that there is no question in the end about attribution (whether the 17 project contributed to change in outcomes). The indicators also need to fit to the scale of the intervention: if the interventions are mainly local in scope, the appropriate outcome indicators would be to show local changes; if the changes are global in scope, then global indicators are called for. 69. Innovative, eco-friendly building designs are becoming increasingly available in a variety of climactic zones (include Mongolia’s, characterized by cold and windy conditions) and need to be promoted as a way of saving resources and improving air and water quality. For Mongolia, where heating costs can eat up a third of the budget of government services, the eco-friendly designs like those used in the EFA-FTI Project can set a good precedent and be used as a prelude to even greater efficiencies. 70. Regarding school construction, given the short construction season in Mongolia due to weather limitations, as well as booming demand, future projects need to plan carefully, assess local capacity and supervise implementation closely. Retaining a civil works engineer on the supervision team can be helpful. 7. Comments on Issues Raised by Grantee/Implementing Agencies/Donors (a) Grantee/Implementing agencies See Annex 7. (b) Cofinanciers/Donors Not Applicable. (c) Other partners and stakeholders Not Applicable. 18 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions) Component 1: Extending access 6.23 6.21 99.62 Component 2: Improving teaching 9.39 9.39 100.00 and learning environments Component 3: Extending access to 13.15 13.16 100.09 pre-school education in rural areas Component 4: Project management 0.63 0.64 101.82 Total Cost 29.40 29.40 100.00 (b) Financing Appraisal Actual/Latest Type of Percentage of Source of Funds Estimate Estimate Cofinancing Appraisal (USD millions) (USD millions) Free-standing Single Purpose Trust Fund EFA-FTI 29.40 29.40 100 Total 29.40 29.40 100 19 Annex 2. Outputs by Component 71. Data in this assessment is taken from Project progress reports, aide memoires, and ISRs that were drafted after the original completion date of December 31, 2009 (when most Project activities were finished), and includes outputs recorded by the end of 2009 (with the exception of the Baganuur School Complex, which was delivered in 2013). This assessment is not entirely consistent with the assessment of outputs in the body of the ICR (Section 3.2 above) since that one is based on a smaller (original) set of performance indicators and sometimes includes results packaged in different ways (e.g., it counts classrooms rehabilitated compared to schools where rehabilitation was done (below). Nevertheless, the main message in the two accountings is consistent: it is one of the Project’s meeting its targets on almost all indicators. 72. Component 1. Extending access (approximately US$6.2 million). This component contained three sub-components: (i) Rehabilitate and improve service facilities in 30 rural schools and dormitories • rehabilitation of school buildings: updated target 57; actual 55 (virtually met); • rehabilitation of dormitories: target 23; actual 23 (target met): • remodeling school toilets and hand washing places; dormitory bathrooms and cultural information centers; target 30; actual 30 (target met). (ii) Provide furniture and equipment to schools and dormitories: • teaching kits and learning materials for 6-year-old first grade classrooms: target 1750 schools, actual 1750 (target met); • desks and chairs for 6 year olds: target 1077, actual 1077 (target met); • dormitory facilities: target 88 dormitories; actual 88 (target met); • supply of kitchen facilities for school canteens (schools): target 35, actual 35 (target met); • supply of furniture and equipment to schools for children with special needs: target 6 schools; actual 6 (target met). (iii) Build a new general secondary school complex: the Baganuur Secondary School Complex (covering grades 1-11) was completed at the end of 2013 and inaugurated in early 2014; two other (smaller) new schools (in Huvsgul and Uvurkhangai) were completed under the Project: target: new schools built 3; actual number 3 (target met). 73. Component 2. Improving teaching and learning environments (approximately US$9.4 million). This component focused mainly on the provision of “adequate and continuous in-service training” for classroom and dormitory teachers. It had 3 subcomponents, as follows: (i) Training classroom and dormitory teachers • in teaching methodologies in mathematics, language and ICT: Math and ICT teachers in 2007: target 3000; actual 2200 (partially met); in 2009: target 1962, actual 1962 (target met). (Note: English [language] in-service training was dropped.); • in coping with the new in-take of 6 year olds (first grade teachers): in 2008 target 1432 (100 percent of new first grade teachers), actual 1449 (101 percent) (target met); in 2009 target 1602 (100 percent); actual 1602 (target met); in 2010 target 1778, actual 1778 (target met). • in providing better care to boarding students (training dormitory teachers): in 2008 20 target: 500 (100 percent of dorm teachers), actual 590 (118 percent) (target exceeded). • Added to the original targets was the training of social workers for the new 6-year- old cohort: target 580; actual 580 (target met). (ii) Supplying • mobile ICT stations for teachers: target 550, actual 550 (target met). • electronic books for teachers (# of teachers reached): target 3,000, actual 6,000 (target exceeded). (iii) Rehabilitating and equipping laboratories in teachers’ colleges: • in UlaanBaatar: target 7 labs, actual 7 labs (target met); • in selected (3) aimags: target 27 labs; actual 27 labs (target met). 74. Component 3. Extending access to pre-school education in rural areas (approximately US$13.2 million) by providing mobile ger pre-schools and rehabilitating the kindergarten teachers’ college in UlaanBaatar. This component includes three sub-components: (i) Deployment of mobile ger pre-schools and supply of furniture, equipment and learning materials: target 171; actual 171 (target met); furniture, equipment and learning materials provided: target 100 percent; actual 100 percent (target met). (ii) Rehabilitation of kindergarten teachers’ college: target 100 percent of rehab needs; actual 100 percent (target met). (iii) Provision of laboratories and learning centers to kindergarten teachers’ college: target 35; actual 35 (target met). 75. Component 4. Project management (approximately US$0.6 million). This component includes operational costs of the Project Management Unit and technical audit of the grant. Performance indicators for this were: • establishment of EDCM to supervise and approve EFA-FTI technical proposal and procurement plan (yes) – target met; • establishment of PIU (yes) – target met. 76. The Project also tracked the number of joint missions conducted with donors and findings reported to EDCM (target 5, actual 5 – target met). 77. There is no information on the implementation of (a) technical audit(s) of the grant. 21 Annex 3. Economic and Financial Analysis 78. The public expenditure analysis in the 2009 Program Document provided a general rational for investing EFA-FTI trust funds in Mongolian education by showing the high cost of delivering education in the country (due to its low population density and hash climate) and the fact that, despite growing national budgets for education, the government has not been able to raise funds for need capital expenditure and development costs. The analysis also points to the growing disparity between urban and rural areas, in which the former are losing population and resources, and later growing too fast to accommodate comfortably all of the children who need places in school. These conditions call for improvement in the efficiency of government expenditures in education, and policies and programs that address growing urban-rural disparities. These are the economic and financial issues that this annex addresses. 79. This ICR provides a discussion of the efficiency aspects of the project in terms of unit costs, as well as a review of the current trends in education financing. The ICR has not conducted a “Cost-Benefit Analysis” where it calculates a “Net-Present Value” or “Internal Rate of Return” in the absence of good data that links project activities with specific educational attainments clearly attributable to the project. Project efficiency 80. One way to examine project efficiency is to examine unit costs of project inputs/outputs compared to those of comparable products/programs. Of the many EFA-FTI Project inputs/outputs, four are amenable to a unit cost analysis, namely, the cost for training grade 1 teachers dealing for the first time with a 6-year-old cohort, the cost of the mobile kindergartens, the cost of the grade 1 furniture (provided to classes admitting 6-year-olds), and school building energy costs. These unit costs are compared to the cost of similar items as an estimate of efficiency. 81. Training of grade 1 teachers. During July-August in 2008 the Project trained around 100 teacher trainers for 3 days each; 1449 grade 1 teachers, 532 managers, and 590 dormitory teachers for 19 days each. In November it conducted a 3-day follow-up training for 1430 classroom teachers and 590 dorm teachers. The total number of training days for trainers, classroom and dormitory teachers, and managers was 53,043 and the Project budget provided for this was MNT 509 million or (at the exchange rate on July 1, 2008) US$266,839. The cost per trainer/teacher/manager was just over US$5. EFA-FTI READ Non-READ Project Unit cost for grade 1 teacher training US$5.03 US$3.14 US$7.62 82. This shows that the unit cost for this training was higher than that for the Bank-supported READ project but significantly lower than that specified in the Education Sector Plan (2006- 2015) for training in general (non-READ). Economies in the EFA-FTI Project resulted from the decision to undertake the training “in house” (using MES trainers) instead of relying on contractors as originally planned, which would have been more expensive. 22 83. Mobile Ger-Kindergartens. According to Project figures the cost for two gers in 2008 was just over US$2,000 (two are used in one kindergarten) and their materials and equipment was priced at approximately US$2,000 also (a total of US$4,000). Assuming that 20 children are accommodated in a ger on average, the cost per child is around US$100. Costs in a stationary kindergarten in 2008 (covering furniture, equipment, and learning materials, but not construction costs) was about US$45,000 for a building with a capacity of around 150 children (US$300 per child). Ger Kindergarten Stationary Kindergarten Facilities costs per child US$100 US$300 84. First grade furniture for 6-year-olds. Because 6-year-olds, who entered the first grade for the first time in 2008, are smaller than the previous cohorts (7- and 8-year-olds) right-size furniture was provided in many Project classrooms. The Project shows a unit cost for such furniture (desk and chair sets) of US$37.80 in 2008. Comparable would be the desks and chairs purchased for new stationary kindergartens, which in 2008 were listed as having a combined unit cost of US$144. Lower prices for the 6-year-olds could be attributable to economies of scale, since over 1,000 units were purchased, and international contract bidding. 6-year-olds in grade 1 New Kindergartens Unit cost for a desk and chair US$38 US$144 85. Energy consumption at Baganuur School. The new school complex constructed in Baganuur with EFA-FTI grant funds adopted an innovative energy and heating system, developed by GTZ (German aid), which is promising to bring significant economies. Its estimated energy consumption is as follows (to be confirmed during the course of 2014): Baganuur School Existing Schools Annual energy consumption per square 70 kwh 350-400 kwh meter 86. If this school complex does show promise during its first year or two it will be used as a model for other new schools, leading to significant recurrent cost reductions throughout the system, especially given that almost 15 percent of total educational expenditures in 2005 went to heating costs. 87. These items for which unit costs were calculable represent only a portion of total expenditures under the Project, but they do show that it was able to improve the cost- effectiveness of the education sector in many ways. 88. Financial analysis. Various WB project documents have revealed increases in public spending on education from the early transition period (1997) – 16 percent, to the year of Project launching (2007) – 17.5 percent, to the original EFA-FTI Project completion date of the project (2009) – 23 percent. These increases in financial allocations were accompanied by commensurate increases in per student costs. For example, the public expenditure per primary 23 school pupil as share of GDP per capita increased 37 percent from 2007 to 2011. 16 In pre- primary education over the same period, per student spending by the government increased from US$146 to US$643, largely due to a 2008 statute which required the state to provide free books, toys, and meals to all pre-school participants. 17 89. Over this period, government expenditures were almost entirely for recurrent costs, with the average share of the Ministry of Education budget devoted to capital expenditures ranging between a mere 5 and 10 percent. Consequently, the Government of Mongolia has looked to the international donors for help in covering its capital costs. This is reflected in the budgets of the two EFA-FTI/GPE projects in Mongolia, with EFA-FTI allocating 92 percent of its grant to civil works and goods (48.5 percent to the former and 43.5 to the latter); and the GPE Project allocating even a higher proportion – 98 percent – to the same (82 to civil works and 16 to goods). The forthcoming IDA Project (Education Quality Reform Project), represents a break from that norm, given that it funds no construction (aside from minor repairs funded through school grants), and devotes about 1/3 of its funds to training and management There is also evidence that the Government is starting to put more of its funds into capital expansion (e.g., plans to use state funds in building kindergartens within its “New Soums” project and up to 200 new mobile ger kindergartens), which is consistent with its recently reaching the status of a middle income (IDA level III) country. 90. State funding for education did increase over the past decade, but this did not mean that different parts of the country benefited equally. As the century unfolded, gaps between rural and urban areas began to widen. This revealed itself dramatically as changing urban-rural disparities in poverty levels. In the mid-1990s the proportion of population below the poverty line was about 35 percent, with urbanites being somewhat more impoverished than rural residents (38.5 vs. 33.1). By the end of the first 21st century decade (2010) the overall gap was still just above 35 percent, but the positions had reversed themselves: ruralites, on average, fell much deeper into poverty than urbanites -- by a full 20 percentage points (47 vs. 27 percent). 91. The government made narrowing the gap a priority and already by 2012 the figure in rural areas had declined almost 12 percentage points to 35.5 -- still high, but declining much faster than in urban areas where the rate only fell 4 percentage points (to 23.2 percent). Realistically, the main cause of this narrowing was not so much government policy as it was the rural poor seeking better opportunities in the city (by then more than 2/3 of Mongolia’s population was urban). Pockets of poverty (new unabsorbed rural migrants) began to build up there, too. 92. This created a disequilibrium in education. As rural areas lost population rural schools began to shrink, leading to a decline in their resources (given a strict national formula of per capital financing). Under-resourced schools in rural areas have contributed to declines in educational attainment there: four times more out-school school children among those in the 8- 16 Source: World Bank, EdStats. 17 World Bank, Early Childhood Education Project, Project Appraisal Document, 2012. 24 12 age range in the countryside than in UB and aimag centers; and 38 times more primary school dropouts in rural areas than in the national capital. The disequilibrium also meant problem in cities, largely in the form of overcrowding or lack of access. Many urban schools now have to triple shift, which reduces the instructional time and any possibility of individualized instruction. 93. Also, urban areas cannot absorb the massive influx of preschoolers who are seeking places in kindergartens. This is compounded by the recent increase in the total fertility rate (from 2.2 in 2000 to 2.7 in 2009), in part due to the government’s introduction of a child care cash transfers payable to every family for every child. At current levels of immigration and fertility and without any increase in supply, the gross enrollment rates in kindergartens could decline sharply from 76 to perhaps 50 percent by 2015, well below the targeted level of near 80 percent. 94. The global education partnership (EFA-FTI and GPE) grants were positioned to address the above disequilibrium by providing more resources to rural schools, by building more kindergartens for both herders and the neo urbanites, and even by erecting some new urban schools to help reducing triple shifting and to pioneer energy-saving power and heating systems that will help bring down operating costs. Having contributed to such causes for nearly a decade as the supervising entity for these trust funds, the Bank is poised to start a new chapter of educational support, that for improving educational outcomes through reforms to educational quality, which invests more in development than in capital costs. 25 Annex 4. Grant Preparation and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending/Grant Preparation Cristobal Ridao-Cano Economist, TTL EASHD Carol Chen Ball Senior Operations Officer EASHD Marguerite Clarke Senior Education Specialist HDNED Tserendagva Odongua Consultant DECDG Maria Caridad Araujo Economist EASHS Sabrina Gail Terry Program Assistant EASHD Supervision/ICR Cristobal Ridao-Cano Country Sector Coordinator, TTL EASHD Carol Chen Ball Senior Operations Officer EASHD Marguerite Clarke Senior Education Specialist EASHD Kin Bing Wu Lead Education Specialist, TTL EASHD Undariya Tumursukh Education Specialist EASHD Deepa Sankar Senior Economist SASED Donald Bundy Lead Specialist AFTHE Byambatsogt Jugder Human Development Specialist EASHD Ochir Lkhagvasuren Financial Management Analyst EASFM Gerelgua Tserendagva Procurement Analyst EASR2 Junxue Chu Senior Finance Officer LOAFC Prateek Tandon Senior Economist, TTL EASHE Jun Zeng Procurement Specialist EASP2 Pagma Genden Human Development Specialist EASHD Takiko Igarashi Junior Professional Officer EASHE Rabia Ali Economist EASHE Rajendra Swamy Consultant EASHE Tungalag Chuluun Operations Officer EASHD Dulguun Byambatsoo Financial Management Analyst EASFM Rajiv Aggarwal Consultant EASHE Maya Razat Program Assistant EASHD Anna Cornardo Program Assistant EASHE Dean Nielsen Consultant EASHE 26 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY07 0.0 0.0 Total: 0.0 0.0 Supervision/ICR FY08 5.1 38.3 FY09 19.3 133.7 FY10 20.8 51.4 FY11 3.1 1.7 FY12 0.0 9.1 FY13 0.0 7.2 FY14 11.4 57.3 Total: 59.7 298.7 27 Annex 5. Beneficiary Survey Results Not applicable. 28 Annex 6. Stakeholder Workshop Report and Results Not applicable. 29 Annex 7. Summary of Grantee's ICR and/or Comments on Draft ICR Implementation 95. In cooperation with MES, the EDCM, consisting of over 40 donors operating in the education sector in Mongolia, coordinates implementation of the EFA-FTI program with other donors’ activities in the sector, approves the EFA-FTI’s annual technical proposal and procurement plan, assist in program planning and implementation, share information, and monitor the overall process of the program as articulated in TOR that was approved by the EDCM in 2007. 96. MES was the executing agency for the Program. EFA-FTI continued to be supported by the Program Management Unit (PMU). The Catalytic Fund Program has been managed by the PMU. Its Executive Director also had the dual role of being the Director of the ADB-funded Education Development Program to ensure greater coordination among donors. The Deputy Director was in charge of daily operation of EFA-FTI. Six other program staff members were responsible for procurement, financial management, training, civil work, monitoring and evaluation and driving. 97. The EFA-FTI program implementation followed the World Bank’s procurement and financial management rules and procedures. The EFA-FTI program staff members have been trained in the World Bank procurement and financial management rules and procedures. The program team ensured the policies and procedures for the flow of funds, settlement of payments, accounting and reporting that are set forth in the Financial Management Manual. The manual has been approved by the World Bank. Monitoring and Evaluation (M&E), Design, Implementation and Utilization 98. The PMU in coordination with MES took the lead on monitoring and evaluation activities for the Program. 99. The Project output and outcome indicators were used in monitoring and evaluation. These monitoring indicators have been identified by the Mid-Term Review conducted in 2008 and agreed by MES and the EDCM. Recurrent costs for data collection and monitoring of the EFA-FTI activities under this Technical Proposal supported by Program Management (Component 4) budget. 100. The data related to output indicators provided by the PMU semi-annually. The MES provided outcome related data annually. Progress in meeting target indicators reviewed by the PMU semiannually and reported to the MES and the EDCM. All data collected sex- disaggregated in order to understand and respond to gender issues. Safeguard and Fiduciary Compliance 101. Financial managements and Audits: Financial management was rated satisfactory throughout implementation. FM team members participated in supervision and carried out 30 financial management reviews and each mission confirmed that the project has improved its financial management system to ensure it can provide, with reasonable assurance, accurate and timely information on the project implementation progress and whether the grant proceeds are being used for their intended purposes. There were minor issues that were flagged during FM reviews such as poor contract management, insufficient documentation for distributed goods. According to the FM team recommendation, the PMU accurately reviewed the goods distribution related documents upon the submission by aimag and district educational units. Other minor issues included were absence of tagging/coding the PMU assets that were commented on by the FM team and resolved prior to the next supervision mission. 102. Procurement: The PMU has developed the annual Procurement Plan which is approved by the donors and the Bank and made available at the PMU/MES and in the program’s database and on the Bank’s external website. The Procurement Plan is updated annually to reflect the content of the technical proposal. 103. The project’s procurement performance has consistently been rated satisfactory. Bank procurement staff was an integral part of supervision, and their reviews confirmed that the PMU procurement documents were properly filled and procurement guidelines were followed. The only one major issue was the construction of school complex in Baganuur district of Ulaanbaatar. Assessment of Outcomes Relevance of Objectives, Design and Implementation Rating: Highly relevant 104. Relevance of Objectives. The project development objectives were highly relevant to the country’s sectoral needs when the project was developed. The project addressed social inequality by increasing access to quality education for the poor, especially those living in rural areas. The thrust of the project was consistent with government’s education sector strategy, which emphasized increasing equity, access, and improving quality of education. The project objectives were consistent with the government’s ESMP 2006-2015, which aims to improve education standards and curricula and alleviate unemployment and poverty. 105. Relevance of Design. The Project was highly relevant at the design stage and remained so at the time of the project completion. The Project’s design and objectives were fully in line with government’s long-term strategic framework aiming to contribute to the Millennium Development Goals. 106. Relevance of Implementation. The relevance of project implementation is judged to be high. The EFA-FTI goal is to ensure universal completion of basic education and to close the gender gap and rural/urban gap in enrollment and school completion. To help Mongolia move towards the goal of quality basic education for all, the FTI grant aims to: (i) improve teaching- learning environments for primary education and strengthen the capacity of primary school teachers; (ii) provide mobile preschools to serve the herder population; (iii) construct and rehabilitate schools; and (iv) improve rural school facilities (to make them more child friendly, more healthy, and safer). 31 107. The EFA-FTI grant aids about 4 percent to the government’s annual expenditure for primary and secondary education and preschool education of roughly US$234 million. It provided universal coverage in training of first grade and dormitory teachers, training of teachers in UB Teachers’ College, and provision of mobile ICT teaching stations. In the provision of electronic books for primary teachers, the national coverage reached 78 percent. The EFA-FTI has built on the foundation of ger-kindergarten developed by UNESCO and UNICEF, and has vastly expended the service to herders, were access was limited before. Therefore, the project has contributed to improving the teaching and learning environment in Mongolia, particularly in rural areas. Achievement of Project Development Objectives 108. The performance indicators were outcome focused. All the outcome indicators were achieved on time which can be evidence that the PDO was achieved. 109. School visits found excited results on the ground. The project outcomes were not diluted across the country ensuring that the same follow-up activities were taking place in all the schools visited. 110. The demand and needs for services carried out under the Project were clear from the fact that many teachers, rural soum governors, parents of the students and remote herders have positive feedbacks and expressed their gratitude for these activities organized within the project. Efficiency 111. The project is rated efficient in its achievement of outputs of investment and in the impact of interventions to improve the access, quality and relevance of pre-school and basic education. With respect to efficacy, the project achieved most targets set at appraisal within the project period and within the original cost estimate, under a reduced scope. Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 112. The Project’s theme is to address the disparities between rural and urban schools by supporting activities that will specifically benefit rural schools and provide them with the opportunity to improve their educational services. Program activities take into account specific constraints and needs faced by rural schools and have been designed under consultation with rural teachers, other development partners, and NGOs whom have worked in rural schools. (b) Institutional Change/Strengthening 113. Institutional capacity has been strengthened through in-service training programs for primary teachers and administrators. The topics covered included a new teaching skills and methodology, teacher development, and teaching methodology improvements. Most of the 32 teachers who participated in a survey indicated that the training was useful and that they could apply the knowledge and skills they had learned to their jobs. 114. Teachers’ professional development network in rural areas helped to work schools closely and share their experience, improve teaching methodology inspiring teachers to develop teaching aids by themselves together with students. It emphasized the approach to empower local teachers to improve their self-development skills in a situation of lacking funding and support from the central government. Assessment of Bank and Borrower Performance Bank Performance Borrower Performance (a) Government Performance Rating: Satisfactory 115. The performance of the borrower was rated satisfactory. The government provided adequate counterpart funding requirements and demonstrated strong commitment throughout the project period. The government was strongly committed to education reforms and took significant steps to improve the quality of basic education. The project design was highly relevant to the government’s ESMP 2006-2015, and the Policy on Education of MES. It helped increase access to and completion of basic education, especially among disadvantaged groups in rural areas. The project was effective in achieving outcome indicators, including an increased gross enrollment rate in primary and secondary education; it reduced dropout rates, increased pre-school enrollment by providing mobile ger kindergartens; increased completion rate in primary and secondary schools in rural areas. The project was efficient in achieving outcomes and outputs, with targets achieved and some exceeded, within budget and within the project period, except the construction of Baganuur school complex. The project is likely to be sustainable, because project interventions have been implemented and integrated into the education system. Institutional capacities at rural schools and kindergartens have been strengthened. Overall, the government fulfilled its commitments and obligations. (b) Implementing Agency or Agencies Performance Rating: Satisfactory 116. The MES displayed strong ownership of the project and provided the needed guidance and continued support to the PMU. The Project Steering Committee (PSC) met regularly to review the progress of project implementation. The PMU was committed and was able to complete project activities on schedule. Despite delay in the construction of Baganuur school complex, the PIU managed to strengthen the construction supervision and management team and provided additional resources and vehicles for them to function effectively. The staff of the PMU provided continuity and stability for project implementation throughout the project period. Their knowledge of WB guidelines and procedures was useful in their work and in dealing with WB. Overall, the performance of the borrower and the MES was rated satisfactory. 33 (c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory 117. Based on the implementation analysis above, overall borrower performance is rated satisfactory. Project Specific Achievements 118. Component 1. Extending access. This component includes three sub-components: (a) Rehabilitate and improve service facilities in 30 rural schools and dormitories: • 55 rural school buildings rehabilitated. • 23 dormitory buildings rehabilitated. • Water supply and sewage systems of 30 schools were improved. (b) Provide furniture and equipment to schools and dormitories: • 1750 schools were provided teaching kits and learning materials for 6 year old first grade classrooms. • 1077 classrooms were supplied desks and chairs for 6 year olds. • Kitchen equipment distributed to 35 school canteens. • Furniture and equipment was supplied to 6 schools for children with special needs. (c) Build a new general secondary school complex: 2 new school-kindergarten complexes with each capacity of 250 children were built in Arvaikheer soum, Uvurkhangai aimag, Murun soum, Khuvsgul aimag. A school complex with capacity of 640 children was built in Baganuur district, UB. 119. Component 2. Improving teaching and learning environments by providing adequate and continuous in-service training. It has three sub-components: (a) Training classroom and dormitory teachers: Conducted 5 methodological t nationwide training for all first grade teachers, dormitory teachers and social workers, 2 ger kindergarten methodological workshops, 1 mathematic and 2 ICT trainings for primary teachers, 1 in-service training for teacher educators of Mongolian University of Education. In all above trainings approximately 10,900 primary teachers involved and gained methodological skills and knowledge. (b) Supplying: 550 mobile ICT stations and 6,000 electronic books provided to primary teachers. (c) Rehabilitating and equipping laboratories in teachers’ college: 7 laboratories of teachers’ college were equipped in Ulaanbaatar. 120. Component 3. Extending access to pre-school education in rural areas by providing mobile ger pre-schools and rehabilitating the kindergarten teachers’ college in Ulaanbaatar. This component includes three sub-components: 34 (a) Deployment of mobile ger pre-schools and supply of furniture, equipment and learning materials • 171 mobile ger kindergartens provided together with furniture, equipment and learning, materials in remote areas of each soum. (b) Rehabilitation of kindergarten teachers’ college. (c) Provision of laboratories and learning centers to kindergarten teachers’ college. • Rehabilitation and supply of furniture and equipment for kindergarten teachers college was completed in 2009. 121. Component 4. Project management. This component includes operational costs of the Program Management Unit and technical audit of the program. • The EDCM was established by initiation of MES to develop and implement Education Development Master Plan and strengthen the coordination of participating organizations in education sector. EDCM also supervised and approved EFA-FTI technical proposal and procurement plan. • PIU was established and was in charge of day-to-day operation. The project implementation follows the World Bank’s procurement and financial rules under the Memorandum of Understanding signed by MES, ADB, and WB. Project Overview 122. Donor countries had encouraged Mongolia’s Education Development Master Plan for 2006-2015 which was approved by Government of Mongolia. Effort of Mongolian Government and supports given from education sector donors were recognized and Mongolia was approved to receive the grant from EFA-FTI Catalytic fund in 2006. On March 9 2007, EFA-FTI project aimed to support effective implementation of Education Sector Master Plan started its activities following the General Contract between Ministry of Finance and World Bank, and Memorandum of Understanding on the implementation of the Grant between Government of Mongolia, World Bank and Asian Development Bank were signed respectively. 123. The main purpose of the EFA-FTI project is to improve access to basic education, enhance its quality services that are consistent with the system transition to 12 year general secondary education and assist to Education Sector Master Plan implementation. The project objectives were highly relevant to country priorities and consistent with global priorities. 124. EFA-FTI project had 4 components. First, extend access; second, improving teaching and learning environments; third; extending access to pre-school education in rural areas; and fourth, project management. 125. Project financing and cost: Total amount of grant was US$29.4 million dollars. World Bank has served as a trusted party with responsibilities to guide the project activities, implementation and funding. 35 126. Under the supervising of the EDCM and the World Bank, MES and PIU successfully implemented the project and contributed greatly for the effective transition to 12 years general secondary education and achievements towards Millennium Development Goals, Education Sector Master Plan, and Government Action plans. Project Implementation Overview 127. The implementation of the project has greatly improved the conditions and teaching quality for almost all Mongolia’s rural and urban public primary schools students and has brought a significant change in teaching and learning environments specifically in rural areas. The project was carried on smoothly during the implementation period. Most of the project targets were met and some were exceeded. Achievements are supported generally by progressively positive statistics in rural primary education sub-sector and there is evidence of increased enrolment or access to primary education. 128. MES has strengthened the national level management and paid more attention on the implementation stages providing technical and policy supports to project implementation agencies and committed to the projects sustainability. One of the evidence of sustainability is to continue to build ecofriendly in power consumption and heating, energy efficient school complexes using the design created for the Baganuur School in the future. Another evidence is the Government of Mongolia plans to create more ger kindergartens, thus 100 mobile ger kindergartens were supplied within “Global Partnership for Education and Early Childhood Education” project, 136 ger kindergartens were provided under “Early Childhood Education for Rural, Nomadic and Migrant Children” project funded by ADB. In 2014, 1 billion 448 million MNT was already allocated in the state budget for the procurement of 180-200 ger kindergartens. Based on norm-based variable costs, the Government of Mongolia already allocated the recurrent costs of above mentioned ger kindergartens in the state budget. Also, MES plans to build more kindergartens using the Canadian design that was used in the EFA-FTI project, and expenditure of building 8 wood framed kindergartens was allocated in 2013 and 2 more wood framed kindergartens was allocated in 2014 respectively. 36 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable. 37 Annex 9. List of Supporting Documents Catalytic Fund Program Document, Education For All – Fast Track Initiative Grant, World Bank, 2009. Grant Letter, Education for All - Fast Track Initiative (EFA-FTI) Catalytic Trust Fund Grant Number TF058043, 2007. Implementation Completion Report, Kingdom of Cambodia: EFA-FTI Education Sector Support Scale-up Action Program, World Bank, 2013. Memorandum of Understanding among the Ministry of Education, Culture and Science of Mongolia and Asian Development Bank, and the International Bank for Reconstruction and Development, International Development Association, 2007. Mid-Term Review Aide Memoire, Education for All – Fast Track Initiative Project, World Bank, 2008. Project Appraisal Document, Early Childhood Education Project, World Bank, 2012. Implementation Completion Report (IDA-H2210 IDA-H3730 TF-96333), Rural Education and Development (READ) Project, World Bank, 2013. Restructuring Paper, Mongolia: Education For All - Fast Track Initiative Catalytic Trust Fund Project, Grant No. Tf058043, World Bank, 2012. 38 M 1