Document of The World Bank Report No: ICR00003184 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-42050 TF-56841) ON A CREDIT IN THE AMOUNT OF SDR46.2 MILLION (U$67.5 EQUIVALENT) AND A GRANT IN THE AMOUNT OF US$25.3 MILLION TO THE REPUBLIC OF INDONESIA FOR AN EARLY CHILDHOOD EDUCATION AND DEVELOPMENT PROJECT June 27, 2014 Human Development Sector/Education Indonesia Country Department East Asia and Pacific Region i CURRENCY EQUIVALENTS (Exchange Rate Effective June 27, 2014) Currency Unit = IDR IDR 12, 099.21 = US$1 IDR 1.00 = US$0.08265 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS BAN-PT National Accreditation Board IKIP Teaching Training Institute for Higher Education BAPPENAS Agency for National IG Inspectorate General Development Planning BPS Badan Pusat Statistik or KB Kelompok Bermain or Bureau of Statistics Playgroup BSNP Board of National Standards of KDP Decamatan Development Education Project CDD Community Driven MDG Millenium Development Development Goals CDW Child Development Worker MOEC Ministry of Education and Culture CT Community Teams MOF Ministry of Finance CPICU Central Project Implementing NEST National Early Childhood and Coordinating Unit Specialist Team CSO Civil Society Organization NGO Non-Governmental Organization or Civil Society and Local Organization DESP-TF Dutch Education Support Trust NMC National Management Fund-World Bank Consultant administered DDTK Child Growth and PAUD Pendidikan Anak Usia Dini or Development Early Childhood Directorate DGHE Directorate General of Higher Perda Local Government’s Education Regulation DGNFE Directorate General of Non POM Project Operational Manual Formal Education DGQITEP Directorate General of Quality PPCU Provincial Project Improvement of Teacher and Coordinating Unit Education Personnel DIPA Government’s Annual Budget P2TK Teachers and Education Personnel Directorate DPIU District Project Implementing RA Raudhatul Afthal or Islamic Unit Kindergarten ii ECED Early Childhood Education and RENSTRA Medium Term Planning Development ECEDP Early Childhood Education and RMC Regional Management Development Project, the Consultant Project ECED-TF ECED Netherlands Trust Fund RPJM Medium Term Development – Government administered Plan EDI Early Development Instrument SK Surat Keputusan or Decree (“School Readiness”) EFA Education for All SUSENAS National Socio Economic Survey IFR Interim Financial Report TK Taman Kanak-Kanak or Kindergarten GOI Government of Indonesia Tupoksi Organization roles and function Vice President: Axel van Trotsenburg Country Director: Rodrigo A. Chaves Sector Manager: Luis Benveniste Project Team Leader: Rosfita Roesli ICR Team Leader: Rosfita Roesli ICR Primary Author: Suzana Nagele de Campos Abbott iii INDONESIA Early Childhood Education and Development Project CONTENTS Data Sheet A. Basic Information....................................................................................................... v B. Key Dates ................................................................................................................... v C. Ratings Summary ...................................................................................................... vi D. Sector and Theme Codes .......................................................................................... vi E. Bank Staff ................................................................................................................. vii F. Results Framework Analysis .................................................................................... vii G. Ratings of Project Performance in ISRs .................................................................... x H. Restructuring (if any) ................................................................................................. x I. Disbursement Profile ................................................................................................. xi 1. Project Context, Development Objectives and Design ........................................ 1 2. Key Factors Affecting Implementation and Outcomes ........................................ 6 3. Assessment of Outcomes ................................................................................... 15 4. Assessment of Risk to Development Outcome .................................................. 21 5. Assessment of Bank and Borrower Performance ............................................... 23 6. Lessons Learned ................................................................................................. 25 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 26 Annex 1. Project Costs and Financing .......................................................................... 27 Annex 2. Outputs by Component ................................................................................. 28 Annex 3. Economic and Financial Analysis ................................................................. 40 Annex 4: Project Impact Evaluation ............................................................................. 42 Annex 5. Bank Lending and Implementation Support/Supervision Processes ............ 62 Annex 6. Beneficiary Survey Results ........................................................................... 64 Annex 7. Stakeholder Workshop Report and Results................................................... 65 Annex 8. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 66 Annex 9. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 75 Annex 10. List of Supporting Documents .................................................................... 76 MAP iv A. Basic Information Early Childhood Country: Indonesia Project Name: Education and Development Project IDA-42050,TF- Project ID: P089479 L/C/TF Number(s): 53814 1,TF-56841 ICR Date: 06/16/2014 ICR Type: Core ICR Lending Instrument: SIL Borrower: INDONESIA Original Total XDR 46.20M Disbursed Amount: USD 93.44M 2 Commitment: Revised Amount: USD 92.8M 3 Environmental Category: C Implementing Agencies: Ministry of Education and Culture Cofinanciers and Other External Partners: Netherland Minister of Foreign Affairs and Development Cooperation B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 07/06/2005 Effectiveness: 12/11/2006 12/11/2006 05/15/2012 Appraisal: 03/14/2006 Restructuring(s): 05/15/2013 Approval: 06/27/2006 Mid-term Review: 02/01/2011 08/02/2011 Closing: 12/31/2013 12/31/2013 1 TF 53814 was reported under separate GRM 2 This figure reflects total disbursement under the project: IDA Grant 42050 (USD 70.04M) and Dutch TF 056841 (USD 23.40M). There were exchange rate gains to the SDR:USD exchange rate 3 This figure reflects the total funds made available for the project: IDA Grant 42050 (67.50M) and Dutch TF 056841 (USD 25.30) v C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Substantial Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) 4 Potential Problem Project Quality at Entry Yes at any time (Yes/No): (QEA): Problem Project at any Quality of Yes time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 5 5 Pre-primary education 90 90 Sub-national government administration 5 5 Theme Code (as % of total Bank financing) Child health 17 17 Decentralization 17 17 4 QALP-2 was initiated in late 2009 and completed in July 14, 2010. The rating given for Quality of Design was 2 (Satisfactory/ Likely) and Quality of Supervision was 3 (Moderately Satisfactory/ Likely) vi Education for all 33 33 Participation and civic engagement 17 17 Rural services and infrastructure 16 16 E. Bank Staff Positions At ICR At Approval Vice President: Axel van Trotsenburg Jeffrey Gutman Country Director: Rodrigo A. Chaves Andrew D. Steer Sector Manager: Luis Benveniste Christopher J. Thomas Project Team Leader: Rosfita Roesli Mae Chu Chang ICR Team Leader: Rosfita Roesli ICR Primary Author: Suzana Nagele de Campos Abbott F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The development objective of the project is to improve poor children's overall development and readiness for further education, within a sustainable quality ECED system. To achieve this objective, the project will: (i) increase poor communities' capacity to engage in participatory planning that will result in new or improved ECED services for their children and families (see Component 1); (ii) prepare the foundation for a sustainable ECED system through budgetary commitments from participating districts, establishment of a national quality assurance and professional development system, and district capacity building (see Component 2); (iii) ensure continuous improvement of service delivery and system building through establishing effective project management, and monitoring and evaluation (see Component 3). Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Increase in Early Development Instrument (EDI) scores of children entering kindergarten or 1st grade of primary school (stated as declines in percent of Indicator 1 : children vulnerable on each domain). Values are for 4 year old cohort from 2009 and 2013 surveys * Physical: 18.5 * Physical: 7.9 Value * Social: 6.6 * Social: 8.1 quantitative or N/A * Emotional: 35 * Emotional: 28.5 Qualitative) * Lang and cognitive: * Lang and vii 88.2 cognitive: 2.5 * Communication: 1.1 * Communication: 1.1 Date achieved 04/01/2009 05/01/2013 Comments Vulnerability in child development declined especially in physical, emotional, (incl. % language and cognitive. Social slight increase. No change in communication achievement) (already low). Detailed analysis is available in Annex 3. Improvement in children's development. Values are for 1 year old cohort of Indicator 2 : children from 2009 and 2013 surveys. N/A corresponds to item beginning dropped at endline since cohort outgrew the development measures. Gross motor (out of 20): 19.9 Gross motor (out of 20): Fine Motor (out of 13.7 12): 11 Fine Motor (out of 12): Language (out of 6.1 26): 19.6 Value Language (out of 5) 2.9, Cognitive (out of quantitative or Cognitive (out of 8): 4.4 N/A 8): N/A Qualitative) Socio emotional (out of Socio emotional 2) 1.6 (out of 2) N/A Receptive Language: 37.1 Receptive Expressive language: 39.8 Language:N/A Expressive language: N/A Date achieved 06/01/2009 05/01/2013 Comments Using child development scores, outcomes improve in all domains and language. (incl. % 2010 midline data shows increase: cognitive 7, receptive lang 85.5, expressive achievement) lang 100. N/A corresponds to item beginning dropped. (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Percentage of parents/communities who have received information regarding Indicator 1 : ECED Value (quantitative 0.0 80.00 99.80 or Qualitative) Date achieved 12/31/2007 12/31/2007 09/12/2013 Comments Target exceeded. Community facilitation successfully reached out parents/ (incl. % communities. Once services are running, they also serve as demonstration to achievement) attract more parents. Number of communities submitting an expression of interest to receive grants for Indicator 2 : ECED service development Value 0.0 6000 5990 (quantitative viii or Qualitative) Date achieved 12/31/2007 12/31/2007 09/12/2013 Comments (incl. % Target 99 percent met. Target not met in a few remote villages achievement) Indicator 3 : Number of block grants awarded to communities Value (quantitative 0.0 6000 5990 or Qualitative) Date achieved 12/31/2007 12/31/2007 09/12/2013 Comments Target 99 percent met. Block grants that were paid in 3 consecutive years, (incl. % supported village to establish and operate new services. Target not met in a few achievement) remote villages. Indicator 4 : Increased enrollment in ECED services by poor children Value (quantitative 0.0 738000 673162 or Qualitative) Date achieved 12/31/2007 12/31/2007 09/12/2013 Comments Target 91% met. Last monitoring was done in mid-2013. New enrollment was (incl. % expected starting in Aug 2013. achievement) Percentage of targeted districts with regulations supporting ECED positions in Indicator 5 : district government Value (quantitative 0.0 80.00 96.10 or Qualitative) Date achieved 12/31/2007 12/31/2007 06/17/2013 Comments Target exceeded by 20 percent. 48 out of 50 districts established permanent (incl. % ECED position and issued various district and even village regulations achievement) supporting ECED with long term sustainability effect. Indicator 6 : Development of standards and recognition of model ECED centers Value (quantitative 0.0 36 25 or Qualitative) Date achieved 12/31/2007 12/31/2007 09/12/2013 Comments National standard for ECED services issued in 2008. Target for establishment of (incl. % model centers meeting the standard is met in 25 out of 36 target districts (69 achievement) percent) due to district financial constraint in providing matching fund. Percentage of targeted districts with qualified and functioning district training Indicator 7 : teams Value (quantitative 0.0 80.00 72.20 or Qualitative) Date achieved 12/31/2007 12/31/2007 09/12/2013 Target 90 percent met. All of trainers trained all ECED teachers. However some Comments trainers did not conduct follow-up supervision due to lack of transport support (incl. % for visiting teachers in remote areas. District trainers continue to serve after achievement) project closed. ix Indicator 8 : Percentage of villages undertaking annual community assessments Value (quantitative 0.0 80.00 80.90 or Qualitative) Date achieved 12/31/2007 12/31/2007 09/12/2013 Comments (incl. % Target fully met. achievement) Indicator 9 : Percentage of villages submitting routine reports to districts Value (quantitative 0.0 80.00 96.00 or Qualitative) Date achieved 12/31/2007 12/31/2007 09/12/2013 Comments Target exceeded. Communication between village (with project supported (incl. % services) and district improve. achievement) Indicator 10 : Impact study undertaken Value Impact survey Impact survey (quantitative N/A completed completed or Qualitative) Date achieved 04/01/2009 05/01/2013 Comments 3 rounds of survey were completed in mid-2013 and preliminary result (incl. % disseminated achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 09/07/2006 Satisfactory Satisfactory 0.00 2 10/11/2007 Moderately Satisfactory Moderately Satisfactory 0.00 3 12/01/2008 Satisfactory Moderately Satisfactory 8.45 Moderately 4 06/12/2009 Satisfactory 19.89 Unsatisfactory 5 02/02/2010 Satisfactory Moderately Satisfactory 22.07 6 01/15/2011 Satisfactory Moderately Satisfactory 55.84 7 12/19/2011 Satisfactory Satisfactory 64.75 8 04/03/2012 Satisfactory Satisfactory 69.11 9 12/26/2012 Satisfactory Satisfactory 70.16 10 07/07/2013 Moderately Satisfactory Moderately Satisfactory 70.16 11 12/29/2013 Moderately Satisfactory Moderately Satisfactory 70.23 H. Restructuring (if any) x ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Reallocation of Financing Proceeds. To allow more fund 05/15/2012 N S S 69.78 for training and workshop under Part A of the Project Extension Closing date to 05/15/2013 S S 70.16 December 31, 2013 I. Disbursement Profile The graphic represents disbursement profile of IDA Grants only. xi 1. Project Context, Development Objectives and Design 1. The SDR46.2 million credit for the Early Childhood Education and Development Project (ECED) was approved by the Bank’s Board on June 27, 2006, signed on September 13, 2006. The project became effective on December 11, 2006 and closed on December 31, 2013. The Project was co-financed by a US$25.3 million grant from the Government of the Netherlands Trust Fund (the ECED-TF), approved by the Bank acting as a Trustee of a Dutch Trust Fund (TF) from the Netherlands Ministries of Foreign Affairs and of Development Cooperation. The Government administered the ECED-TF. The closing date of the Loan and the Grant was December 31, 2013. 1.1 Context at Appraisal. 2. Background. At the time of project preparation in 2006, Indonesia ranked as a lower- middle income country, with a gross domestic product (GDP) per capita of US$4,248. Indonesia had restored its macroeconomic and financial stability and economic growth had increased from 3.8 percent in 2001 to 5.6 percent in 2005 while the percentage of the population living in poverty in the period 1999-2007 had fallen from 27 to below 16 percent in 2007. 3. Despite all of these achievements, Indonesia continued to under-perform compared with its neighboring countries in terms of access to quality health, education and other basic services, as reflected in the millennium development goals’ (MDGs) indicators. Indonesia’s centralized development planning and service delivery systems had increased overall access to basic services but have fallen short in ensuring quality. 4. The Government of Indonesia (GOI, the Government) was committed to poverty reduction, and had laid out a comprehensive medium-term poverty reduction strategy in its Medium Term Development Plan (RPJM). The Government’s main approach towards poverty reduction was to increase incomes by means of productivity improvements and decreasing expenditures by means of lessening basic needs burden, such as education, health, and infrastructure that eases and supports social economic activities. 5. Decentralization of public service delivery had been an important element for effective implementation of the poverty reduction strategy. Indonesia was one of the more decentralized nations in the world. Districts were responsible for one-third of all government spending, and half of the development budget. Most spending on education, health and infrastructure was district based, and the districts employed three-quarters of the civil service. In 2006, 90 percent of the Ministry of National Education’s (MOEC) budget had shifted from the central to the district levels in the form of block grants channeled directly to the schools. Limited experience and capacity at both the district and school levels to manage the funds and lack of knowledge about effective inputs for quality education was a major constraining factor in the implementation of the decentralization laws for the sector. 6. Education was regarded as one of the most important elements in poverty reduction, and early childhood development as an integral part. The education gap between the rich and the 1 poor in Indonesia was pronounced, with poor children in poor districts exhibiting the lowest outcomes. Specifically, research suggested that districts with higher levels of preschool enrollment tended to have lower primary school dropout rates. More directly, children’s participation in ECED programs appeared to improve their readiness for primary school education through improving their overall development across domains. An evaluation of the World Bank-financed Early Child Development Project found that children participating in the project for two years had higher scores for school readiness at age 6 5. These positive effects for ECED program enrollment were even greater for the less privileged children who were living in poor districts or whose parents had lower levels of education. 7. Based on international evidence that showed the value of ECED, and building on some of its early efforts, the GOI was committed to prioritize ECED, as reflected in several government strategic documents, but it faced important challenges. The strategic documents included the National Education System Act No. 20 (2003), the Education for All plan and the Government’s National Medium Term Planning 2004-2009 (RENSTRA 2005). Challenges included: (i) the fragmentation of ECED service delivery and the delivery of health- and nutrition-related services and of parent education; (ii) extremely low participation in ECED, especially in poor communities (enrollment was only 8 percent compared with a 24 percent global average enrollment rate for low-income countries); (iii) lack of government investment in ECED and failure of the private sector to reach the poorest children; (iv) insufficient provision of services for families, infants and toddlers (despite the fact that approximately 90 percent of villages had Posyandus, i.e. village-based health posts); (v) limited progress toward a quality assurance system for ECED, including quality standards and monitoring and supervision; (f) lack of ECED training for managers, teachers and field workers; (vi) fragmentation of responsibility for ECED services across ministries and directorates overseeing different forms of early childhood development services; and (vii) limited district-level capacity to implement ECED programs, especially relevant since districts’ responsibilities had been expanded to include implementation, financing and supervision of ECED services in their respective areas. 8. The GOI’s strategy aimed to provide expansion of access to quality ECED services with equity through major investments in: (i) promoting community-based and decentralized service delivery, and strengthening community capacity in service delivery; (ii) targeting the poorest communities and households for public financing of ECED service provision; and (iii) strengthening system management capacity in planning, quality assurance, and monitoring and evaluation at central, provincial and district levels. The Project was designed to support the GOI’s strategy in ensuring that children aged 0 to 6 from economically disadvantaged families and rural areas participate in and benefit from non-formal integrated quality ECED services. 5 World Bank, Early Child Development Project (approved July 28, 1998), Project Appraisal Document Report No. 18151 dated July 7, 1998. 2 1.2 Original Project Development Objectives (PDO) and Key Indicators 9. The Project’s Development Objective was to improve poor children’s overall development and readiness for further education within a sustainable quality ECED system. To do this, the Project was to: (i) increase the capacity of poor communities to engage in participatory planning that would result in new or improved ECED services for their children and families; (ii) prepare the foundation for a sustainable ECED system through budgetary commitments from participating districts, establishment of a national quality assurance and professional development system, and district capacity building; and (iii) ensure continuous improvement of service delivery and system building through establishing effective project management, and monitoring and evaluation. 10. The Project’s progress towards its PDO was to be measured through the following PDO indicators: (i) increase in early development scores of children entering kindergarten or first grade of primary school; and (ii) increase in Child Growth and Development (DDTK) scores of children 0-3 years. The intermediate indicators were: Component 1: (i) percent of parents/communities who have received information regarding ECED during community mapping exercises; (ii) number of communities submitting an expression of interest to prepare an ECED grant proposal (by completing a form distributed during village pre-conditioning process); (iii) number of block grants awarded to communities; (iv) increased enrollment in ECED services by poor children. Component 2: (i) development of standards and recognition of model ECED centers; (ii) percent of targeted districts with regulations supporting ECED positions in district government; and (iii) percent of targeted districts with qualified and functioning district training teams. Component 3: (i) percent of villages undertaking annual community assessments of existing ECED services and unmet ECED needs; (ii) percent of villages submitting routine reports to districts on project activities and uptake of ECED services; and (iii) impact study undertaken. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 11. The Project’s Development Objectives were not revised during the project period. 1.4 Main Beneficiaries 12. The primary beneficiaries of the Project as described in the Project Appraisal Document (PAD) were poor children ages 0 to 6. Through geographic targeting at all levels, the Project was to focus on an estimated 738,000 children living in 50 of the poorest districts of the country and in the poorest kecamatans, villages and communities within those districts. Once a community was identified and received a grant, all children with that community were to benefit from the ECED services implemented, including children with disabilities. The Project’s secondary beneficiaries were to include: (i) parents and caregivers, particularly those of children ages 0 to 3 that would receive relevant information and support from trained community members on how to best nurture the development of their children; (ii) the 6,000 teachers and 6,000 child development workers that would receive both systematic training in early childhood care and development and close supervision in their daily practice with poor children; and (iii) civil 3 society and local organizations (NGOs) already active at the community levels whose presence the Project would capitalize upon to be involved in all stages of the process, including training, service delivery and independent monitoring. The Project was also expected to have a demonstration effect on the Government, which could then expand the coverage of ECED services for poor children in Indonesia. 1.5 Original Components 13. The Project objective was to be achieved through three components: 14. Component 1: Increasing Integrated ECED Service Delivery Through Community- driven Mechanisms in Targeted Poor Communities (US$96.65 million). This component aimed at delivering integrated ECED service to poor communities in fifty districts following a Community-Driven Development (CDD) approach. The 50 districts chosen to participate in the Project were selected according predefined criteria, detailed in the Project Appraisal Document. This component comprised four sub-components, as follows: 15. Sub-Component 1.1: Establishing Training Programs and Providing Staff Development (US$12.20 million). This sub-component was to finance the training of 50 members of the National Early Childhood Specialist Team (NEST), 150 district trainers, 600 facilitators (200 three-member teams consisting of an ECED Specialist, A Family Support, Health and Nutrition Specialist, and a Community Development Specialist), 6,000 teachers (for services to children ages 3 to 6) and 6,000 child development workers (CDW) for services to children ages 0 to 3 and their families. 16. Sub-Component 1.2: Preconditioning, Launching and Facilitation of the Community Grant Process (US$18.89 million). This sub-component was to finance: (a) the selection of eligible kecamatans and villages within about 50 participating districts (about 60 villages per district, that is a total of 3,000 villages nationwide); (b) the sensitization of villages about the importance of ECED services (preconditioning phase); and (c) the launching and facilitation of the community grant process. 17. Sub-Component 1.3: Providing Block Grants to Poor Communities. This sub- component was to finance the provision of grants to about 3,000 villages (with an average of two beneficiary communities or dusuns per village) in about 50 districts. Grants were to be awarded twice a year on the basis of a proposal review conducted by the DPIU. 18. Sub-Component 1.4: Recognizing Model ECED Services and Using Them for Demonstration Purposes. This sub-component was to finance additional block grants to one ECED model program per eligible kecamatan (about 10 model programs per district). 19. Component 2: Developing a Sustainable System for ECED Quality. This component was to help ensure quality in the programs financed under Component 1, and create conditions that would support sustainable, scaled-up provision of holistic, quality ECED services. 4 20. Sub--Component 2.1: Quality Assurance. This sub-component was to finance consultancy services, professional development workshops, and materials development to assist the Government in developing standards and recognition procedures to support quality ECED delivery. Assistance was to be directed to relevant units in MOEC and related agencies, in consultation with the Board of National Standards of Education (BNSP). 21. Sub-Component 2.2: Institutionalization of Quality ECED at the District and Provincial Level. This sub-component was to finance facilitation of coordination among and partnership of ECED-related agencies; inclusion of ECED in local governments’ regulation (Perda) on organizational structure, roles and function (Tupoksi) in Dinas Pendidikan as well as other ECED-related agencies. In addition, it would support expert training and workshops to improve the knowledge bases of the key ECED-related officials; and provide matching grants on a competitive basis to district governments and existing ECED service institutions to establish such resource centers to support the training. 22. Component 3: Establishing Effective Program Management, Monitoring and Evaluation. This component was to help ensure continuous improvements of both ECED service delivery and system building over time. 23. Sub-Component 3.1: Program Management. This sub-component would finance both project management at the central and district levels. For project management at the central level, the Project would finance activities to support MOEC in implementing its mandated roles and functions. For project management at the district level, this Sub Component would finance activities to support participating districts in implementing their mandated roles and functions. 24. Sub-Component 3.2: Monitoring and Evaluation. This sub-component was to finance the development of a monitoring and evaluation (M&E) and supervision system for ECED programs at village/community and district levels, including those ECED programs not supported by the Project. It also would support formal evaluation 6 of project benefits. 1.6 Revised Components 25. The Project’s components were not revised. 1.7 Other significant changes 26. A project restructuring was approved on May 15, 2012. The purpose of this restructuring was to reallocate funds among expenditure categories in both the Credit Agreement and the Trust Fund Grant Agreement to reflect the larger number of village teachers trained (reflecting greater demand), the reductions in Regional Management Consultants (RMC) due to procurement delays and the need to increase technical support for project management and monitoring. The closing date of the Dutch trust fund was extended from June 30, 2013 to December 31, 2013 to allow for 6 This “formal evaluation” is what became the Project’s Impact Evaluation (Section 2.3). 5 continued financing of activities under sub-component 2.1 and to continue financing consultants supporting project management under Component 3. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 27. Project Preparation. The Project addressed the GOI’s priorities and counted upon its commitment and ownership, including at the district and village levels; it was also in line with the World Bank’s Country Assistance Strategy (CAS). The Government was committed to poverty reduction and to improving services to the poor by strengthening decentralization of service delivery. Expanded provision of ECED services were seen as fundamental for Indonesia to achieve MDG targets in education and health, and those set by RENSTRA and the EFA Plan. Yet, ECED was still a relatively new sub-sector, and there were challenges in the “start up” phase. Consistent with the CAS focus on areas most crucial to poverty reduction, including the lack of quality service delivery to poor people, the Bank was well placed to assist the GOI in closing both the financial and technical capacity gaps that the Government had identified. 28. The Bank preparation and appraisal teams consisted of technical experts that were appropriate for the development of the project. A Quality Enhancement Review (QER) was also conducted and the project design took into consideration comments which included the need to: (i) provide services for 0-3 year old children; (ii) include indicators that measured overall child development for children 0-3 rather than specific standalone health or nutrition indicators; (iii) provide a simple operations manual for use at the local level; and (iv) establish a mechanism within the project management structure for coordination between various ministries such as a steering committee. The preparation team incorporated lessons from earlier and ongoing Bank- supported projects in Indonesia as well as other Bank-supported ECED projects. However, lessons learned were limited at the time because there were few standalone ECED projects under implementation in the Bank and ECED was a relatively new investment sub-sector in the Bank. The preparation team also identified the appropriate critical risks to the project and proposed mitigation measures within the project design. 29. Project Design. The PDO for the project was appropriate for an outcome-focused project, which was also the view of the 2006 QER and the project decision meeting reviewers. In addition, the results framework, at the time of preparation, was appropriate to measure achievement of the PDO. The framework had two PDO indicators to measure overall development and readiness for further education and appropriate intermediate indicators to monitor progress toward development of a sustainable quality ECED system as well as that of component implementation. The QER and peer reviewers all confirmed this judgment. The DDTK scores of children 0-3 indicator was selected specifically to respond to the QER recommendation. It was also an existing local instrument that could be used to strengthen linkages with the health sector. Baseline data for the PDO indicators was to be determined by a baseline data survey in year 1 of the project due to the fact that the Indonesia ECED system was just being developed and a data system was not yet fully functional. These were areas that the project would support for further development. 6 30. The Project was designed to be implemented within the Government’s institutional structure and budget framework, utilizing a system of disbursing a flat rate of block grants. The Central Project Implementing and Coordination Unit (CPICU) reporting to the Director of Early Childhood in MOEC was responsible for monitoring overall implementation and the coordinating activities in 50 poor districts at the village level. 7 The budget framework and disbursement flows that were used had been applied under other Bank-financed projects in Indonesia, including those in the education sector. By targeting poor districts rather than poor families at the village level, the design ensured a manageable implementation approach. This, coupled with knowledge that children benefit most from the ECED experience when they interact with children from more favorable socio-economic backgrounds, made targeting districts more appropriate. Moreover, the project design ensured that the 50 districts had the capacity and were committed to the project as demonstrated by financial contributions and establishment of institutional structures. Participating districts also signed a Memorandum of Understanding (MoU) between MOEC and district governments. The MoU included selected strategic measures that districts needed to adopt to have long-term impact, such as: (i) the incorporation of ECED (as a new sub-sector) in the district’s 5-year Development Plan (Renstrada); and (ii) the establishment of an ECED unit within local education office (as a district anchor in the ECED program). Establishing an ECED unit eventually lead to districts allocating local funds to continue advancing the ECED agenda. 31. A more community-driven approach, consistent with the GOI’s decentralization framework, was incorporated under the Project, where villages were to be empowered, with support from facilitators, to decide upon those services that best suited their needs. An extensive review was carried out of Bank support to various Community-Driven Development (CDD) models applied in several sectors in Indonesia. Based on experience, the Project adopted an approach that channeled grants from districts to communities, along with direct involvement of local government (especially at the district level) to ensure both technical inputs during implementation and financial support for recurrent costs. 32. The design included components that reached all levels of key stakeholders and strengthened their respective roles, thereby ensuring a holistic approach to building sustainability. Specifically, the Project supported: (i) the MOEC at the central level to develop standards and policy; (ii) provincial level to strengthen their coordination role; (iii) the district level to strengthen the enabling environment, implementation and monitoring capacity; and (iv) the community level to understand the benefit of ECED for their children as well as establishing and managing community based ECED service. Moreover, project support at the community level consisted of the key ingredients, such as facilitation, fund and training for teachers, while requiring villages to contribute to the infrastructure, which was another measure to ensure sustainability. Participatory planning was a central feature in the villages’ and communities’ definition of priorities. Through established fora, villagers identified priority communities and their representatives, who would, in turn, through participatory planning processes, identify priority needs for delivering ECED services. 7 The project included 50 poor districts and 6,000 ECED centers in the districts. 7 33. The project design also included a parallel co-financing grant in the amount of US$25.3 million from the Royal Netherlands Government. This grant, which was recipient-executed, supported the development of a sustainable quality ECED system by supporting: (i) the establishment of core for the National Early Childhood Specialist Team (NEST); (ii) the development of materials to ensure the that project would begin with basic capacity for implementation and facilitation of the community grants, project evaluation and capacity building; and (iii) international and most local consultancy services, fellowships and training. This support was fundamental in financing initial activities required for the Project, which included: (i) a decree on participating districts and on the establishment of the CPICU; (ii) expressions of interest by all 50 districts; (iii) MoUs between the MOEC and district governments; (iv) selection of communities consultants; and (v) the design of the impact evaluation and instruments for the baseline survey. These activities ensured that implementation could begin in a timely manner. The donor-financed grant represented an instrumental and flexible vehicle in support of enhanced implementation. 8 2.2 Implementation 34. Overall implementation was positive. Implementation began with a 2006 implementation preparation mission pending effectiveness. The CPICU was already preparing implementation plans for FY2006 and 2007; the Bank team and PAUD (Pendidikan Anak Usia Dini or Early Childhood Directorate) held discussions on the development of training materials for the NEST trainers; criteria for NEST national trainers had been confirmed; discussions began on the development of ECED standards; and on the feasibility of building in a randomized designed in a few districts for the impact evaluation. By 2007, implementation of all project activities was underway mostly with support from the Dutch trust fund. Credit disbursements were low due to the fact that a government decree was issued cutting travel costs for government programs, impacting government officials’ ability to travel to the 50 districts and support project implementation. This was resolved when the government clarified that the cuts did not apply to externally funded projects. Once this was resolved, credit disbursements increased and the project continued to move forward. 35. In 2009 and 2010, project activities continued to be implemented. The baseline survey for the impact evaluation took place in 2009 with a second round of data collection in 2010. In addition, the project had: (i) established community-based ECED services; (ii) distributed the first tranche of block grants; (iii) developed and piloted parenting training modules; (iv) provided district level support for ECED teachers; (v) issued national ECED standards through the 2009 Minister Decree 58; (vi) worked with higher education leaders to identify desired competencies for ECED graduates; and (vii) issued district level financial regulations for supporting ECED within their district budgets. It was at this time that the Bank’s Quality Assessment of Lending Portfolio (QALP-2) assessed the likelihood of the project achieving its PDO as Moderately Likely, partly because it was too early to assess the overall quality and impact of the ECED 8 Reporting on the trust fund has been through the Bank’s Grant Reporting and Monitoring (GRM) form. 8 interventions. It found the project well designed, with strong government commitment at all levels. 36. While there were many achievements; there were also challenges. In 2009, due to MoEC reorganization, most of key personnel involved in project design were replaced. This included the Director General, Director of ECED (implementation unit) and CPICU coordinator. At this time, the new Director requested substantial changes to the project design and implementation arrangements by significantly reducing the role of consultants at all levels and replacing them with government staff. The procurement process of key consultant packages (mostly regional management consultants) were put on hold during this time. The Bank team had several meetings with the new Director explaining the importance of the original design which led to an agreement to keep the original project design and implementation arrangements. This however, slowed the implementation process. 37. In late-2010, another reorganization took place. The responsibility for formal and non- formal ECED was merged under the “Directorate General of ECED, Non-Formal and Informal Education”. This led to an increase in directorates relevant for the project: the Directorate for ECED (formal and non-formal) and the Directorate for Professional Development of Teacher and Personnel of ECED, Non-Formal and Informal Education (P2TK PAUDNI). Although project implementation remained under the Directorate for ECED, the reorganization of staff slowed implementation activities again. One notable change occurred when in 2010, approximately 130 community facilitators resigned for a variety of reasons (i.e., secured permanent civil service employment, ran for local offices, etc.), which required the project to recruit new facilitators and establish procedures to ensure continuity in village support. Financial management during this period presented challenges and was downgraded to moderately unsatisfactory. The QALP-2 also rated implementation as moderately unsatisfactory primarily in response to shortcomings in financial management and procurement processes. 38. By 2009, it was also increasingly evident that there were challenges with the PDO indicators, particularly with the DDTK Scores of children 0-3 years. Although it was selected based on recommendations from the preparation stage QER, it became increasingly obvious that this indicator was more appropriate for determining whether a child had special needs rather than measuring children’s development over time. Moreover, it was not feasible to find and train enough psychologists for the administration of the DDTK considering the scale of the project. The implementation teams reviewed the Early Development Index (EDI) and determined that it was more suitable for the purposes of the project since it is a school readiness instrument. It covered the same aspects of development as the DDTK but was more relevant to the projects aims and did not require formal psychologists to undertake the assessment, making it easy to complete by either the caregiver or teacher of the child. Examples of the proxy indicators used to replaced most of the DDTK indicators included: (i) the number of tasks children are able to complete (covering the domains of gross motor, fine motor, language, cognitive and socio- emotional development), and (ii) the percentage of children able to answer the receptive and expressive language questions correctly. (See Annex 3 for the full comparison of indicators.) At the same time, the team made the decision to use the impact evaluation to gather, analyze and report on achievement of the PDO. This is judged to have been an appropriate decision on the part of the Bank and government implementation teams because of the data challenges and the 9 fact that this approach could provide more direct information on project attribution and PDO achievement. 39. The 2011 mid-term review (MTR) confirmed that the PDO was appropriate and that there was a high likelihood of achieving the PDO. This was important particularly in light of the modification to the PDO indicator that measured child development for children ages 0-3. At the time of the MTR, the project was also on track to complete all project activities with a credit disbursement rate of 91 percent and 76 percent for the Dutch trust fund. The Bank also began to dialog extensively with the government at the central, district and community level about the need to have all necessary policies and institutional measures in place to ensure that the ECED model would be sustained. It was during the MTR that the government requested a reallocation of the Credit funds between disbursement categories. The Bank team agreed to put forth the necessary project restructuring documentation for the reallocation in April 2012. In late 2012, another restructuring was processed to extend the closing date of the TF which was completed in April 2013. At the time of the final restructuring, the implementation unit also made the request to restructure the result framework to formally reflect the use of the proxy indicators as a replacement to the DDTK. The package was submitted to the Ministry of Finance and Bappenas. However, the government felt that the agreements within the project documentation was adequate and was not inclined to pursue the process further. 40. At the time of project closing, the structure of a successful ECED program was in place, with many sustainable features such as training and monitoring, as well as a strong understanding and demand from the beneficiary district governments. Furthermore, the delivery model is replicable and can be scaled-up across Indonesia. This is a substantial achievement. Over the project implementation period, the Bank team conducted six formal implementation support missions and the Bank’s Indonesia country office team provided almost continuous on- the-ground support from preparation through completion. The make-up of the Bank’s teams was well balanced with financial management staff well represented on missions. The project closed as scheduled on December 31, 2013 and had disbursed 99 percent of the credit (SDRs 45.9 million or US$70.4 million equivalent) and 92 percent of the Dutch trust fund (US$23.4 million). 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 41. Design. The M&E design was twofold. The first focus of the M&E design was to develop a system to monitor project outcomes. The development of the system was important since there was no formal ECED data gathering system at the time the project was designed. The system would provide information for project outcomes as well as project outputs and was to be managed by the CPICU. The second was an independent evaluation of the ECED community- based interventions would be assessed through an impact evaluation that also looked at trends in some child development indicators. Putting in place a system to monitor project outcomes was seen as critical for sustainability of ECED programs. As such, design focused on developing a monitoring, supervision and evaluation system for ECED programs at village/community and district levels, including those ECED programs not supported by the Project. This system would be developed through annual community assessments comprising: (i) registration of children aged 0-6, including background information on family circumstances, current participation in ECED programs and anthropometric assessments; (ii) an inventory of ECED services and 10 resources accessible to a village/community; and (iii) an assessment of the developmental levels of children entering kindergarten and school using a “school readiness” instrument 9. 42. The development of an ECED monitoring system was judged to be highly relevant for the institutionalization of ECED sub-sector. Moreover, the original results framework was judged to have been appropriate at project design because it was monitoring child development and readiness although no specific target value was assigned for the respective indicators. Although gathering data related to the DDTK scores of children 0-3 proved to be difficult and, more importantly, not the appropriate indicator for assessing child readiness 10 , the project team quickly made the adjustment and adopted new indicator measures that proved to be appropriate. The intermediate indicators were linked to the components and were appropriate to measure intermediate level achievements. The inclusion of the impact evaluation was highly relevant and an important design feature for understanding the effectiveness of project interventions. 43. Implementation and Utilization. During implementation monitoring and evaluation focused on: (i) development of a system that tracked the intermediate outcome indicators, and (ii) an impact evaluation which evaluate the PDO indicators (school readiness and child development. 44. The monitoring system was developed and is user-friendly and reliable, and an improved data entry application was developed to support the system. Monitoring activities were managed by community facilitators (village level), by the DPIUs and the RMC/District Consultants (district level), by the provincial implementing units (provincial level) and the CPICU and NMC (central level). The system is now capable of providing comprehensive data from the field to guide decision-making and project improvement. The system counts on data entry of the 5,990 project-supported village implementing units level, and is capable of aggregating data by village, sub-district, district, and province, and for the whole 5,990 units. Training on use of the monitoring system was provided through a cascade model to the NMC, RMCs and district consultants and field facilitators at the community level. 45. The collected data in the monitoring system encompasses the intermediate indicators in the Project’s results framework, including carefully tracking the number of new enrollments in each of the 5,990 project supported centres. 9 School readiness would be measured using the EDI, an instrument correlated with later school performance that looks at physical health and well-being, social competence, approaches to learning, emotional maturity, language development, cognitive development, communication skills and general knowledge. The instrument was considered to provide communities with an easy to understand and psychometrically sound instrument to monitor their efforts to improve early year’s outcomes over time. This instrument had been used in developed and developing countries, and was adapted to the local context. 10 The DDTK is a diagnostic tool developed based on a Denver test and adapted to Indonesia’s country context that tracks individual development of children. It also measures whether a child has a special need and, as such, should generally be used as part of an intervention rather than for monitoring and evaluation. There were important issues, including institutional, in terms of the availability and linkages with professionals to track individual children and the consequent financial implications for its usage. This made its use under the Project inappropriate 11 46. Five cycles of annual monitoring were carried out under the Project, and in each of these, improvements were made to revise and improve the monitoring instrument and data entry system, and reduce the bottlenecks in data entry to improve timeliness of information. This was so successful that by the end of the third cycle information was received from districts within two months compared to six months in the previous cycle. The results of the data analysis were routinely discussed with the ECED Directorate and district representatives. The system has developed a rating system for ECED centers using a spider web diagram (Annex Figure 1 in Annex 2) that can provide an indication of the performance of individual ECED centers, and aggregate the data at higher level. 47. This capacity is used to map the centers and the specific aspects in service delivery that need improvement. The data provided from the monitoring system is particularly important for district level supervision as well as informing resource allocation and focusing support to the needy areas. Furthermore, feedback and monitoring from project-supported activities (e.g., professional development, including teacher training) were routinely carried out and used to adjust and improve programs, manuals, and the design of specific activities. Based on experience with the monitoring system supported by the Project, the MOEC expressed interest in a comprehensive monitoring system that would allow them to map the performance of ECED services across Indonesia The M&E system developed in the project is being adopted by Directorate of Teacher and personnel PAUDNI, especially for supporting school supervisor in their tasks in quality assurance. The comprehensive M&E system put in place under the Project is a major accomplishment. It is being utilized to rate ECED centers according to various criteria that reflect the quality of service delivery, identifying areas that need attention to improve overall quality, not only in project-supported centers, but also, by linkage to MOEC monitoring systems to all ECED services across the country. 48. The impact evaluation carried out under the Project has provided valuable information to inform and document the Project’s impact. The Bank’s implementation support team provided just-in-time advisory support to the MOEC in all phases of carrying out the IE, and set up a team in the country office to produce technical and policy papers that make use of the IE’s rich data to guide its education sector work. 11 For each round of the survey, a joint GOI-Bank team worked closely to ensure the quality of data being collected by the government, through contracted survey firms. In the first two rounds (2009 and 2010), the Bank team provided intensive support to the CPICU to ensure reliability of data as well as build the capacity of the CPICU to analyze the data and produce reports based on the analysis. In the end, the Bank used the data provided by the system to produce the 2013 Directions in Development report Early Childhood Education and Development in Poor Villages of Indonesia: Strong Foundations, Later Success. (See Annex 4 for full details of the impact analysis methodology and data analysis). 11 P110517 (ID-ECD Evaluation) 12 2.4 Safeguard and Fiduciary Compliance 49. Safeguard Compliance. The Project was classified as Category C for environmental purposes so an environmental management plan was not prepared. Nevertheless, the Project Operational Manual (POM) contained appropriate guidance concerning how to address issues of waste disposal, noise, etc., in the event that community projects involved minor construction/renovation. 50. The Project did trigger OD4.20 (now revised as OP4.10), Indigenous Peoples policy, as isolated vulnerable peoples were present in some areas of the 50 districts covered. A Framework for Treatment of Indigenous or Isolated Vulnerable People was prepared that identified activities to be carried out to ensure that the Project addressed the special needs of these groups. The specifics of these activities were included in the POM. Section 3.4 presents a summary of the Social Assessment carried out upon project completion. 51. Fiduciary Compliance. There was a risk assessment of both financial management and procurement as high, in part due to the lack of capacity at all levels, and in part due to the Project’s decentralized implementation. To address these risks specific action plans in each of these areas had been agreed, and the Project included financing for identified critical activities (e.g., strengthening of MOEC internal control capacity by financing the development of audit manuals, including audit or community block grants and the training for staff of the Inspectorate General). The project operational manual (POM) documented the financial management system and procedures (including procurement) to be followed by the Project’s implementing agencies at all levels. Consultants were contracted to support the CPICU in fiduciary areas. 52. Despite this, financial management and procurement issues surfaced, especially in early implementation, requiring intensive and repeated follow-up by the Bank’s implementation support team. For financial management, this was evidenced by delays in: (a) appointing a financial management consultant to the CPICU; (b) introducing minor financial management revisions to the POM; (c) contracting the technical assistance to strengthen the MOEC’s internal audit function; (d) submission of Interim Financial Reports; (e) conducting transaction reviews and payment verifications, appropriate recording of expenditures and filing of supporting documents and addressing expenditures backlogs; and (f) providing follow-up to observations in the Project’s external audit. These issues were addressed by early 2012, and consultants at the district level were contracted, significantly improving financial management performance at the district level. External audits were timely and unqualified, although they did occasionally contain observations. 53. Procurement. Procurement also surfaced as a major issue that required attention and adjustment during implementation. The Project’s original procurement plan had envisaged the contracting of a National Management Consultant (NMC) and five Regional Management Consultant (RMC) firms to strengthen the capacity of project management in participating districts (including for procurement). Each of the RMCs would contract community facilitators to provide support at the village level. Procurement of the RMCs suffered delays, due in part to weak capacity of the procurement team and the re-organization in MOEC, so in the meantime, the CPICU directly contracted 600 community facilitators individually. After the contracting of 13 two RMCs had been concluded (RMC1 and RMC4), they took over the contract of the community facilitators working in their respective regions. For areas without RMCs, 34 district consultants were contracted directly by CPICU, and the rest of community facilitators continued to be contracted by CPICU on individual basis. These individuals, contracted as non-consulting services, had been contracted by early 2011, resolving an issue that was delaying much needed support to the districts and villages/communities. Although involving a minor adjustment from the Project’s original design, the substitution of the three RMCs by individuals represented an effective approach in to support implementation. Other issues that affected procurement processes were the delays in getting the procurement committee together, multiple changes in its members, and persistent issues with the quality of bidding documents and bid evaluation reports. These were the focus of follow-up and technical support and advice by the World Bank’s specialists. Project procurement plans were updated at least annually, and used effectively to monitor compliance with scheduled activities. 2.5 Post-completion Operation/Next Phase 54. The FY13-15 Country Partnership Strategy for Indonesia reports that the GOI has indicated a preference for not seeking external financing for education at this time. Nevertheless, the CPS proposes that the Bank remain engaged through knowledge services, and linking of education, including ECED, with other operations. 55. Specifically, starting in late 2012, the Project promoted a pilot collaboration between ECED and the Bank-financed National Community Development Program (Program Nasional Penberdayaan Masyarakat – Generasi Sehat Cerdas, PNPM Generasi) funded by the GOI’s Ministry of Home Affairs 12. Both programs share a common CDD approach, and the PNPM Generasi’s focus on maternal and child health and basic education provides a logical opportunity to link it with ECED, especially by using Posyandu as the entry point. Indicators related to ECED services for children aged 0-6 were added to the Generasi menu using the PNPM Generasi's performance based block grant mechanism to increase participation of the poor in ECED services. 13 The pilot leveraged the experience and materials developed under the ECED to train sub-district and village facilitators in the benefits of ECED, who then raised awareness among rural community members of the importance of ECED. PNPM Generasi block grants were invested in activities aimed at improving access to ECED services, including transportation to ECED centers, procurement of educational materials, rehabilitation of posyandu buildings to make them appropriate for use as ECED centers, and so on. The pilot did not provide fund for the local government. In 2014 the pilot was operating in three districts (Boalemo, Sumbawa, and Cianjur). 12 PNPM Generasi is an incentive-based community-driven development program that focuses on maternal and child health, as well as on universal education. The program provides incentives to communities to meet a number of maternal and child health indicators, including those on ante-natal care, immunizations and growth monitoring. Incentives are also provided for meeting or exceeding enrollment thresholds for 7-15 year olds. The pilot location was Sumbawa and Boalemo districts (where the project overlap) and Cianjur (non ECED project district) 13 These indicators included: “every child under 3 years receives joint stimulation/nurture once a week” and “every child 3-6 years receives early childhood stimulation at a minimum 3 times per week”. 14 56. In addition, the Directorate P2TK (Professional Development for ECED Teacher and Teaching Personnel)in 2012-12, requested the Bank to conduct a review in two areas: (a) capacity of ECED supervisors, and (b) training programs for ECED teachers. The review produced a draft of policy recommendations for improving the capacity of ECED supervisors with a specific focus on conducting efficient monitoring of quality of services and quality of teacher training. The review also produced recommendations on improving curriculum and delivery mechanism of the teacher training program. The review and recommendations drew significantly on the training materials and other resources produced under the Project and, therefore, have continued the effort of institutionalizing the Project’s achievements. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 57. Relevance of Objectives. The relevance of the Project’s objective is high. The Project’s objective was relevant at the time it was prepared and continues to be relevant today. It was consistent with the Government’s priorities and with the Bank’s CAS when it was prepared, and is still very relevant to the GOI and the Bank although the GOI is no longer seeking external financing for education. Participation of children aged 0-6 in ECED services remains low, with significant gaps in participation among rural and urban areas, and among income levels. As a result, by the time children from poor families begin primary school they are already behind in essential areas of development and school readiness. Yet, it is these very children from poor families that stand to benefit the most from ECED services. Therefore, the Project’s objective of improving poor children’s overall development and readiness for further education and of providing a sustainable quality ECED system continues to be very relevant. 58. Relevance of Design. The relevance of the project design is substantial. The Project’s design included activities at all levels of government that were needed to put in place and deliver a quality ECED system in 50 districts, while strengthening capacity to replicate this system in others. The Project established implementation units, embedded in the education office, at the Central level (CPICU), Provincial Level (PPIU) and District level (DPIU). At each level, especially at the district level, budget is allocated to build capacity of the respective government unit, including coordination meetings, study visits as well as technical assistance. At the same time, project activities also sensitized project communities about the importance of ECED and training community members to serve as ECED teachers at the local level. The community- driven nature of the Project, with districts selected according to pragmatic criteria, built upon proven models of CDD projects to incorporate districts’ and communities’ needs. The selection of PDO indicators at the time of design was judged to have been appropriate. During implementation there were challenges with the PDO indicators as describe above. However, these modifications made are judged to have been appropriate for measuring achievement of the PDO as well as impact of the project on the ECED system. 59. Relevance of Implementation. The relevance of project implementation is judged to be substantial. As indicated above, implementation of the majority of project activities was successful. The Bank’s implementation assistance was responsive to emerging needs. While the Project was implemented almost entirely within the parameters defined during preparation, the 15 Bank’s implementation assistance demonstrated flexibility when issues emerged. The implementation support team responded pragmatically to the MOEC’s request to contract individual consultants to substitute for the originally planned RMCs in three regions. But it did so after ensuring that a monitoring mechanism was put in place so that the maximum of 34 individual consultants received coaching, and supervision from the national management consultant and the CPICU. Similarly, when it became apparent that the MOEC did not have the experience in M&E, the Bank provided greater than expected support to the GOI in developing the monitoring system and providing just-in-time assistance in the collection of data and evaluation of results of the Project’s impact evaluation. The development and institutionalization of the ECED data system that resulted in utilization of the data, mostly by the introduction of the spider web diagram is judged to be a project a success. Finally, the Bank’s implementation support has played a strong, catalytic role in working with the various ministries at the central level, and districts at the local level to disseminate the Project’s achievements, explore options for a continuation of support to districts and communities and pilot alternatives to sustain and expand its model to other districts and communities. 3.2 Achievement of Project Development Objectives 60. The Project’s PDO was to “to improve poor children's overall development and readiness for further education, within a sustainable quality ECED system.” The Project’s achievement of its Development Objective is rated moderately satisfactory. The PDO addressed two aspects: first, to improve poor children’s overall development and readiness for further education, and second, to do that within a sustainable quality ECED system - achievements in both of these aspects are addressed separately below. 61. Improving poor children’s overall development and readiness for further education. A randomized impact evaluation (IE) carried out between 2009 and 2013 presents rigorous evidence on the impacts of the project on a number of child development outcomes. The IE tracked the development outcomes of two cohorts of children in 2009, 2010 and 2013. These cohorts were aged 1 and 4 years old in 2009. Using a range of experimental and quasi- experimental methods (See Appendix 3 for details) the IE provides evidence to support the contention that the project “improved poor children’s overall development and readiness for further education.” Specifically: • The project led to an improvement in the physical health and well-being as well as the language and cognitive development of the 1-year-old cohort at end-line. This cohort of children outgrew the development measures used to assess them at the time of the baseline. Thus at the end-line, they were assessed using a locally adapted version of the Early Development Instrument. This is the same instrument that has been administered to the older cohort in past rounds of the IE. • On average the data show that the project led to fewer improvements in development outcomes for the 4-year-old cohort than the 1-year-old cohort. There is an increase in the emotional maturity of the older cohort of children. Other 16 point estimates, though positively signed, are not statistically distinguishable from zero. 14 • However, when the data are disaggregated, they reveal that these average results mask substantial improvements for poorer children in the 4-year-old cohort. There is compelling evidence that the project led to improvements in poorer children’s social competence, language and cognitive development as well as emotional maturity. 15 62. The two cohorts of children studied in the impact evaluation provide complementary evidence on the Project’s impact. While several moderately sized development impacts were found, the strongest evidence of an impact across all analyses is for improvements in children’s language and cognitive development. This domain was found to be the weakest for children in rural Indonesia when baseline assessments of the children were conducted in 2009 and is one of the domains most strongly associated with school readiness. 63. Taken together, the evidence from the IE suggests that exposure to the Project improved “poor children's overall development and readiness for further education.” The extent of the improvement in children’s outcomes that can be attributed to the Project varies depending on the aspect of school readiness under consideration. It also varies depending on the cohort of children being studied. 64. Developing a Sustainable Quality Community-based ECED System. The project has been moderately successful in meeting the sustainability issue, as substantiated by the achievement of selected intermediate outcome indicators, summarized as follows: 65. With respect to the indicator “percentage of targeted districts with regulations supporting ECED positions in district government”, 48 out of 50 districts have established permanent ECED positions within their education office, exceeding the target by 16 percent. In addition, the project helped establish technical committees in 47 of the 50 districts to further support coordination of quality ECED services. In addition to supporting ECED positions, the majority of districts and several villages have issued regulations or decrees in support of ECED 16. 66. With respect to the indicator “development of standards and recognition of model ECED centers”, target was partially met. National Standard for ECED services were issued through Ministerial Decree 58 in 2009 consisting of four categories: (i) Developmental Achievements in Early Childhood; (ii) Educators and Education Personnel; (iii) Content, Process and Assessment; 14 This lack of statistical significance is a reflection of the fact that the evaluation was designed to be able to detect a minimum effect size larger than the one obtained in reality. Thus the design does not have enough statistical power to detect the effect sizes actually observed. 15 These results are robust to alternative approaches. Comparing the evolution of outcomes between children whose parents report having poor parenting practices at baselines reveals a similar story. Together these results are consistent with the project having improved outcomes for the most vulnerable groups. 16 Examples of regulations in support of ECED issued by districts and villages are Peraturan Bupati (Decree of District Head) and Peraturan Daerah (District Regulation) as well as Peraturan Desa (Perdes) 17 and (iv) Facility and Infrastructure, Management and Financing. Although not yet fully enforced, they do represent a significant achievement and lay the foundation to improve the quality of programs, teachers and training/professional development. The target for the establishment of the district resource centers, with matching grants from district government, was missed: 25 model centers were established compared to the target of 36. However, the project did meet the target of providing 10 model centers per district with additional block grants. 17 67. With respect to the percentage of targeted districts with qualified and functioning district training teams, the target was 90 percent met. 18 All of district trainers completed their training. However some trainers did not conduct follow up supervision due to lack of transport support to visit teachers in remote areas. District trainers continue to serve even after the project closed. The extensive training and capacity building activities conducted at the national, district and community levels of the ECED system has undoubtedly raise the quality of ECED services being provided 19. 68. In addition to meeting the project intermediate indicators, the following sustainability- related activities have been achieved with project support: (i) as envisaged, all 50 districts signed an MoU prior to their participation in the project that included actions to support the institutionalization of quality ECED at the district and provincial levels; (ii) as part of the MoU, participating district ECED financing increased from US$356,000 in 2006 to US$11.91 million in 2013 which is a considerable achievement considering there was not a legal obligation to allocate ECED funding at the provincial or district levels; (iii) all districts have ECED supervisors 20 who are responsible for quality control and supervision of ECED centers; (iv) each district has a Forum PAUD which consists of people from different ECED backgrounds to support and advocate ECED; (v) 36 districts have included ECED in their strategic development plan; 23 districts allocated funds for volunteer teacher incentives, 40 districts to monitoring and evaluation and more than half the districts to infrastructure, coordination and operational program work. 3.3 Efficiency 69. The Project resulted in an efficient use of resources, and was implemented efficiently. A full ex-post economic analysis is provided in Annex 3 and summarized below. The PAD conducted a cost-benefit analysis of the Project, which estimated that the average project benefit cost ratio would be 6:1. This analysis assumed a 5 percent discount rate, rates of return to education ranging from 29.8 percent for primary education to 13.8 percent for higher education. It further assumed that the target of 738,000 children would be reached and that the target for educational attainment in project districts would be the average educational attainment of non- 17 Under the Project, two types of model centers were established: (i) one model center identified within each district that would receive a US$200,000 matching grant from the Project (to be matched by at least US$100,000 from the district budget); and (ii) 10 model centers per district that received additional block grants of US$25,000. 18 The target was for all 50 districts to establish a training team. 19 Please see Annex 2 for the complete list of trainings and capacity building activities 20 57 percent of districts elected their supervisors while 43 percent of districts assigned supervisors. 18 project districts. Lastly this analysis assumed that benefits would accrue in the form of increased wages till age 60. 70. This analysis has been updated using the actual number of children reached (673,162 as of June 2013) and the actual observed increases in educational attainment. These vary depending on the age cohort analyzed. For the four-year old cohort the estimates are 0.1 years on average and 0.4 years for the one-year-old. It also uses more conservative estimates of rates of return: which range from 6.8-10.6 percent as estimated by Duflo (2001) and from 6.1-12.3 percent as estimated by Patrinos et. al. (2006). 71. Assuming a 6.5 percent rate of return to education (averaging the bottom end of the rates of return reported in the papers above), and that children do not begin to realize the benefits of increased wages until age 18 and that they do so for 40 years, a 0.1 year increase in schooling results in a benefit-cost ratio of 1.3. 21 Similarly, a 0.4 year increase in schooling results in a benefit-cost ratio of 4.3. Using the lowest rate of return assumed in the PAD (11.2 percent) suggests a much correspondingly higher benefit-cost ratio of 2.1 – 7.3. Thus even the most conservative cost-benefit estimates would suggest that the Project did better than break even. This is an underestimate of the benefit given the conservative estimates of returns to education used, the shorter-than-usual time horizon for accrual of benefits as well as the fact that these are only private returns. Social returns to education have not been factored in. 72. Efficiency in the Use of Resources. The final allocation of funding under each of the Project’s three components is provided in Annex 1. Under component 1, no more than 20 percent of the funds received by each village could be used on physical infrastructure or construction according to the Project’s design. Thus to compare efficiency, the amounts other projects have spent on classroom construction is compared with the amount communities could spent on construction under the Project. With one exception, the Indonesia ECED Project spent less on physical infrastructure than other education projects, which had a similar component suggesting a relatively more efficient use of project resources. The Project was also implemented efficiently and completed, as scheduled, with no extensions of the closing date. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 73. The Project’s overall outcome is rated Moderately Satisfactory. The objectives were and continue to be highly relevant. The project help develop an ECED mechanism that establish a framework for the delivery of ECED services to poor communities. This mechanism counts on strong institutions at the national level, including training, standards and monitoring and evaluation, developed and implemented a model for efficient delivery of ECED services that can be sustained and expanded using alternative funding mechanisms, including those provided at the district level and through other complementary programs that transfer funds through block grants. Although there were data collection problems with the original PDO indicators, stemming 21 The 2012 GDP per capita in PPP terms was US$4876. In our calculations of rate of return, we assume that rural wages are a third of this number. 19 from capacity constraints and resources for collection of data in 6,000 ECED centers, the proxy indicators are judged to have been substantially relevant to measure progress toward achievement of the PDO. These indicators also show attribution of project interventions to PDO achievement. The efficiency of the project is also judged to be substantial. Project Relevance Achievement of PDO Efficiency Overall Rating (Efficacy) Substantial Moderately Satisfactory Substantial Moderately Satisfactory 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 74. By focusing on 50 of the poorest districts in the country, the Project had a poverty focus by design. The IE confirmed that the Project’s impacts were greater for children from poor households and disadvantaged groups. 75. A Social Assessment was carried out upon completion to assess to what extent the Project was able to reach the poor indigenous children. The general finding was that ECED services in remote areas were able to reach indigenous children in areas accessible by transport, but that special efforts would be required to cover very remote areas. Although it found that these centers were providing good services to the children mostly due to the enthusiasm of the volunteer teachers that deliver them, it also found that ECED teachers in non-urban areas require more training, especially in the form of real-life examples as opposed to books and guidelines, and supervision to provide them with support and back up. Finally, Indonesia’s indigenous peoples’ communities in remote areas cannot simply rely on school management and the communities’ willingness to pay—additional financial support is required. (b) Institutional Change/Strengthening 76. The Project had a strong institutional focus at all levels of government. The Project succeeded in putting in place a framework for ECED, generating demand at the sub-national level, and challenging districts and communities to explore and test innovative methods to sustain ECED services and increase their impact (by coordinating with complementary services). The model also depended upon integrated actions and activities at all levels, all of which received strengthening and assistance to improve their institutional capacity for implementation. The Project had an important impact on institutional capacity, change and strengthening as described above. (c) Other Unintended Outcomes and Impacts (positive or negative) 77. The Project had an important impact both on district governments and the MOEC that expand beyond it completion. At the district level, districts have considerable flexibility of their constrained budgets, but it was interesting to see that several have adopted innovative strategies to internalize ECED activities in the context of other programs supported by national regulations, thereby promoting synergies and efficiency in delivery. For example, one district has developed 20 its own program---Stimulation, Intervention and Optimization of Services for Young Children (SIOLA)—that combines ECED with parenting education, growth monitoring and micro-credit. Another district issued local regulation in support of POSPA-BKB integrating Posyandu (Health Post), PAUD (ECED service) and BKB (Parenting Education Program). 78. At the national level, based on experience with the monitoring system supported by the Project, the MOEC has expressed interest in a comprehensive monitoring system that would allow them to map the performance of ECED services across the country (not just those supported by the Project). The Project supported the piloting of an integrated system designed to integrate the Project’s monitoring system into the Government’s routine system employed at the ECED Directorate, thereby providing for continued monitoring of the Project’s activities through the GOI’s system upon completion. The piloting addressed four pillars necessary to support the function of the system, especially at the sub-national level, including readiness and capacity assessments in the area of: (a) institutional and legal framework; (b) human capacity; (c) organizational arrangements for monitoring and evaluation; and (d) infrastructure support, including the use of management information systems. The piloting is testing a web-based monitoring and evaluation system that will enhance accountability and improvements in ECED delivery by making the results widely accessible by relevant stakeholders. The pilot was conducted in 9 districts in 6 provinces, and the findings will be used to strengthen the MOEC’s ECED monitoring system at the national and sub-national level. In addition, the P2TK Directorate is interested in adopting this piloted monitoring and evaluation system to be applied by school supervisors as tools to implement the role of Penilik in ECED quality assurance. A collaborative activity between the Bank and P2TK Directorate to adopt the system is underway. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not applicable. 4. Assessment of Risk to Development Outcome Rating: Substantial 79. Risk to Development Outcome is assessed as substantial. Risk to development outcome is linked to sustainability, a topic that became a central focus of attention by the Bank’s implementation support starting in earnest during the MTR, increasing in attention through the Project’s completion. There are several aspects of sustainability, and therefore risk to development outcome, that present different challenges and risks: (a) that districts and ECED centers that received support under the Project will be able to sustain ECED programs; (b) that the MOEC will sustain the systemic quality improvements for ECED (e.g., teacher training, national standards, monitoring); and (c) that the GOI will sustain and expand the ECED model supported by the Project at the national level, providing ongoing support to both districts, villages and centers that have already received support and to those that have not. Each of these is discussed below. 80. The risk that districts and ECED centers that received support under the Project will be able to sustain ECED programs was very much linked to the Project’s design and PDO, and the focus of selection of districts and villages initially and throughout implementation (e.g., in MOU, 21 of institutionalizing the model in district and village budgets and structures, etc.). All project- supported districts have demonstrated strong commitment to ECED, and have shown strong, bottom-up demand to sustain the project investments in their villages and centers. So far, they have demonstrated willingness to contribute resources, within their means. Parental contributions are an important support for funding centers’ operations. Most centers (66.1 percent) apply a fee of, on average, Rp.7,700 per month, but only half of the parents were able to pay the fee. In addition, districts have flexibility in terms of allocating resources to ECD from their whole budget envelope, and many are in fact doing so, but resources are limited and there are competing priorities. However, between 2 percent and 9 percent of services are reported to have stopped operating due to lack of operational funding, incentives for teachers, as well as insufficient numbers of children in the villages willing to pay for ECED services. 22 At the other extreme, some districts have transformed some of the “satellite” services into full service ECED centers, with their own facilities and teachers. 23 Therefore, main concern to districts’ ability to sustain project investments relates to the risk of availability of continued funding in amounts required, especially in the absence of continued financial support from the national government either through the model developed under the Project or through another mechanism that allocates incremental funding. Nevertheless, despite limited support, by project closure above 90 percent of the project supported services are still operating. This risk would be considered moderate if funding from the national government were to become available in amounts required; in the absence of that funding, this risk is considered high. 81. The risk that MOEC will sustain the systemic quality improvements (National Standards, professional development, monitoring and evaluation) for ECED is considered negligible. The GOI has demonstrated strong commitment to ECED throughout implementation, and to putting in place the instruments needed to ensure the quality of service delivery. It has capitalized on project-supported activities, and expressed intention to expand and continue to build upon them. 82. The risk that the GOI will sustain and expand the ECED model supported by the Project at the national level is substantial, especially in view of financial commitments to ECED directly. The GOI has set an ambitious enrollment target of 75 percent by 2015 (up from 67 percent in 2010), focusing on the provision of access in rural areas. The ECED directorate has introduced the Education Assistance Program (BoP), which will provide operational funds for ECED services of Rp. 7,200,000/year. Nevertheless, for 2011, the BoP would cover a modest 8 percent of the total age group. In the absence of increased financial commitment directly for ECED, one option that has produced promising results would be to link ECED service delivery with the National Program for Community Empowerment (PNPM) funded by the GOI’s Ministry of Home Affairs. 22 The majority of centers that have closed operations are in Papua (30 percent) and North Maluku (25 percent). Failure in sustaining services in these two areas appears to be the result of: (a) the less participatory process in selecting service location resulting in the selection of communities with a small number of children aged 0-6, and (b) the high mobility pattern of the families who travel with the young children for considerable lengths of time, for which the introduced services are not compatible. 23 More than 700 “satellite” services are reported to become full service ECED centers. 22 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 83. Bank Performance in ensuring quality at entry is considered moderately satisfactory. Project preparation included: (i) good analytical work, (ii) incorporated lessons from the earlier pilot ECED project and other CDD operations, and (iii) underwent a QER and peer review process. A qualified team of technical experts prepared it, with strong support from fiduciary specialists, and with budget support from the Netherlands Trust Fund. The preparation team adopted a pragmatic approach to project design, making a rational decision not to pursue state- of-the-art ECED service delivery, but to start with a model that was feasible, replicable and responsive to village and community needs. The project included activities needed to achieve the PDO, and opted for a simplified governance structure and implementation arrangements, working within the GOI’s institutional structure and systems. The preparation team also adopted a pragmatic approach to the Project’s poverty focus by focusing on 50 of the poorest districts and in the poorest villages and communities within those districts. The PDO and PDO indicators were judged to be appropriate for the project. As indicated earlier, the DDTK PDO indicator was revised due to issues related to monitoring the indicator but the proxy indicators were appropriate for monitoring the PDO. The Royal Netherlands Government parallel financing from the Dutch also represented an instrumental and flexible vehicle to ensure project readiness and support of enhanced implementation. (b) Quality of Supervision Rating: Moderately Satisfactory 84. Quality of Supervision is rated moderately satisfactory. A highly capable team of specialists in the various areas, including fiduciary, that remained intact (with the exception of a change in the Bank’s Task Manager due to internal rotation) provided implementation support throughout the Project’s six-year implementation period. The team provided continuous, on-the- ground support, and focused that support on development impact, and on identifying issues, following up and resolving them. There was consistency in the team’s support across the multiple instruments, including the Project, analytical and advisory support, trust fund management and policy dialogue on ECED, thereby ensuring coherence and consistency on policy dialogue across GOI ministries and its various levels of government. All aspects of the Project’s reporting were thorough and well documented, including Implementation Status and Results Reports and Aide Memoires among others. Implementation support began discussing transition arrangements to internalize and expand the ECED model at the MTR, when initial results of the Project’s positive implementation and impacts were becoming apparent. The team worked intensively with officials of the MOEC, with district governments, and with other GOI ministries to disseminate the results, and provide a catalytic source promoting sustainability at the district and central levels, and expansion to other districts. Finally, QALP-2 reported that the review panel was impressed with the attention given to the Project by sector management and its input to resolving technical issues. The team quickly identified issues related to the DDTK and 23 substituted appropriate proxy indicators. The Bank team recognized the need to formally restructure the project’s results framework however, the GOI was reluctant to request the restructuring. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 85. Based on the analysis above the overall Bank performance is rated moderately satisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 86. Borrower performance is rated moderately satisfactory. Borrower is taken to mean the GOI in general, including local governments and the MOEC in particular. The Borrower had in place a good enabling environment for the Project’s implementation, including supportive macroeconomic, sectoral and institutional policies that ensured efficient implementation, and the availability of timely counterpart resources in required amounts. The Government’s commitment to the Project and its objectives was consistent throughout implementation. The GOI’s commitment was apparent not only through taking early actions needed to ensure timely implementation (e.g., issuing necessary decrees, Memoranda of Understanding) and assigning necessary counterpart funding (with delays occasionally at the start of the budget year), but more importantly through the several policy, institutional and legal measures adopted throughout implementation to improve the framework for ECED. These included the issuance of the National Policy Strategy and Design on Holistic and Integrated ECED in 2008, followed by the issuance of Guidelines for Development of Holistic and Integrated ECED in 2009 and corresponding regulations in 2013. Finally, in 2009 the GOI issued the National Standards on ECED, and important milestone for ensuring quality ECED service delivery. The GOI also worked closely with the Bank’s implementation support team to evaluate and learn from the Project, and to identify stronger prospects for its continued operation upon completion, including coordination with other government programs. Going forward, the MOEC will need to ensure the financial sustainability of the Project’s model and its expansion to other districts in Indonesia. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 87. Implementing agencies performance is rated moderately satisfactory. Implementing agencies included the CPICU in the MOEC and the 50 districts that were targeted under the Project and the communities/villages within those districts that received block grants to expand ECED services. Performance of the CPICU was moderately satisfactory, mostly the result of issues early on with procurement and financial management as indicated above. Although staffed with specialists in both areas, the issues remained unresolved until 2012. Staffing changes described earlier may have influenced this to some extent. Overall, the CPICU, supported by the 24 NMC and the Project’s institutional framework of RMCs, Community Facilitators, etc. demonstrated capacity to manage a relatively complex project, keeping implementation on track. 88. Performance of implementing agencies at the district and community level was satisfactory, especially given the relatively weak institutional capacity of most of these agencies. The demand was strong, and they worked closely with the support of project facilitators to reach out to key stakeholders in communities and villages. In most cases, implementing agencies at the district and community level embraced the Project and the model it created, demonstrating willingness to put in place the policies, structures, resources and processes needed to promote sustainability. In some cases, districts and communities explored innovative options to incorporate ECED services within a broader framework of other, complementary programs under their jurisdictions. They also demonstrated strong commitment to the Project through the staff and financial commitment they provided throughout its implementation, and to the need for timely monitoring of its results in order to continuously improve service delivery. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 89. Based on the analysis above the overall borrower performance is rated moderately satisfactory. 6. Lessons Learned 90. Lesson 1. The impact evaluation revealed that younger children (1 year cohort by baseline) benefit the most from ECED. This is likely because they benefit from a full exposure to the Project. Their parents received facilitation, the village services were established and fully operational by the time they were old enough to join and they started earlier whereas the older cohort (4 year by baseline) had shorter exposure to the activities supported by the Project. 91. Lesson 2. The Project’s interventions were provided in a package that included facilitation, block grants (for operation) and teacher training that proved to be very effective. Moreover, each village group received facilitation under the Project during three years, long enough for them to be made aware of the ECED services. This period was important to generate demand for ECED at the village level and also to build ownership by the local population by providing incentives for them to establish and operate ECED centers. Together, these design features were important for obtaining positive achievements and to ensure sustainability at the community and district levels. 92. Lesson 3. Flexible financing for initiating activities prior to signing and effectiveness was instrumental for ensuring that project implementation begins quickly. The parallel financing provided by the Royal Netherlands Government allowed the project implementation team to hire needed technical assistance prior to signing and effectiveness. This ensured that people were in place to support critical project components when the project became effective. Moreover, the block grants could also begin right as the project became effective so that villages could begin their ECED programs quickly. 25 93. Lesson 4. The structure of the Bank’s sector assistance teams can have strong impact on its delivery of effective, integrated assistance. For the Project, the team responsible for implementation support was also responsible, with different task management responsibilities, for analytical and advisory services, trust fund management and overall policy dialogue. This provided for consistent and coordinated assistance delivery to the GOI on ECED. The fact that the team was largely based in the field increased its effectiveness, especially in coordinating with other teams delivering assistance to explore options for sustainability of ECED service delivery (e.g., coordination with the PNPM team). 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies Comments from Implementing Agency was received and documented in Annex 7. The comments are considered consistent with the general ICR. (b) Cofinanciers By the time of the submission of the ICR, the comments from the co-financier, the Dutch Government, was not available. Once received it will be documented in the project file. (c) Other partners and stakeholders Not applicable. 26 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Actual/Latest Percentage of Components Estimate (USD Estimate (USD Appraisal millions) millions) Component 1: Increasing Integrated ECED Service Delivery Through 96.65 93.23 96.48 Community-driven Mechanisms in Targeted Poor Communities Component 2: Developing a 12.20 7.71 63.20 Sustainable System for ECED Quality Component 3: Establishing Effective Program Management, Monitoring and 18.89 6.29 33.30 Evaluation Total Base Cost 127.74 107.23 83.94 Physical Contingencies 0.00 0.00 Price Contingencies 0.00 0.00 Total Project Costs 127.74 107.23 83.94 Total Financing Required 127.74 107.23 83.94 (b) Financing Appraisal Actual/Latest Estimate Estimate Percentage of Source of Funds Type of Co-financing (USD (USD Appraisal millions) millions) IDA Credit 67.50 70.041 103.76 Dutch Trust Fund 25.30 23.40 92.49 Borrower 34.94 13.79 39.47 1 There were exchange rate gains due to the SDR:US$ exchange rate. 27 Annex 2. Outputs by Component Component 1: Increasing Integrated ECED Service Delivery Through Community-driven Mechanisms in Targeted Poor Communities 94. This Component was aimed at delivering integrated ECED service to poor communities in fifty districts following a Community-Driven Development (CDD) approach. The fifty districts chosen to participate in the Project were selected according to the following criteria: (a) low participation rate of 0-6 years children in ECED services; (b) low Human Development Index and Poverty Index and classified as poor districts by KepMen Pembangunan Daerah Tertinggal 2005; (c) commitment to developing ECED agenda in their respective districts as indicated by: (i) budget allocation for ECED programs; (ii) existence of a unit and staff, with a clear and comprehensive mandate for managing ECED; (iii) existence of a PAUD forum which allowed for coordination among a wide range of stakeholders, including bureaucrats of the ECED-related offices, women’s organizations, parliament members, academics, and practitioners; (iv) existence of an commitment to an action plan to coordinate ECED-related offices to support the integration of early education, health, nutrition and parenting education aspects of ECED; and (v) readiness to finance some of the project activities (after two years of implementation), so that the operation of the ECED services at community level could sustain after project completion. 95. A community, for purposes of the Project, was defined as a neighborhood (dusun) within an Indonesian village. Typically, Indonesian villages consist of about five dusuns, each with approximately 60 children aged 0 to 6 (i.e., 30 children aged 0 to 3 and 30 children aged 4-6)— an optimal size for service delivery. Accordingly, service delivery was to take place at the dusun (community) level. It was expected that approximately two dusuns per village (i.e., those with the highest numbers of poor households and children aged 0-6) would receive services in the context of the Project. The community of these neighborhoods would establish a Community Team (CT) consisting of an elected Leader, Treasurer and other members. The block grant would be channeled to the CTs, who would be responsible and accountable for the use of grants. Sub Component 1.1: Establishing Training Programs and Providing Staff Development 96. This Sub Component financed: • Selecting, according to agreed criteria, and training a National trainer team consisting of 32 National Early Childhood Specialist Teams (NEST) and 10 Community Development Specialists (CDS) from among ECED-related agencies and forums, including universities that offered ECED programs; • Developing a resource book for the NEST and CDS; • Developing modules and materials for the Training of the District Trainers, for the Training of Community Facilitators (prepared by NEST); • Developing modules and materials for training teachers and Community Development Workers (by NEST and selected Master Trainers), assisted by a training specialist, and adapted to reflect local knowledge and culture; • Delivering a program of 700 hours training to NEST and CDS, preparing them to serve as lead trainers for the Project; 28 • Financing overseas study programs in Singapore and Italy, to build NEST capacity and leadership potential about current, innovative ECED practices; • Delivering a program of 500 hours training for 192 Master Trainers (including 150 district and 42 provincial participants (Master Trainers are responsible for training teachers and CDWs); • Delivering a 500 hour training (classroom and on-the-job) for 600 Community Facilitators; • Evaluating the performance of NEST, Master Trainers, Facilitators, teachers and CDWs; • Training 11,828 teachers and CDWs in all 50 districts (by Master Trainers and, in some cases, NEST members); • Delivering additional training to Community Facilitators to assist communities in adjusting their planning to adjust to additional resources received (as a bonus for good performance and the results of combining two payment tranches into one); • Developing booklets for families of young children (28 on various topes), including the recruiting of an illustrator-consultant to make the booklets attractive and engaging for parents, to respond to the GOI’s priority of family-centered ECED • Conducting a National Coordination Workshop (ECE Summit) in December 2010 to disseminate information about the Project’s status and to renew the commitment at all levels. The participants endorsed an accord drawn up by the Project’s implementing units laying out their commitments for future sustainability, with specific commitments at each level (CPICU, districts/provinces and communities/villages, including: o Agree with different ministries in creating ECED as holistic-integrative, integrating ECED with Poysandu, and parenting; o Provide incentives for teachers o Develop strategy for institutional arrangement in view of the current integration between Formal (Kindergarten) and Non-Formal ECED; o Develop professional support for supervisors to carry out their duties in ECED; o Include ECED in district development planning blue print; o Ensure the village implementing units were in compliance with GOI nomenclature; o Include ECED in village regulation to ensure sustainable use of ECED centers; and o Conduct continuous socialization on the important of early childhood education. • Conducting capacity building activities for ECED district stakeholders, including a workshop to raise awareness of the various stakeholder to design an exit strategy for handover of project responsibility and to create a more strategic plan for ECED at the district level. This workshop served to link the district stakeholders with District Consultants and Facilitators, and renewed both district commitment and responsibilities; • Designing a set of evaluation instruments to assess the knowledge, attitudes, and relevant skills of various categories of project personnel to review the effectiveness of 29 these individuals and the training they received: NEST members, provincial and district Master Trainers, Community Facilitators, teachers and CDWs; • Conducting a half-day session workshop on parenting education and support for families with young children, to socialize NEST and Master Trainers to the GOI’s conceptual framework, supporting research, and recommended activities in establishing “family-centered ECED”; the output was a set of background materials and follow-up plans discussed by participants. • Training trainers (ToT) at provincial and district levels, including combined theory and practical classes and on-the-job training in the form of ECED and Child Development Worker (CDW) training; • Delivering training to Community Facilitators and teachers (by NEST); • Facilitating the joint meeting of the Directorate of ECED and the Directorate for Teaching and Education (P2TK) through a workshop to improve the professional development support for ECED teachers; Sub Component 1.2: Preconditioning, Launching and Facilitation of the Community Grant Process 97. This Sub Component financed: • Preparing, finalizing, and printing the Community Grant Operational Manual and distributing it to the District Project Implementing Units and Community Facilitator Teams, and later revising it based on feedback from those teams; • Selecting and training 446 Community Facilitators, in three batches to socialize the Project to the community and facilitate Block Grant implementation at the village level; • Supporting 1000 communities in establishing village implementing units (TPKs) in preparing their community-based proposals; • Strengthening the TPK’s readiness to manage grants; • Selecting and verifying the participation of 3,000 villages and about 6,000 TPKs; • Recruiting District Consultants in regions that did not count on RMCs, to: o Carry out functions of planning, supervising, monitoring and evaluating the implementation of Block Grants in compliance with procedures and manuals; o Consolidate and validate monitoring and evaluation data on services, target beneficiaries, and KPI, including reporting; o Ensure that services are of high quality; o Ensure the flow of information from lower levels to the central level by strengthening district and village implementing unit capacity; o Coordinate with all stakeholders to support the Project at the local level; and o Manage the complaints handling mechanism. • Carrying out various activities to assist districts in providing sustainability for the Project; these included evaluation workshops organized by provinces to allow districts to identify successes and challenges; • Conducting workshops, organized by the districts that have produced sustainability plans that include strategies for districts to carry out, including: o Inviting village leaders to receive orientation about the Project and its sustainability expectation, as well as to ensure that the ECED centers receive 30 full village commitment in terms of budget, sectoral support and community participation including insertion of ECED in district strategic planning; o Ensuring valid and legal status of the land and building where the service is operating; o Ensuring formal registration of ECED centers, their access to BoP and participation in the clusters; Sub Component 1.3: Providing Block Grants to Poor Communities 98. This Sub Component financed: • Providing Block Grants to about 5,990 village implementation units in three batches, each of which was delivered in three installments. • With these grants, village units carried out their plans to prepare and/or improve their ECED centers and provide home visits, to purchase educational toys and pay salaries of trained teachers and Community Development Workers; • Materials were organized into modules on topics such as Child Development, Learning through Play, Music, Mathematics, and Children with Special Needs. They provide extensive background resources that can be used in a variety of ways. • In the ECED centers, children participate in learning activities provided by highly dedicated teachers and Community Development Workers; • Community groups managing the service were highly committed in administering their services; • Intensifying linkages with Poysandu to that early stimulation and parenting programs for children aged 0-3 (and their parents) is provided to those attending the monthly weight measurement program; Sub Component 1.4: Recognizing Model ECED Services and Using Them for Demonstration Purposes 99. This Sub Component financed: • Planning a system to recognize ECED programs that exemplify “Best Practices or Model Centre”; • Developing a manual that lays out the criteria, review and selection process for awarding grants for “Best Practices”; Criteria are designed to support future sustainability by identifying centers that could serve as models for others (10 per district); the criteria for the 10 other centers in each district include those that aim to reach out to more families by scaling up their services; supporting purchase of additional learning materials; offering apprenticeships or other on-the-job training for local teachers/future teachers and developing ECED advocacy materials; and • Conducting a competitive selection process in 50 districts to select 10 ECED centers in each district (total, 500 Model Centers) that met the required eligibility criteria for being awarded grants for Best Practices. 100. Under the Project, two types of Model Centers were established: (a) one District Resource Centre identified within each district that would receive a US$200,000 matching grant 31 from the Project (to be matched by at least US$100,000 from the district budget), and (b) 10 Model Centers per district that received additional block grants of US$25,000. The implementation for centers receiving additional block grants was relatively fast, whereas implementation in centers receiving matching grants was relatively slower since it required districts to allocate local budgetary funds to match the project grant. Nevertheless, this approach has shown good progress, but there is wide variation in the quality of the District Resource Centers, and several issues that need to be addressed. Some of the successful cases, with their upgraded facilities and capacities, have used their centers to host teacher cluster meetings in their sub-district; others have integrated their services with Posyandu or parenting programs as a buy- product of the improved community participation that the additional block grant supported. In other cases, however, centers allocated too high a proportion of their matching grants to physical improvements, neglecting investment in teacher professional development. Component 2: Developing a Sustainable System for ECED Quality 101. This Component would help ensure quality in the programs financed under Component 1, and create conditions that would support sustainable, scaled-up provision of holistic, quality ECED services. Sub Component 2.1: Quality Assurance 102. This Sub Component financed: • Establishing and Ad Hoc Committee to develop the ECED standards; • Drafting, consulting and revising (based on feedback from provinces, experts, etc.) standards for ECED (by BSNP); • Expanding ECED standards to both formal and non-formal settings; • Supporting the issuance of National ECED Standards that received an endorsement from the Minister of MOEC through Ministry Regulation No. 58 2009. • Conducting a workshop with the objective of finalizing the technical guidelines of the ECED Standards, that concluded with a series of publications detailing technical guidelines in each of the ECED Standards categories. • Disseminating ECED standards with the objective of socializing a broad range of stakeholder about their meaning and use, thereby raising their awareness and understanding of them 103. The National Standard on ECED follows BSNP procedures for developing all education- related standards. The Decree sets national standards in four categories: (a) Developmental Achievements in Early Childhood; (b) Educators and Education Personnel; (c) Content, Process and Assessment; and (d) Facility and Infrastructure, Management and Financing. The standards cover both formal and non-formal ECED in one document. Although they are not yet enforced, the National Standards represent a significant achievement, laying a necessary precondition to improve the quality of programs, teachers and training/professional development. Sub Component 2.2: Institutionalization of Quality ECED at the District and Provincial Level 32 104. This Sub Component financed: • Providing facilitation and support for the establishment of ECED units in provinces and districts; (48of 50 participating districts and 17 among 21 provinces) • Providing facilitation and support for the establishment of technical coordination committees in provinces and districts (19 and 47) • Providing facilitation for inserting ECED in the district strategic development plan of 36 districts, budget allocation to support ECED in the annual planning and budget document, and financial incentive especially for teachers volunteering their services in 23 districts 105. This achievements have several advantages: (a) ECED is mainstreamed in the local development agenda and guaranteed budget support (based on local fiscal capacity); (b) there are strengthened local policies and regulations to administer ECED, including the establishment of an ECED sector in government; (c) increased opportunities for support from cross-cutting areas for ECED intervention (e.g., Posyandu), and (d) ECED access and quality receive greater attention from development actors. 106. In addition to funding from district budgets, several ECED centers have begun to rely on alternative sources of funding to ensure institutionalization, including: (a) charging a minimum fee to parents; (b) accessing the village fund; (c) applying for district and central schemes and issuance of supporting regulations for ECED expansion; (d) support from the central government in the form of various block grants, most notably the ECED operational fund (BoP) equivalent to US$720/year/service; and (e) pursuing linkages with the private sector, through corporate social responsibility schemes. Communities have shown willingness to share resources in terms of MOECy, goods and materials and labor. Almost 45 percent of project ECED centers received support funding from the village community, and about 20 percent from the village allocation fund. Despite the relatively low levels, this support demonstrates villages and communities willingness to contribute to the sustainability of ECED centers, and the importance of continuing a process for disseminating services and results to mobilize funding. Component 3: Establishing Effective Program Management, Monitoring and Evaluation Sub Component 3.1: Program Management 107. This Sub Component financed: • Contracting the National Management Consultant (NMC) to intensively monitor and assure the quality of program implementation at district and community levels, and to supervise and function as a back-up to Regional Management Consultants (RMCs), and later facilitators in three districts; The NMC provided management support to the directorate of ECED, especially in the preparation of the Annual Work Plan, Terms of Reference, Procurement Plans as well as financial reporting. • Contracting two RMCs to monitor closely and assure quality of Project implementation and provide technical support to the implementing unit at the district level (DPIU); • Contracting a Procurement Specialist to provide ECED management with an expert in the procurement of goods and services and that is able to provide advice and feedback 33 to the CPICU on procurement issues, including preparing Procurement Plans, providing feedback on procurement issues, and assisting the CPICU procurement team in carrying out procurement processes, and assisting with procurement reporting; • Contracting a Consultant in ECED Policy to provide ECED management with a policy expert that is able to provide advice and feedback to the CPICU on ECED policy issues, including on the integration of those policies with other supporting sectors; • Contracting a Financial Specialist to provide advice and support to the CPICU on overall management, monitoring and supervision of project finance and to provide capacity building to the CPICU on financial management; • Contracting a Notary to provide ECED management with a notary who is able to provide legal advice and guarantees in any Memoranda of Understanding or contracts made by the CPICU; • Contracting a Database Specialist to provide ECED management with support to develop the Project’s database, maintain it, and provide capacity building for CPICU staff • Contracting a Web Specialist to develop and maintain the Project’s website and train ECED Staff in its use; the user-friendly website uses various media to promote and inform about ECED to outsiders and insiders alike; • Contracting a Senior Survey Specialist to ensure that the midline impact evaluation was implemented in accordance with the Terms of Reference and to ensure the quality of data gathering; the specialist ensured that the midline survey was carried out with high quality procedures and processes; • Contracting three Survey Specialists to ensure that the midline impact evaluation was implemented in accordance with the Terms of Reference and to ensure the quality of data gathering; the specialists ensured that the midline survey was carried out with high quality procedures and processes; • Contracting a Data Specialist to ensure that the midline impact evaluation was implemented in accordance with the Terms of Reference and to ensure the quality of data gathering; the specialist ensured that the data generated in the midline survey was of acceptable quality and validity and could be used for the final impact evaluation; • Contracting a Communication and Publication Specialist to establish the Project’s media campaign and publications, to support outreach and advocacy for ECED; and • Contracting four staff to provide administrative and secretarial assistance for managing remuneration and making payments to Community Facilitators. Sub Component 3.1.2: Project management at the district level. 108. This Sub Component financed: • Issuing a Decree of the Minister of MOEC on the 50 participating districts; • Issuing a Decree on the establishment of the CPICU; • Issuing a Decree on the establishment of the Project Steering Committee; • Securing expressions of commitment to participate in the Project by the 50 districts; 34 • Formalizing MoU between the MOEC and the Bupatis of the participating districts reflecting the districts’ commitment to implement collaboratively the Project, and specifying the rights and obligations of each party with respect to the institutionalization and capacity building at the local level, financing, monitoring and evaluation of the Project; (19 provinces, 47 districts) • Putting in place and training Coordination Committee at the province and district level (19 and 47); • Selecting communities/villages within districts based on four selection criteria; (3000) Sub Component 3.2: Monitoring and Evaluation 109. This Sub-Component financed • Designing the impact evaluation and draft instruments for the baseline survey; • Designing the Project’s monitoring system by the CPICU (with World Bank assistance); • Contracting base line and midline surveys • Field testing the instrument • Developing data collection instruments, manuals and reporting mechanisms to cover monitoring and evaluation activities conducted at the village, district, province and central levels; monitoring activities are managed by Community Facilitators (at the village level), by the district project implementing units and the RMC/District Consultants (district level), by the provincial implementing units (provincial level) and the CPICU and NMC (central level); • Developing a solid monitoring and evaluation system that is user-friendly and reliable, and improved data entry application was developed to support the system; the solid monitoring tool can provide comprehensive data from the field to guide decision- making and project improvement; • Incorporating questionnaires about project quality in the village implementing units’ monitoring format and guidelines; • Simplifying monitoring and evaluation instruments, reporting formats and guidelines for the village-implementing units, the Community Monitoring Teams and the Community Facilitators in response to the real circumstances in the field. The semi- annual village implementing unit monitoring instrument are all linked to data entry; the monitoring system has automatic data validity, data cleaning and data analysis and reporting, with possibility of linking data to the internet • For the impact evaluation: o Finalizing the set of survey questionnaires (village head, midwife, head of child health center, caregiver, child and household) o Overseeing the implementation of baseline survey (a first in Indonesia as this is the first time measures like the EDI, strengths and difficulties questionnaire (SDQ), Kessler 10, and child development tests were administered); o Disseminating the report from the baseline survey to government agencies and development partners; • Piloting the integration of the Project’s monitoring and evaluation system into the Government’s routine monitoring and evaluation system employed at the Directorate 35 of ECED, developing data collection, analytical and reporting system and developing manuals including those for ECED supervisors; the piloting assesses the four pillars necessary to support the functioning of the system especially at sub-national levels, including readiness and capacity assessments in the area of: o Institutional and legal framework; o Human capacity; o Organizational assessments for monitoring and evaluation; and o Infrastructure support, including the use of management information systems; 110. The comprehensive monitoring and evaluation system put in place under the Project is also a major accomplishment. It is being utilized to rate ECED centers according to various criteria that reflect the quality of service delivery, identifying areas that need attention to improve overall quality, not only in project-supported centers, but also, by linkage to MOEC monitoring systems to all ECED services across the country. 111. As explained in 2.3 the most significant factor that contributes to the utilization of the monitoring system is the introduction of the automatically generated spider web diagram (figure 1). Twelve composite indicators are tracked, based on individual data collected. These include: (i) institutional; (ii) infrastructure; (iii) frequency of services; (iv) teacher and non-teacher staff; (v) administration of ECED center; (vi) learning process; (vii) health and nutrition; (viii) parenting; (ix) financing; (x) parent/community participation; (xi) partnerships of ECED center with other relevant stakeholders; and (xii) community-based monitoring implementation. 36 Figure 1: M&E System Spider Web Diagram Status of Progress of Intermediate Indicators Indicator Baseline Target Achieved Intermediate Results Indicators Component 1: Increasing integrated ECED service delivery through community-driven mechanism in targeted poor communities % of parents/communities who have received None 80 99.8 information regarding ECED Number of communities submitting an None 6,000 5,990 expression of interest Number of block grants awarded to None 6,000 5,990 communities Increased enrollment in ECED services by 23% 31% 24 673,162 poor children 3-6 years (738,000) Component 2: Developing a sustainable system for ECED quality % of targeted districts with qualified and None 80 96.1 24 The regular monitoring system tracks new enrollment of each of the 5,990 project supported centers to meet the target mentioned in page 8 of PAD : “The project will target an estimated 738,000 children ages 0 to 6 and their parents/ caretakers living in approximately 6,000 poor communities, in 3,000 villages within 50 poor districts….” 37 functioning district training teams Development of standards and recognition of None 36 25 model ECED centers % of targeted districts with regulations None 80 72.2 supporting ECED positions in district government Component 3: Project management, monitoring and evaluation % of villages undertaking annual community None 80 80.90 assessments % of villages submitting routine reports to None 80 96.0 districts Impact study undertaken None Yes Yes Examples of Incorporation of ECED in District Planning Documents and Local Regulations District Supporting Regulation ACEH TENGGARA Surat Edaran (SE) Gubernur Aceh on BKPG/ Family Support (Bantuan Keuangan Pemakmue Gampong) 2013 to support ECED dan Posyandu. ACEH TENGAH ECED is part of district planning (RPJMD 2007-2012) which is adopted in Qonun (District Decree) No. 4 tahun 2007 on RPJMD, stating: policies to improve ECED (teacher capacity building, infrastructure development, class support facilities) TAPANULI TENGAH ECED is implicitly mentioned in the district mid-term development blueprint (RPJMD No.3/12012), stating: to improve access and quality of early childhood, and non-formal education services PESISIR SELATAN There is an inclusion of ECED in the district dvelopment planning policy (Renstrada – RPJMD 2010-2014) – it reads: Improve access to meet demand of ECED services. SIJUNJUNG ECED is part of the district mid-term RPJMD 2011-2015, aiming for early childhood education assurance for all children 0-6 SOLOK ECED is part of district mid-term development goal (RPJMD 2011- 2015), saying targeting 67% ECED enrollment by 2015, through access and community participation SOROLANGUN Local mid-term development plan and strategy (RPJMD 2011 – 2015) specifies support for teachers and Dana Operasional PAUD (DOP) TOBA SAMOSIR ECED is incorporated in the local mid-term development strategy blueprint of RPJMD 2011-2015, stating that the district needs to expand ECED access in 16 sub-district (kecamatan ); improve non- formal schooling for children aged 3-5 and develop their potentials for school readiness. Implemented programs include: renovations and new classrooms establishment, educative toys provision, treacher incentives. Official regulations with regard to ECED issued by local district are: 38 1. Perbup No. 8/2011 on village funds arrangement for villages which run ECED services. 10% of 70% ADD is allocated to ECED. 2. Perbup No. 14/2013 on ECED provision and implementation GARUT ECED is part of local development blueprint (RPJMD 2011-2015), saying “to improve ECED access” SUMEDANG ECED is inserted in local district development plan strategy policy (RPJMD 2008-2013), stating the importance of ECED as one of contributing factors to district human development. SUKABUMI ECED receives formal recognition as one of district education development goals in RPJMD 2010-2015. It sets an indicator of ECED enrollment needs to be achieved from 34.26% to 70% by 2015 A Bupati Decree was issued in 2010 on holistic integrative ECED (Perbup Nomor 41 tahun 2010 OGAN KOMERING ECED as one of district target development goals is imprinted in ILIR (OKI) RPJMD 2009-2014 (district medium-term development blueprint) – stating: improve access and supporting facilities of ECED to meet target enrollment by 2014. GUNUNG KIDUL ECED is one of prioritised sectors in Gunung Kidul. It is set forth in RPJMD 2010, section 4 indicatig achievement targets for the district. BANJARNEGARA ECED receives formal recognition as one of district education development goals in RPJMD 2010-2015. It sets an indicator of ECED enrollment needs to be achieved from 34.26% to 70% by 2015. A Bupati Decree was issued in 2010 on holistic integrative ECED (Perbup Nomor 41 tahun 2010) SAMBAS A Distritct Decree (PERDA) on early childhood education has been at its final process stage. It is expected that district education Perda will be adopted and be fully effective in 2013. 39 Annex 3. Economic and Financial Analysis Cost benefit analysis 112. The PAD conducted a cost-benefit analysis of this Project, which estimated that the average project- benefit cost ratio would be 6:1. This analysis assumed a 5 percent discount rate, rates of return to education ranging from 29.8 percent for primary education to 13. 8 percent for higher education. It further assumed that the target of 738,000 children would be reached and that the target for educational attainment in project districts would be the average educational attainment of non-project districts. Lastly this analysis assumed that benefits would accrue in the form of increased wages till age 60. 113. The ICR updates this analysis using the actual target number of children reached (673,162 as of June 2013) and the actual observed increase in educational attainment (0.1 years on average for the 4-year-olds and 0.4 years for the 1-year-olds). It also uses more conservative estimates of rates of return: which range from 6.8-10.6 percent as estimated by Duflo (2001) and from 6.1-12.3 percent as estimated by Patrinos et. al. (2006). 114. Assuming a 6.5 percent rate of return to education (averaging the bottom end of the rates of return reported in the papers above) and that children do not begin to realize the benefits of increased wages until age 18 and that they do so for 40 years, a 0.1 year increase in schooling results in a benefit-cost ratio of 1.3.25 Similarly, a 0.4 year increase in schooling results in a benefit-cost ratio of 4.3. Using the lowest rate of return assumed in the PAD (11.2 percent) suggests a much correspondingly higher benefit-cost ratio of 2.1 – 7.3. Thus even the most conservative cost-benefit estimates would suggest that the Project did better than break even. This is an underestimate of the benefit given the conservative estimates of returns to education used, the shorter-than-usual time horizon for accrual of benefits as well as the fact that these are only private returns. Social returns to education have not been factored in. Table 1: Cost-benefit-analysis for the 1-year-old cohort Per beneficiary ($) Total ($) Discounted stream of income per beneficiary 331 223,124,103 Discounted cost per beneficiary 76 51,469,388 B-C 255 171,654,715 Return for each USD invested 4.34 4.3 Assumptions: Number of beneficiaries = 673,162. Annual cost per beneficiary = USD 27. Additional years of schooling = 0.4. Benefits start at age 18 and continue for 40 years. Returns to education = 6.5%. Discount rate = 5%. Average annual earnings = 33% of 2012 GDP per capita in PPP terms. 25 The 2012 GDP per capita in PPP terms was US$4876. In our calculations of rate of return, we assume that rural wages are a third of this number. 40 Table 2: Cost-benefit-analysis for the 1-year-old cohort Per beneficiary ($) Total ($) Discounted stream of income (B) 367 247,152,852 Discounted cost (C) 76 51,469,388 B-C 291 195,683,464 Return for each USD invested 4.80 4.8 Assumptions: Number of beneficiaries = 673,162. Annual cost per beneficiary = US$ 27. Additional years of schooling = 0.48. Benefits start at age 18 and continue for 40 years. Returns to education = 6%. Discount rate = 5%. Average annual earnings = 33% of 2012 GDP per capita in PPP terms. Efficiency 115. While each village received US$18,000 over three years with which to rehabilitate or establish early childhood services, no more than 20 percent of the funds could be used on physical infrastructure or construction according to the Project’s design. Thus to compare efficiency, we compare the amounts other projects have spent on classroom construction with the amount communities could spent on construction under the Project. With one exception (which took place 5 years before the Indonesia ECED project), the Indonesia ECED Project spent less on physical infrastructure than other education projects, which had a similar component. Table 3: Comparison of construction costs under different projects Implementation Rate per m² Cost per Project Notes year (US$) classroom (US$) Baseline 2004 54 3,500 1 Laos EDP II* IDA 2005 77 5,000 2 AusAID 2009 86 5,600 3 Baseline 2009 94 6,000 4 Laos EDP II 291 schools 2010 110 7,100 5 AF* 47 schools 2013 144 9,330 6 Indonesia ECED 6,000 centers 2009 - 3,600 7 Notes: All m² rates = total contracted cost / total m² of building. 1 EDP II notional cost is 54 per m². 2 The average school construction cost financed by IDA is $77 per m², inclusive of furniture. 3 The average school construction cost financed by AusAID is $86 per m², inclusive of furniture. 4 EDP II AF notional cost is 94 per m². 5 The average school construction cost of 291 EDP II AF initially schools is $110 per m², inclusive of furniture. 6 The average school construction cost of 47 EDP II AF additional schools is $144 per m², inclusive of furniture. 7 At most 20% of the block grants could be used on physical infrastructure / construction. 8 * built by villages. Source: Laos EDP II, ICR 41 Annex 4: Project Impact Evaluation 116. The Project Development Objective was “To improve poor children's overall development and readiness for further education, within a sustainable quality ECED system.” Project outcome indicators were defined using specific child development scores including: a. DDTK Scores for children 0-3 and b. Locally-adapted Early Development Instrument (EDI) for children entering kindergarten or first grade of primary school. 117. As stated earlier, DDTK scores were not collected for children 0-3 due to the high costs of doing so – both in terms of financial and human resources – nor were children entering kindergarten or first grade of primary school routinely assessed using the Early Development Instrument. Instead DDTK scores for younger children were proxied by a series of indicators which looked at children’s growth and development from a variety of angles. This series of indicators covers almost all the information that would have been collected had the DDTK been used. Table 4: Comparison on Domain in DDTK and ECED Project Indicators Domain DDTK Indicators ECED Project Indicators Child Growth Nutrient Status Weight for Height Weight and Height Macrocephali Head-Perimeter Measurement N/A Microchephali Head-Perimeter Measurement N/A Child Development Developmental Pra-Screening Questionnaire EDI, SDQ, Child Task Performance, Delays (KPSP) Card Sort, Drawing Task 26 Seeing and Hearing Seeing and Hearing Ability Tasks EDI, Card Sort, Drawing Task Disorder Mental Emotional Mental and emotional Mental Emotional Questionnaire EDI, SDQ Disorder (KMEE) Autism Checklist for Autism in Toddlers N/A 26 The EDI covers the following developmental domains; physical health and wellbeing, social competence, emotional maturity, language and cognitive development and communication skills and general knowledge. The Strengths and Difficulties Questionnaire (SDQ) covers 5 scales – emotional symptoms, conduct problems, hyperactivity/ inattention, peer problems, and pro-social. The card sorting task is an indicator in scoring children using card sorting games differentiated using color, shape, and border. Child also asked to draw a human/figure and then a house. In baseline and midline, EDI is generated from Caregiver Questionnaire while in end-line it comes from, both Teacher and Caregiver Questionnaire. For this purpose, the panel respondents (children) divide into 2 cohorts: 1- ≤2 years old and 4≤5 years old in baseline. More detail, below are key questions comparing DDTK and EDI. 42 Domain DDTK Indicators ECED Project Indicators Inattention and Test of Inattention and EDI, SDQ Hyperactivity Hyperactivity (GPPH) Table 5: Key Questions in DDTK and ECED Project Indicators (EDI and Child Performance Test) Domain DDTK Indicators ECED Project Indicators Physical health & wellbeing Age 1- ≤2 years KPSP: Child can recognize the unknown EDI: Child looks happy when being old person; looks doubtful or happy meeting carried by someone he knows the unknown person • KPSP: Child can hold any • EDI: Child bites, holds, and objects, for example beans or observes any objects. raisins. • Child Performance Test-CPT: • KPSP: Put a pencil on your Child can easily pick up small hand’s child, can you get it back objects using a thumb and easily? another finger. • Child walks by himself with or EDI: Child walks by himself without without holding to something. assistance although he sometimes fall • Child walks across the room with down balance. KPSP: Child sits by himself without help CPT: Child sits straight on his own firmly KPSP: Without holding on something, EDI: Child has been well coordinated can child bent down to pick up his toys (for example, walking/running without and stand back? stumbling on something) KPSP: Child lifts his body into standing CPT: Child lifts his body from sitting position without help to standing position KPSP: Child can walk backward (about 5 CPT: Child can walk backwards steps or more without falling down) Age 4- ≤5 years KPSP: Child can stand on one foot EDI: Child has been well coordinated old without holding on something (for example, walking/running without stumbling on something) KPSP: Child can jump on one leg without CPT: Child can jump up on one leg holding on something (2-3 times) three times KPSP: Child can draw circle Child Performance Test-CPT: Child can draw a circle KPSP: Child can put on his clothes and EDI: Child can use toilet/WC on socks without any help his/her own (including taking off pants/skirt and put it on again, passing water/moving his/her bowels, clean after his/herself and wash hands)? KPSP: Child asked to define the color of Card Sorting Games: Child asked to pictures order the cards based on color, shape, and its combination Social Competence KPSP: Child can play together with his EDI: Child can play or work together peers and follow the rules with his friends. EDI: Child respects other children. 43 Domain DDTK Indicators ECED Project Indicators EDI: Child obeys the rules and order. Emotional Maturity Section SE Q28-57 KMEE: Child easily gets angry without EDI: Child easily get angry/upset particular reason KMEE: Child acts solitaire; avoiding his EDI: Child often like being alone, or peers and family prefer to play alone KMEE: Child has destructive behavior • EDI: Child often kick, bite or hit other children or adults • EDI: Child often fight with other children • EDI: Child often argue with adults KMEE: Child feels anxious • EDI: Child look afraid or anxious • EDI: Child feel scared easily • KMEE: Child has inattention or • EDI: Child easily distracted or easily distracted when doing unable to concentrate in doing something. anything • GPPH: Child shows hyperactivity • EDI: Child unable to sit still or (easily get cry, easy to calm calmly down―often move his body, • EDI: Child never stop moving, etc.) cannot be calm. • EDI: Child is easy to get cry. KMEE: Child feels confused and EDI: Child cannot make decision on sometimes make him hard to something communicate or making decision KMEE: Child complaints on headache, EDI: Child often complain of stomachache, or other physical disorder headache, stomachache or other aches KPSP: If child is crying if the CPT: Child cries when the caregiver parent/caregiver go away from him leaves Language & Cog Skills Section BHS Q8-33 KPSP: Shows his body parts EDI: Name five of his body parts when requested/told KPSP: When given direction, child can EDI: Follow simple and complicated follow it. For example, put the paper on directions the floor KPSP: Child can spell his name EDI: Child can write his own name KPSP: Child defines the longer line EDI: Child can define higher number between 2 lines in the picture from 2 different numbers (for example, 3 is higher than 2) KPSP: Child asked to draw a cross line Child asked to make any graffiti KPSP: Roll a ball to the child, is he roll Child push ball to make it roll the ball back to you? Communication Skills & general knowledge Section BHS Q1-7, Section SE Q26 KPSP: Child could answer the question. EDI: Child knows the knowledge For example, what do you do when you about his surroundings. For example, 44 Domain DDTK Indicators ECED Project Indicators are hungry; if horse is gig then a mouse is dog is barking; apple is a fruit; etc. small, etc. EDI: Child is curious with his surroundings and asking questions. 118. In addition, the locally adapted EDI was used as one of the key instruments in the impact evaluation of the Project. While the EDI was used for the older cohort since the time of the baseline, as the younger cohort out grew the DDTK proxies it was also used to measure their development at end-line. Thus the older cohort was assessed using the EDI three times while the younger cohort was assessed using the proxy indicators for the DDTK twice (at baseline and midline) and using the EDI once (only at end-line). 119. Consequently, while the PDO cannot be measured according to the Key Indicators originally stated in the PAD, reasonable conclusions on the achievement of the PDO can be drawn using the multiple proxy indicators available. 120. The economic and financial analysis will assess whether the Project led to a. An increase in enrollment in ECED services. b. An improvement in “poor children’s overall development and readiness for further education.” 121. Data from two sources are analyzed: a. The SUSENAS – a nationally representative household survey (which cover all 50 project districts as well as the rest of the districts in Indonesia) b. Data collected on two cohorts of children for the impact evaluation of the Project (which covers 9 of the 50 project districts). 122. The analysis starts by looking at enrollment trends in project districts and comparing them to enrollment trends in non-project districts using data from the SUSENAS. Next the analysis exploits the availability of detailed observations on children’s outcomes collected as part of the project impact evaluation to assess if there are any changes in these outcomes and whether these changes can be attributed to the project. 123. The following findings emerge and are discussed in detail below: • The project led to an increase in enrollment in early childhood services. • The project led to an increase in total years of schooling. • The project led to an improvement in the physical health and well-being as well as the language and cognitive development of the 1-year-old cohort at endline. • On average the data show that the project led to fewer improvements in development outcomes for the older cohort. However, when the data are disaggregated, they reveal that the average results mask substantial improvements for poorer children. 124. Taken together, the evidence from the impact evaluation suggests that exposure to the Project improved “poor children's overall development and readiness for further education.” The 45 extent of the improvement in children’s outcomes that can be attributed to the Project varies depending on the aspect of school readiness under consideration. It also varies depending on the cohort of children being studied. The impact of the Project on enrollment: results from the SUSENAS 125. The SUSENAS is a nationally representative household survey, which is conducted annually. The July round of the survey is representative at the district level. We identify the 50 project districts in the SUSENAS and compare their ECED enrollment rates to those of the rest of Indonesia. Table 1 shows that the scope and coverage of the question about ECED enrollment has changed over time. However at any given point in time the definition is the same for project and non-project districts. Table 6: SUSENAS question on ECED enrollment and coverage Question Age Year Type of ECED July March July March 2001 TK, KB, TPA Ever/currently enrolled Ever/currently enrolled 3-6 3-6 2002 TK, KB, TPA Ever/currently enrolled Ever/currently enrolled 3-6 3-6 2003 TK, KB, TPA Ever/currently enrolled Ever/currently enrolled 3-6 3-6 2004 TK, KB, TPA Ever/currently enrolled Ever/currently enrolled 3-6 3-6 2005 TK, KB, TPA Ever/currently enrolled Ever/currently enrolled 3-6 3-6 TK, KB, TPA, 2006 POSPAUD, SPS Ever/currently enrolled Ever/currently enrolled 2-6 3-6 TK, KB, TPA, Ever or Currently 2007 POSPAUD enrolled Ever/currently enrolled 0-6 2-6 TK, KB, TPA, POSPAUD, Ever or Currently Ever or Currently 2008 0-6 0-6 Religious PAUD, enrolled enrolled BKB/Posyandu, SPS TK, KB, TPA, Ever or Currently Ever or Currently 2009 POSPAUD, SPS enrolled enrolled 0-6 0-6 TK, KB, TPA, Ever or Currently Ever or Currently 2010 POSPAUD, SPS enrolled enrolled 0-6 0-6 TK, KB, TPA, Ever or Currently Ever or Currently 2011 POSPAUD, SPS enrolled (*) enrolled 0-6 0-6 TK, KB, TPA, Ever or Currently Ever or Currently 2012 POSPAUD, SPS enrolled enrolled 0-6 0-6 (*) For year 2011 and 2012, data are pooled. 126. Looking at the results of this analysis we see that project districts tended to have lower enrollment rates in the years prior to the Project. However, starting in 2008, there is a change in the enrollment trends: project districts start to overtake non-project districts. The results looking at children 0-6 are not particularly striking: there is an increase in enrollment rates in project districts relative to non-project districts but this increase is not large. Disaggregating the trends into those for 0-3 year olds and those for 4-6 year olds, a clearer picture emerges. The levels are 46 much lower for 0-3 year olds but even here there is a perceptible increase in enrollment rates for project districts that is not apparent for non-project districts. 127. Before the Project, the enrollment rates for 4-6 year olds in project districts are consistently lower than the enrollment rates for 4-6 year olds in non-project districts – by about 4-7 percentage points each year from 2001 – 2007 when the survey questionnaire changes. However, starting in 2008, enrollment rates in project districts begin to overtake enrollment rates in non-project districts. By 2009, which is around the time that project centers started operating, we see enrollment rates in project districts are 1 percentage point higher than in non-project districts. By 2012 project districts report enrollment rates that are about 2 percentage points higher than non-project districts. 128. These impacts appear small at first glance. However, it is important to remember that the Project was only active in 60 villages per district. Thus the fact that impacts on enrollment are visible at the district level is an encouraging sign of impacts taking place at the village level. Put another way, the district level impacts are averaging over all children in a district, those who are living in villages with the Project and those who are living in villages without the Project. As such, it would be unlikely to see this trend if there was no impact taking place in project villages. 47 Figure 2: Enrollment in project districts relative to non-project districts All children 0-6 Year of data 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 50 40 Percent enrolled 30 18 15 15 16 20 14 14 13 14 15 13 10 18 10 15 16 16 14 12 12 12 11 11 9 0 Before ECED project After ECED project Relative Time Project Districts Non-project Districts Net enrollment rate: Children aged 0-6 Children 0-3 Children 4-6 Year of data Year of data 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 50 50 40 40 33 31 32 Percent enrolled Percent enrolled 29 30 28 28 27 28 26 30 30 21 32 33 30 20 20 27 27 24 23 22 23 21 19 10 3 3 4 2 3 3 3 10 2 2 2 2 0 6 6 0 4 4 4 3 2 2 3 2 2 Before ECED project After ECED project Before ECED project After ECED project Relative Time Relative Time Project Districts Non-project Districts Project Districts Non-project Districts Net enrollment rate: Children aged 0-3 Net enrollment rate: Children aged 4-6 Source: SUSENAS July, 2001-2012 Robustness checks on results from the SUSENAS 129. Next we exploit the fact that the project was implemented in predominantly rural areas to test whether this approach (looking at district tends) is a meaningful way of assessing the project’s impact. If it is, we should see little or no change in the enrollment trends for urban areas in project districts but we should see some changes in the trends of the rural areas of project districts. Given that the enrollment rates of 0-3 year olds are very low and do not vary much over between 2001-2012, we focus the remainder of the SUSENAS analysis on enrollment rates of 4- 6 year olds only. 130. Figure 2 confirms the hypothesis just described . There is little or no change in the relative gap between urban areas of project and non-project districts between 2007-2012. Given that the project operated in rural areas, none was expected. However, for rural children, we expected to see a change in enrollment trends after the project began. We see that before the project began enrollment rates were lower for rural areas in the project districts – about 2 percentage points lower. In 2007 they are virtually the same. By 2009, a gap emerges – rural areas in project districts have enrollment rates that are 4 percentage points higher than rural areas in non-project areas. In 2011 and 2012 a gap of 4-5 percentage points continues to be discernible. 48 Figure 3: Enrollment project districts relative to non-project districts: Urban areas versus rural areas Urban children 4-6 Rural children 4-6 Year of data Year of data 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 50 50 42 39 40 40 36 36 36 35 37 Percent enrolled 37 Percent enrolled 37 33 33 37 31 31 31 34 34 29 33 33 33 30 30 31 25 25 29 25 22 28 27 26 20 21 26 25 20 24 20 18 23 22 17 20 20 21 21 18 10 10 0 0 Before ECED project After ECED project Before ECED project After ECED project Relative Time Relative Time Project Districts Non-project Districts Project Districts Non-project Districts Enrollment rate: Rural children age 4-6 Enrollment rate: Urban children age 4-6 Source: SUSENAS July, 2001 – 2012. 131. A second check on the results exploits the stated objective of the project to focus on poor children. We therefore examine the trends in enrollemnt rates across quintiles of the consumption distribution separately for urban and rural areas. As before, the hypothesis is that if there is an impact of the project, it should lead to an increase in enrollment rates for poorer quintiles in rural areas while there should be no discernible trend for poorer quintiles in the urban areas. It is important to remember that this is pushing the SUSENAS data beyond the use for which it was intended. Data are not intended to be representative of quintiles at the urban and rural level. Thus readers are cautioned against putting too much emphasis on the specific levels of enrollment in any given year and instead focussing on the differences in trends across quintiles in urban and rural areas. 132. Figure 3 reveals consistent increases in enrollment rates for the first four quintiles in the rural areas. In the case of the fifth quintile (the richest 20 percent), there seems to have been no impact – as expected if the project was successful in focusing on poorer children. In contrast there is no consistent pattern in the enrollment rates for any of the quintiles in the urban areas. Together Figures 1 – 3 suggest that the project increased enrolment, that it did so in rural areas and that it did so more for poorer quintiles in rural areas than for the richest quintile. 49 Figure 4: Enrollment in project districts relative to non-project districts by quintile and urban-rural areas Children 4- 6: Quintile 1 Urban Children 4- 6: Quintile 1 Rural Year of data Year of data 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 50 50 43 40 40 Percent enrolled Percent enrolled 35 29 29 30 27 28 30 28 30 25 25 30 26 25 24 28 24 28 27 26 22 21 21 25 20 25 19 24 23 18 20 20 16 21 21 21 16 20 20 19 14 14 18 18 17 15 10 10 14 0 0 Before ECED project After ECED project Before ECED project After ECED project Relative Time Relative Time Project Districts Non-project Districts Project Districts Non-project Districts Enrollment rate of urban children aged 4-6: Quintile 1 Enrollment rate of rural children aged 4-6: Quintile 1 Children 4- 6: Quintile 2 Urban Children 4- 6: Quintile 2 Rural Year of data Year of data 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 50 50 40 38 40 35 Percent enrolled Percent enrolled 32 36 32 32 30 30 35 30 30 29 29 30 33 27 26 31 30 30 30 30 30 25 24 24 28 22 21 22 27 25 20 20 25 25 20 20 23 21 18 22 21 20 21 20 20 17 10 10 0 0 Before ECED project After ECED project Before ECED project After ECED project Relative Time Relative Time Project Districts Non-project Districts Project Districts Non-project Districts Enrollment rate of urban children aged 4-6: Quintile 2 Enrollment rate of rural children aged 4-6: Quintile 2 Children 4- 6: Quintile 3 Urban Children 4- 6: Quintile 3 Rural Year of data Year of data 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 50 50 43 39 43 40 40 36 34 34 Percent enrolled 34 Percent enrolled 33 33 38 37 32 36 32 35 35 30 30 30 28 33 34 29 30 32 30 31 29 24 25 24 29 24 22 22 28 25 20 24 24 24 20 24 23 20 22 23 18 10 10 0 0 Before ECED project After ECED project Before ECED project After ECED project Relative Time Relative Time Project Districts Non-project Districts Project Districts Non-project Districts Enrollment rate of urban children aged 4-6: Quintile 3 Enrollment rate of rural children aged 4-6: Quintile 3 50 Figure 4 contd.: Enrollment in project districts relative to non-project districts by quintile and urban-rural areas Children 4- 6: Quintile 4 Urban Children 4- 6: Quintile 4 Rural Year of data Year of data 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 50 47 46 50 45 45 46 42 42 45 40 37 43 37 40 40 40 35 35 35 36 39 Percent enrolled Percent enrolled 38 32 33 37 38 33 32 31 32 31 34 30 34 30 30 27 32 31 30 28 29 27 27 28 26 25 19 20 20 23 10 10 0 0 Before ECED project After ECED project Before ECED project After ECED project Relative Time Relative Time Project Districts Non-project Districts Project Districts Non-project Districts Enrollment rate of urban children aged 4-6: Quintile 4 Enrollment rate of rural children aged 4-6: Quintile 4 Children 4- 6: Quintile 5 Urban Children 4- 6: Quintile 5 Rural Year of data Year of data 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 58 54 50 50 44 44 54 42 42 50 47 41 52 52 46 45 40 49 50 40 37 47 41 40 35 40 41 40 Percent enrolled Percent enrolled 44 34 40 37 38 32 32 37 40 34 35 34 38 37 30 31 31 30 34 29 27 25 20 20 10 10 0 0 Before ECED project After ECED project Before ECED project After ECED project Relative Time Relative Time Project Districts Non-project Districts Project Districts Non-project Districts Enrollment rate of urban children aged 4-6: Quintile 5 Enrollment rate of rural children aged 4-6: Quintile 5 The impact of the Project on child development outcomes: results from the project Impact Evaluation 133. The Project impact evaluation (IE) has followed the lives and development outcomes of two cohorts of children living in 310 villages located in 9 of the Project’s 50 districts (18 percent of project districts). These cohorts were aged 1- and 4-years-old respectively when they were first observed in 2009. They were followed and re-interviewed in 2010 and 2013. 134. While the original intent of the Project had been to offer services to all children ages 0–6, in practice, the most common form of service provision selected by communities was the establishment of a playgroup for four to six-year-olds, typically offered two hours a day, three times a week. The package of services offered at the community level (community facilitation, block grants, teacher training and playgroups) is effectively the intervention evaluated in the economic and financial analysis. 135. As with any randomized impact evaluation, the design was based on a number of assumptions – ranging from assumptions about minimally detectable effect sizes to assumptions about how project implementation would unfold, which services communities 51 would opt for and how project managers, parents and children would respond to the provision of these services. At different points in time, various assumptions in the design were violated (Pradhan et. al., 2013). Each time this happened, the evaluation’s ability to detect an impact of the magnitude envisioned in the design was undermined. As a result, fewer outcomes can be statistically distinguished from zero than the architects of the evaluation had envisioned at the outset. It could be that the project had impacts (positive or negative) on dimensions other than those reported here. However, those impacts are too small to be detected by the data at our disposal. The Evaluation Design 136. Given the scope of the Project, the Government originally decided to roll out implementation in three batches roughly nine months apart. Thus during the design of the evaluation the decision was made to randomly assign villages to planned phases of the Project. Analysts worked with the government to hold a public lottery and assign some villages randomly to the first batch - batch 1 (100 villages) and some villages to the last batch - batch 3 (120 villages). Villages in the second batch (batch 2) were not included in the evaluation since it was felt that nine months was too little time to gauge the impact of such an intervention. 137. Thus according to this design some villages were randomly assigned to control group status but only for 18 months. At the end of this period, batch 3 villages also received the intervention. Given that after this period, there would be no villages without the intervention, analysts also collected information on a group of villages that would never receive the project (90 villages). These villages were not randomly selected. Instead they were identified by government officials as being similar to the villages that had been randomly assigned to batches 1 and 3. Together these three groups of villages (batch 1, batch 3 and villages that never received the project) constitute the source of the data used in the impact evaluation. The Evaluation in Practice 138. Due to issues with timing of budget disbursements, in reality the design above was not adhered to. In reality the government rolled out the program in two phases. The first roll out comprised what was originally referred to as batch 1 and roughly half of batch 2. The second roll out comprised the remaining half of batch 2 and batch 3. Given that project implementation deviated from its original design, for the purposes of the evaluation, we use actual date of funds disbursal to confirm when villages actually received the project. As a result, we end up with 105 villages in batch 1 and 112 villages in batch 3. The remaining 93 villages never received the project. Thus the evaluation in practice only has small deviations from the evaluation in design.27 Figure 4 depicts the evaluation as it was designed and as it was implemented. Implications for analysis and interpretation 27 See tests of balance between original and actual batches as reported in Hasan, Hyson and Chang (eds.), 2013. 52 139. In addition to these deviations from the randomized control trial design, another consideration for the analysis are the power calculations which were undertaken to decide reasonable sample sizes. This evaluation was designed using point estimates from a highly successful impact evaluation conducted in the Philippines. In that evaluation, researchers found estimates of impact that ranged from 0.3 standard deviations to 1 standard deviation. These are large impacts. Based on these large impacts, the evaluation design assumed a certain sample size in order to be able to detect similarly large impacts with a reasonable degree of accuracy. In the event that the assumptions of the power calculations are violated it is possible for the evaluation to find no effect even when one does in fact exist. Therefore, we adjust our analysis below in a number of ways to see if we can detect effects. The Timing of Survey Data Collection 140. The analysis in this ICR is based on three rounds of data – a baseline survey which was conducted from March to June 2009, the first follow-up survey which was conducted approximately 14 months later from July to August 2010 and the second follow-up survey which was conducted between January–May 2013. There were discrepancies between the timing of the surveys and project implementation – these are depicted in Figure 4. As a result, villages in batch 1 had already implemented the ECED project for about 6 months by the time the baseline survey was fielded. Likewise, villages in batch 3 (which had not received the ECED project at baseline) started implementation about nine months before the follow-up survey was fielded. Villages without the project had no services provided through the project either at baseline or at follow-up. In each village, data were collected on households, caregivers and children’s development outcomes. 53 Figure 5: The timeline of the impact evaluation Batch 3 receives The evaluation as designed A lottery is held which assigns villages to one project - in of 3 batches. In addition a group of villages are Batch 2 the evaluation identified that are similar to the villages Batch 1 receives project - in receives this is the Project included in the lottery. In the evaluation these evaluation this is the treatment project - not control group. timeline are the comparison group villages. Data is group included in Not treated for collected on these villages in both rounds of evaluation first 18 the survey months. Subsequently treated. Survey Second round of First round of survey fielded timeline survey fielded 6 months 9 months 18 months 12 months after 20 months after Time 6 months before project starts Project starts after project after project 11 months after project starts after project project starts project starts starts starts starts Survey First round of Second round of No survey is fielded timeline survey fielded survey fielded The evaluation as implemented Batch 3 receives Remaining Batch 1 Some Batch 2 project - in Batch 2 A lottery is held which assigns villages to one receives villages the evaluation villages of 3 batches. In addition a group of villages are project - in receive the this is the Project receive the identified that are similar to the villages evaluation this project ahead control group. timeline project behind included in the lottery. In the evaluation these is the of schedule. Not treated for schedule. Not are the comparison group villages. treatment Not included first 11 included in the group in evaluation. months. evaluation Subsequently treated. 54 The Measures 141. Child development was assessed using an extensive array of internationally-validated and locally-adapted 28 child development measures including: a. The short version of the Early Development Instrument (EDI); b. The Strengths and Difficulties Questionnaire (SDQ); c. Height and weight measurements; d. Demonstrations of child skills (and mother reports of these skills), drawing on a study conducted by the University of San Carlos Office of Population Studies; e. Tests of executive function using the Dimensional Change Card Sorting (DCCS) game; f. Drawing tasks (based on the Draw-a-Man test); g. Expressive and receptive language tasks. 142. Several of the measures above assess multiple domains of child development. A higher score is better for all measures except for the Strengths and Difficulties Questionnaire (SDQ) where lower scores are better and represent fewer difficulties. 29 143. While a large number of domains of development were tracked, not all can be influenced by the interventions being studied in this evaluation. For instance, while height and weight measurements were taken, there was little to no intervention on nutrition. As such, there is little expectation that there would be an impact on this front. For other channels which reduce the ability to detect an impact see Table 2. The Children 144. The original raw data contain information on two cohorts of children in 310 villages: children who were one year old and those who were four years old in 2009 (when baseline data were collected). These children were followed and re-observed in 2010 when they were two and five years old, respectively. Given the Project’s focus on children between birth and the age of six, analysts considered that following these cohorts would allow us to understand the trajectory of child development in the age range of interest: 0–6. 145. In the time that elapsed between baseline and endline, the 1 year old cohort outgrew many of the measures that had been used to assess their development in the first two rounds of the evaluation. Thus at endline they were assessed using the Early Development Instrument. This 28 For details on how instruments were adapted and tested, see Pradhan et. al. (2013). 29 This is true of all SDQ domains. Typically one domain is reverse-scored but in this paper, we have aligned all SDQ domains in the same direction for ease of interpretation. Appendix Table 1 provides details on each of these measures of development. 55 change in the outcome measure necessitates that the analyses be conducted separately for the two cohorts. 146. The economic and financial analysis focuses on whether the package of interventions deployed by the project (community facilitation, block grants, teacher training and playgroups) had any impact on children’s development and readiness for further education. The estimations 147. For the four year old cohort, intent to treat estimates of impact are estimated using a difference-in-difference approach. 30 Empirically this involves estimating the following statistical model: (1) = 0 + 1 + 2 2 + 3 3 + 2 4 + 3 5 + + . 148. Where yit are child development outcomes at baseline (t=1) and subsequent follow-ups (t=2 and t=3) for a child i. Di is the dummy variable indicating 1 for batch 3 and 0 for the comparison group while T2 is the dummy variable denoting the midline and T3 is the dummy variable denoting the endline. Thus, δ1 captures the difference between batch 3 and villages without the project at the baseline. If villages in the comparison group are comparable with those from batch 3 in terms of child development outcomes and ECED enrollment, this estimate should be close to 0. On the other hand, δ2 captures the age effect on child outcomes between baseline and midline, while δ3 captures the age effect between baseline and endline. These capture the changes in children’s outcomes as they get older regardless of the ECED project. The time difference is about 14 months at midline and 34 months at endline. Di T2 and Di T3 are interaction terms between the intervention group dummy and the time dummies. Thus, δ4 indicates the impact of the ECED project between baseline and midline while δ5 captures the impact of the project between baseline and endline. xi is a vector of explanatory variables that include caregiver, household, and child characteristics. 31 We use robust standard errors clustered at the village level. We also run a fixed effects model on regression (1), which controls for all observed and unobserved time-invariant child characteristics. Our results from these fixed effects estimations are reported on below. 149. From a practical standpoint this estimation strategy compares the changes in development outcomes for children living in project villages to the changes in development outcomes for children living in non-project villages. The validity of this strategy rests on the assumption that if the project had no effect or if there had been no project, the differences between project and non- project villages seen at baseline should also be seen at follow-up. 30 This analysis only includes children from Batch 3 villages for whom we have a clean baseline. Their outcomes are compared to those of children living in villages without the project. 31 We control for caregiver, household, and child characteristics as follows: we include variables for child’s gender, age, household size and composition, caregiver’s depression scores, household wealth measured using an asset index and caregiver’s education level. 56 150. It is important to note that the estimated impact of the Project on the average child is not the impact on children who are necessarily enrolled in project-provided services but rather it is the impact on children who were offered the chance to enroll in the project-provided services. This estimated impact is more relevant for policy makers since most social programs are based on voluntary participation of eligible individuals. 151. For the one year old cohort, given the fact that we only have one round of EDI, we adjust our analysis and estimate the following model: (2) yij = I[Project village = 1] ∗ β1 + ∑9 j=2 I[District = j] ∗ βj + ϵ 152. Where yij indicates the outcome for child i in village j . There are two enrolment outcomes: (1) whether or not a child has ever enrolled in one of the following types of services between 2008 and 2013: a project provided playgroup, non-project playgroup, a kindergarten run by MoEC or a kindergarten run by the MoRA and (2) how long a child has been enrolled in each type of service. We control for child, household and background characteristics as follows. We include child’s age, gender, household size, household wealth (measured using an asset index) and a series of indicator variables indicating mother’s level of education. In order to control for child’s background, controls were included for whether or not the child is stunted (and if so whether the stunting is moderate or severe) and for whether or not the child is wasted. Lastly, we have detailed information on parenting practices which are also included. For all models we report robust standard errors clustered at the village level. The results 153. While we run separate analyses it is important to remember that the two cohorts together paint the full picture of project impact. We thus report the results at end-line for the cohorts together for ease of comparison. 154. The project led to an increase in enrollment in early childhood services. This increase was substantial and ranged between 7.2 percentage points for the 4 year old cohort and 15.7 percentage points for the 1 year old cohort. In interpreting these results, it is important to remember that children in these villages could have attended a variety of services. Disaggregating enrollment rates by type of service reveals evidence that there was some switching between different types of services. The numbers shown below suggest that switching away from kindergarten to project playgroups was a more common phenomenon for 4-year-olds than 1-year-olds. In contrast, switching away from non-project playgroups to project playgroups was a more common phenomenon for 1-year-olds. The fact that the overall impacts on enrollment are positive for both groups suggests that children who would not have otherwise enrolled have enrolled as a result of the access provided by the project. 57 Figure 6: The project led to an increase in enrollment rates Non-project Overall Any playgroups playgroups Any kindergartens 0.600*** 0.556*** Proportion of chidlren enrolled 0.500*** 0.400*** 0.300*** 0.199*** 0.200*** 0.157*** 0.100*** 0.072*** 0.000*** -0.100*** -0.069*** -0.091*** -0.200*** -0.131*** -0.300*** -0.233*** 4 year olds 1 year olds Note: Results for 4 year olds correspond to 5 in equation (1). Results for 1 year olds correspond to 1in equation (2) 155. The project led to an increase in total years of schooling. Those who live in project villages attended ECED services for longer than those who live in non-project villages. On average, this was 1.4 additional months (0.1 years) for the 4 year old cohort and 4.9 additional months (0.4 years) for the 1 year old cohort. As with enrollment, unpacking the overall results shows that children spend on average 2.9 - 8.6 more months in playgroups in project villages than they do in non-project villages – depending on the cohort. The 1-year-old cohort is more likely to have benefited from the project than the 4-year-old cohort given their age when the project started. This is clearly seen in these results. 58 Figure 7: The project led to an increase in months of enrollment Non-project Any Overall Any playgroups playgroups kindergartens 10.0*** 8.6*** 8.0*** 6.0*** 4.9*** Months of enrollment 4.0*** 2.9*** 2.0*** 1.4*** 0.0*** -0.3*** -0.6*** -2.0*** -0.9*** -1.5*** 4 year olds 1 year olds -4.0*** 156. Consequently, when we compare the changes in child development outcomes in project villages to changes in child development outcomes in non-project villages we find small and statistically insignificant impacts. This is true across the entire range of child and parenting practices outcomes considered. All estimates are presented in Tables 3- 8. We examine children’s development using the Early Development Instrument, the Strengths and Difficulties Questionnaire, their heights and weights as well the practices of their parents. In general we find no evidence of any impact on child development outcomes when we look at all children in a village. 157. The project led to an improvement in the physical health and well-being as well as the language and cognitive development of the 1-year-old cohort at endline. This cohort of children outgrew the development measures used to assess them at the time of the baseline. Thus at the endline, they were assessed using a locally adapted version of the Early Development Instrument. This is the same instrument that has been administered to the older cohort in past rounds of the IE. 158. On average the data show that the project led to fewer improvements in development outcomes for the older cohort. There is an increase in the emotional maturity of the older cohort of children but the remaining point estimates though positively signed are not statistically distinguishable from zero. 32 32 This lack of statistical significance is a reflection of the fact that the evaluation was designed to be able to detect a minimum effect size which several of the estimates do not cross. 59 Figure 8: The project led to an improvement in children’s outcomes Language and Communications Physical Health Social Emotional cognitive and general and Well-being competence Maturity development knowledge 0.200*** 0.172** 0.160*** 0.150*** Standard Deviation Units 0.117 0.123*** 0.100*** 0.068 0.049 0.046 0.050*** 0.017 0.014 0.000*** -0.050*** -0.025 1 year olds 4 year olds 159. However, when the data are disaggregated, they reveal that the average results mask substantial improvements for poorer children. There is compelling evidence that the project led to improvements in poorer children’s social competence, language and cognitive development as well as emotional maturity. 33 33 These results are robust to alternative approaches. Comparing the evolution of outcomes between children whose parents report having poor parenting practices at baselines reveals a similar story. Together these results are consistent with the project having improved outcomes for the most vulnerable groups. 60 Figure 9: For the older cohort, average impacts masks improvements for poorer children Language and Communications Physical Health Social Emotional cognitive and general and Well-being competence Maturity development knowledge 0.300 0.243** 0.236** Standard Deviation Units 0.250 0.200 0.172** 0.162 0.140*** 0.150 0.117 0.100 0.046 0.051 0.050 0.017 0.014 0.000 All Poor households Conclusions 160. The two cohorts of children studied in the IE provide complementary evidence on the impact of the project. While several moderately sized development impacts were found, the strongest evidence of an impact across all analyses is for improvements in children’s language and cognitive development. This domain was found to be the weakest for children in rural Indonesia when baseline assessments of the children were conducted in 2009 and is one of the domains most strongly associated with school readiness. 161. Taken together, the evidence from the impact evaluation suggests that exposure to the Project improved “poor children's overall development and readiness for further education.” The extent of the improvement in children’s outcomes that can be attributed to the Project varies depending on the aspect of school readiness under consideration. It also varies depending on the cohort of children being studied. 61 Annex 5. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Christopher Thomas Sector Manager EASHE Mae Chu Chang TTL EASHE Susiana Iskandar Co-TTL EASHE Vincente Paqueo Consultant EASHE Sophie Naudeau Senior Education Specialist AFTEE Richard Cibulski EASSD Raj Soopramanien LEGEA Luc Lecuit EAPCO Rizal Rivai Procurement Specialist EAPCO Steve Burgess EASSD Ratna Kesuma Senior Operations Officer EASHE Marion Carey Hyson ECED Consultant Charles Chandler EAP Quality Team Fadia Saadah HNP Sector Manager EASHD Supervision/ICR Mae Chu Chang Lead General Educator EASHE Cristobal Ridao-Cano Country Sector Coordinator EASHS Susiana Iskandar Senior Education Specialist EASHE Rosfita Roesli Senior Education Specialist EASHE Menno Prasad Pradhan Co-TTL EASHE Rosfita Roesli Senior Operations Officer EASHE Rizal Rivai Senior Procurement Specialist EASHE Novira Asra Financial Management Specialist EAPCO Titie Hadiyati Consultant Puspavalli Namasivayam ECED Consultant Erika Dunkelburg ECED Consultant Moesijanti Soekatri Nutrition Specialist Paulus Bagus Tjahjanto Procurement Specialist Ratna Josodipoero CDD Specialist Djoko Hartono Monitoring & Evaluation Specialist Amin Robiarto Consultant Sally Brinkman Consultant Deddy Junaedi Consultant Megha Kapoor Consultant Amanda Beatty Impact Evaluation Specialist Rima Artha Research Analyst 62 Hafid Alatas Operations Assistant Dandan Chen Economist EASHE Amer Hasan Education Economist EASHE Novira Kusdarti Asra Sr Financial Management Specialist EASFM Djoko Hartono M&E Consultant EASHE Joppe Jaitze De Ree Consultant Research Analyst EASHD Budi Permana Procurement Specialist EASR1 Husnul Rizal Research Assistant EASHE Isono Sadoko Consultant EASID Tukiman Tarunasayoga Consultant Mayla Safuro Lestari Putri Consultant Research Analyst EASHE Dedy Junaedy Consultant EASHE Upik Sabainingrum Consultant EASHE Anas Sutisna Consultant EASHE Lulus Kusbudihardjo Consultant EASHE Mulyana Consultant EASHE Haeil Jung Consultant EASHE Angela Kinnell Consultant EASHE Sally Brinkman Consultant EASHE Marion Carey Hyson Consultant EASHE GB Surya Ningnagara Team Assistant EACIF b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY05 19.71 100.26 FY06 66.06 295.93 FY07 N/A 2.26 Total: 85.77 398.45 Supervision/ICR FY07 25.63 73.51 FY08 14.39 42.48 FY09 31.57 64.87 FY10 49.52 96.01 FY11 36.10 95.41 FY12 46.15 118.85 FY13 44.46 124.51 FY14 36.45 111.31 Total: 284.27 726.95 63 Annex6. Beneficiary Survey Results Not Applicable 64 Annex 7. Stakeholder Workshop Report and Results Not applicable 65 Annex 8. Summary of Borrower's ICR and/or Comments on Draft ICR Considered Successful Outcomes 162. Early Childhood Education and Development (ECED) Program is an embodiment of government’s strategy to ensure that all children age 0-6 years from economically disadvantaged family living in rural area can participate. The children will receive benefits from integrated non- formal ECED service, so they too can grow just like children from well-off families. ECED program employs two strategies: (1) Community Capacity Building by implementation of community based approach to build community participation in developing an ECED center that is based on local potential and local wisdom; and (2) Local Government Capacity Building by implementation of Institutional Capacity Building approach. Achievements of each Activity Component are: 163. Improvement of Integrated ECED Services for the Poor. Achievements of component 1 are: • 32 people were trained for 700 training hours as NEST (National Early Childhood Specialist Team) members, and established 10 people as CDD (Community Driven Development) NEST members, provision of regional consultants (RMC) for regions 1 and 4, as well as provision of district consultants to provide managerial support at the regional level. • NEST and CDD NEST had trained 192 people as Master Trainers at provincial and district levels, and 658 people as Community Facilitator Team (TFM) members. The Master Trainers had trained 11,980 ECED educators and staffs, while the TFM members had assisted the community and ECED centers in providing quality ECED service that is holistic, integrated and sustainable. • Trainings for educators and CDW were held by program managers at provincial level or the PPICU (Province Project Implementing and Coordinating Unit) in 21 provinces, funded with de-concentration fund. A total 11,980 people have been trained as educators and CDW through this ECED training program • Establishment of TPK and launching of block grant fund. This component is intended to ensure the availability of ECED service that can improve access and quality of service for children from poor families. The ECED program has disbursed Rp 90,000,000 (ninety million Indonesian rupiah) block grant to every established ECED centers (TPK) that is given in three stages of disbursement in three years period. Until the end of the program, 5,990 activities management teams (Tim Pengelola Kegiatan/TPK) have been established with a total block grant disbursement of Rp 539,100,000,000 (five hundred thirty-nine billion and one hundred million Indonesian rupiah) in 51 districts. The following graph shows the condition of the TPK under the ECED Program: 66 164. Several mobile ECED centers (PAUD Kunjungan) have been established under this ECED program. The following graph shows the development of mobile ECED centers: 67 165. The following information of the performance of the TPK of ECED Program is based on national monitoring and evaluation efforts: Based on collected data, total number of early-age children served by TPK under the ECED Program is 673,162 early-age children, or has reached 91.21 percent of the set target of 738,000 children. However, using average growth rate of new early-age children in previous years, the target could be met by the end of 2013. The following graph shows the increased number of early-age children served every year, cumulatively. 166. The implemented ECED Program has opened up opportunity for the delivery of holistic and integrated Early Childhood Development service that covers integration of health, nutrition, caring and education services, for the protection and welfare of children that is necessary in order to have healthy, intelligent, happy, and of noble-character Indonesian children, as is the motto of ECED. Based on identification result, the following graph shows how the TPK collaborated with other sectors in implementing holistic-integrative policy. 68 167. Development of a sustainable system for quality ECED. The achievements of this component are: • Development of ECED Standards with the enactment of Ministerial Regulation no. 58 year 2009 on the Standards for Early Childhood Education. • Formulation of technical instruction to implement the ECED Standards. • Formulation of Development Pattern for Management of ECED Courses and Trainings. • Formulation of curriculum for Bachelor, Master and Doctoral Degrees on Early Childhood Education. • Formulation of Minimum Service Standard for Early Childhood Education. • Implementation of Media Campaign • Supervisor Orientation • Workshops on Teaching Materials Development • Workshops on ECED Program Management Coordination • Mid-term review • Provincial Coordination Workshops at provincial level • Workshops on ECED program sustainability at district level. • Some social assistance was given to maintain ECED quality: ECED internship, ECED unit, BOP, Parenting, and social assistance from ECED partner organizations. • Workshop on Teaching Materials Development • Workshop on Capacity Building for ECED Trainers • Workshop on NSPK training providers • Workshop on Capacity Building for ECED Supervisors • Workshop on Capacity Building for Recipients of Matching Grant and Additional Block Grant • Training on ECED CSR 168. To improve the quality of ECED staffs, several short courses have been carried out since 2009 to 2013. The short courses include: • Short Course of Reggio Emilia. 69 • Short Course Exploring Reggio Emilia Approach Method in Singapore, attended by the Directorate of ECE, ECED Management, NEST Team, REST, Himpaudi, P2PNFI and BP-PNFI. • Short Course In-depth Study of Reggio Emilia in Italy. • In Country Training, attended by the NEST team of ECED Program and supervisors. • Parenting Short Course, Project Management Short Course, APE Short Course. • In country Training for district staffs, to intern at Model ECED center. • Short Course ECED to China, participated by mothers from award-winning districts, educators, TPK, and TFM, ECED Partners, and top management of the directorate. • Short Course on improving understanding on Early Childhood Education in Alfalah. • Asia-Pacific ECED International Conference • National ECED Conference • Trainings for Assistant Educators 169. Efforts to develop ECED quality assurance system at the district level, in accordance with component 2 of the ECED Program, has encouraged district governments to develop Model ECED (PAUD Percontohan) through the provision of Rp 200,000,000 (two hundred million Indonesian rupiah) Matching Grant from their local government budget allocation. As a result, 29 Model ECED centers have been established in 29 districts, within two fiscal years, four centers in 2011 and 25 centers in 2012. 170. Provincial and District ECED Programs have gained some achievements that are in line with the commitment agreed by the central government, in this case the Directorate of Early Childhood Education and Development, and provincial and district governments. Achievements of MoU at provincial level Mempertahankan Pengelola 42.86% Laporan Kegiatan 95.00% Membentuk Komite Koordinasi 90.48% Forum PAUD 100.00% HIMPAUDI 100.00% Mendukung Kesinambungan 80.95% Kontribusi APBD 95.24% Menetapkan Seksi PAUD 90.00% PAUD Masuk Renstrada 90.48% 70 Achievements of MoU at district level Increased allocation of fund in local government budget for ECED 71 171. In 2006 until 2013, ECED Program has been implemented well by the Central (CPICU), Provincial (PPICU), District (DPIU) and community (village) levels, in accordance with the Program components. Therefore, based on the achievements of the program, the managing organizations of the ECED Program has been successful in performing their tasks in accordance with Project Appraisal Document (PAD), Loan Agreement as well as Finance Agreement. The following graph shows the comparison of those achievements: Issues Considered To Be Not Effective • Appropriate implementation of PDO measurements in this ECED Program has been stipulated in the PAD. It is implemented by measuring the development rate of children age 0-3 years using DDTK and EDI. This process of PDO reporting, as described in the PAD, is difficult to be conducted regularly in short period of time (quarterly). This process requires specific study that involved cross-sector policies (especially health sector), facilities and infrastructures, as well as experts with comprehensive understanding on child development. Therefore, it requires an extended period of time and fully capable resources at the central and field levels. Actually, educators and CDW have been trained to use DDTK, and they have carried out the process at institutional level. However, it is limited only to activities such as weighing, head circumference measuring, and distribution of supplemental food. It has not gone into the analytical part of the process. 72 • Master Trainers did not assist the educators at TPK regularly. If the ECED program would be re-designed, the following changes have to happen: • In the current implementation of ECED Program, there have been many discrepancies between program plan and financial plan. It requires serious attention. Program and financial need assessments have to be conducted and the results incorporated in the documented plan. • ECED Program had two main indicators, namely the KPI and PDO. To this date, the KPI was measured using monitoring and evaluation instrument, while the PDO focused more on DDTK measurement, and it was not effective. Therefore, in Program Indicators identification, specific attention has to be given to indicator measurements (instruments), Duration of measurements, as well as outputs. Those issues have to be determined and regulated in advanced, to produce measurable targets and achievements of the Program. • There should flexibility in fund utilization in accordance with field condition. It has been stipulated as one of the clause in the PAD, and use of funds has to be based on planning and field study. • Use and reporting of RK funds have to be reported to the Bank in Indonesian rupiah, to avoid foreign exchange loss. • Portion of fund sharing with the Local Government Budget (APBD) should be determined and agreed by central and local governments, and reflected in a Memorandum of Understanding. • Appointment of program managers, both at central and local levels, should be based on the Director General’s Decree to prevent any replacement of the program managers. • Lender and central government should perform regular monitoring on MoU compliance. Moreover, there should be reward and punishment system being applied to local institutions that failed to meet stipulations of the MoU or any agreed requirements. Implementation Stage • At the preliminary phase of the program, supervisor must assist TFM. Supervisor should be provided with honorarium or transportation allowance that can be funded by program fund allocation under the monitoring and evaluation component, to ensure that they can carry out effective quality assurance measures at field level. • MOT should be recruited from non-civil servants background and be given incentives from the program fund allocation, to ensure the same level of professionalism in performance as with the TFM. This issue became a recommendation because experience shows that civil-servants-recruited MOT were unable to perform effectively, especially in assisting/mentoring the educators at TPK. • Indicators of targeted early-age children to be served at ECED centers should be clarified, whether it started from age 0 or 2 years. It will clarify the number of early-age children who had been served at center level. Recommendations for the World Bank: • To provide trainings on financial management, procurement of goods and services, as well as other capacity building trainings for the central and regional management teams. 73 • To have more flexible utilization of the Royal Netherlands block grant that can cover not only consultants’ fee but also other needs. 74 Annex 9. Comments of Cofinanciers and Other Partners/Stakeholders By the time of the submission of the ICR, the comments from the co-financier, the Dutch Government, was not available. Once received it will be documented in the project file 75 Annex 10. List of Supporting Documents Project Appraisal Document, Indonesia, Early Childhood Education and Development Project, Report No. 35326-ID, dated May 31, 2006. Implementation Status and Results Reports Mission Aide Memoires World Bank, Early Childhood Education and Development in Poor Villages of Indonesia Strong Foundations, Later Success, Amer Hasan, Marilou Hyson, and Mae Chu Chang, Editors Various informal presentations and informal notes on the Project’s Impact Evaluation and other aspects such as monitoring Indonesia Country Partnership Strategy FY13-15, Report No. 72906-ID, dated December 13, 2012 76 77